gioni06/gpt3-tokenizer
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
时间：2023-01-17 08:19
turanjanin/serbian-language-tools
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
时间：2021-05-24 14:02
hexydec/tokenise
A program for performing lexical analysis, written in PHP
时间：2021-04-17 19:35
infinityloop-dev/graphpinator-tokenizer
Tokenizer subproject for GraPHPinator: GraphQL server implementation for PHP.
时间：2020-11-20 17:12
fi1a/tokenizer
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознанные группы (лексемы).
时间：2020-09-09 16:17
tacoberu/bnf
Framework for building BNF like parsers.
时间：2020-06-21 17:54
phplrt/lexer-contracts
A set of phplrt lexer abstractions
时间：2019-06-23 18:29
phplrt/lexer
The phplrt lexer library that can be used for scanners implementation using PCRE
时间：2019-05-05 18:12
kerasai/behat-tokenizer
Token replacement for Behat test suite.
时间：2019-01-22 16:40
tsukasa/php-tokenizer
Polyfill for php extension Tokenizer
时间：2018-09-28 09:19
remorhaz/php-unilex
Unilex: lexical analyzer generator with Unicode support written in PHP
时间：2018-04-18 10:31
netgen/query-translator
Query Translator is a search query translator with AST representation
时间：2017-03-23 20:57