mehrab-wj/tiktoken-php
最新稳定版本：v1.0.0
Composer 安装命令：
composer require mehrab-wj/tiktoken-php
包简介
a clone of python tiktoken but for PHP! fast BPE tokeniser for use with OpenAI's models.
README 文档
README
PHP Text Tokenizer for GPT models
About
A PHP toolkit to tokenize text like GPT family of models process it.
Forked from semji/gpt3-tokenizer-php to bug fixes and improvement.
Requirements
- PHP 8.1
- mbstring extension details here on how to install mbstring
Usage
First install the package using composer:
composer require mehrab-wj/tiktoken-php
use TikToken\Encoder; $prompt = "Ai is cool"; $encoder = new Encoder(); $tokens = $encoder->encode($prompt); // [32, 72, 318, 3608] // Get tokens count: echo count($tokens); // 4
统计信息
- 总下载量： 70.22k
- 月度下载量： 0
- 日度下载量： 0
- 收藏数： 17
- 点击次数： 1
- 依赖项目数： 3
- 推荐数： 0
其他信息
- 授权协议： MIT
- 更新时间： 2023-04-19