kitoken

0.11.0

Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

License

A permissive license that comes in two variants, the BSD 2-Clause and BSD 3-Clause. Both have very minimal requirements about how the software can be redistributed.

Package Registry

npm Source Homepage

README badge Customize →

Loading dependencies…