Skip to content

OpenNMT/Tokenizer

Error
Looks like something went wrong!

About

Fast and customizable text tokenization library with BPE and SentencePiece support

opennmt.net/

Topics

python unicode natural-language-processing cpp icu tokenizer machine-translation tokenization bpe sentencepiece

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

306 stars

Watchers

19 watching

Forks

74 forks
Report repository

Releases 30

Tokenizer 1.37.1 Latest
Mar 1, 2023
+ 29 releases

Packages

No packages published

Contributors 15

Languages