What's Changed
- feat: add FixedLengthPreTokenizer for fixed-length tokenization by @CodeWithKyrian in #3
- Implement configuration access and reconstruction for Tokenizer by @CodeWithKyrian in #4
- Refactor Tokenizer properties and visibility by @CodeWithKyrian in #5
- Make Encoding class Countable by @CodeWithKyrian in #6
Full Changelog: 2.0.0...2.1.0
Contributors
CodeWithKyrian