tokenizers
PyPIv0.23.1Overview
tokenizers is an PyPI package. The latest version is 0.23.1.
Who should use it
Teams working in the PyPI ecosystem who need this functionality and value a focused solution.
When not to use it
Consider an alternative if you need to have first-class TypeScript types out of the box, or if a more actively-maintained option better fits your PyPI stack.
Pros
- Lean dependency tree (1 direct dependencies).
Cons
- No bundled TypeScript types (may require a separate @types package).
- No license detected — review usage terms before adopting.
Auto-generated from collected registry metadata. No external claims are inferred.
Relationship graph
Dependencies (left) and dependents (right) of tokenizers.
Dependencies
Runtime (1)
Used by
Popular packages that depend on tokenizers.
Frequently installed together
Fundamental package for array computing in Python
Fast, Extensible Progress Meter
Core utilities for Python packages
Client library to download and publish models, datasets and other repos on the huggingface.co hub
Alternative regular expression module, to replace re.
Typer, build great CLIs. Easy to code. Based on Python type hints.
No description provided.
Version history
| Version | Published | License |
|---|---|---|
| 0.23.1latest | 2 months ago | — |
| 0.23.1rc0 | 2 months ago | — |
| 0.23.0rc0 | 2 months ago | — |
| 0.22.2 | 6 months ago | — |
| 0.22.2rc0 | 7 months ago | — |
| 0.22.1 | 10 months ago | — |
| 0.22.1rc0 | 10 months ago | — |
| 0.22.0 | 10 months ago | — |
| 0.22.0rc0 | 10 months ago | — |
| 0.21.4 | 11 months ago | — |
| 0.21.2 | 1 year ago | — |
| 0.21.2rc0 | 1 year ago | — |
| 0.21.1 | 1 year ago | — |
| 0.21.1rc0 | 1 year ago | — |
| 0.21.0 | 1 year ago | — |
| 0.21.0rc0 | 1 year ago | — |
| 0.20.4 | 1 year ago | — |
| 0.20.4rc0 | 1 year ago | — |
| 0.20.3 | 1 year ago | — |
| 0.20.2 | 1 year ago | — |
Frequently asked questions
How do I install tokenizers?
Run `pip install tokenizers` to add tokenizers to your PyPI project.
What license is tokenizers released under?
No license was detected in tokenizers's metadata; review the repository before use.
How popular is tokenizers?
tokenizers has approximately 0 downloads per week and 1 known dependent packages.