</>PackageGraphPackageGraph

tokenizers

PyPIv0.23.2

Not recommended for new projects

No — tokenizers should not be adopted for new projects. No license declared — you have no explicit right to use this code.

  • ?Adoption: grade ?, Not reported by this registry.

    No download data reported for this package yet.

  • AMaintenance: grade A, Excellent.

    Last published 6 days ago. 22 releases in the last 2 years.

  • BWeight: grade B, Good.

    Pulls in 18 transitive packages, about 8.8 MB installed (at least — some of the tree is still being crawled).

  • BSupply chain: grade B, Good.

    This package lists no maintainer.

  • FLicense: grade F, Failing.

    No license declared — you have no explicit right to use this code.

Grades are computed deterministically from registry metadata collected by PackageGraph — downloads, dependents, publish dates, the resolved dependency tree, maintainer count, and declared licenses. No third-party scores are used.

Install cost

What you actually take on by adding tokenizers to a project.

1
Direct dependencies

Runtime packages this one declares itself.

18
Total installed

Distinct packages in the full runtime tree, deduplicated the way a package manager would.

8.8 MB
Install size

Unpacked size of this package plus its entire runtime tree.

4
Tree depth

Longest resolved dependency chain below this package.

Part of this dependency tree has not been crawled yet, so these figures are a lower bound rather than a final total.

Licenses in the dependency tree

Apache-2.0BSD-3-ClauseMITMPL-2.0MPL-2.0 AND MIT

Every license you inherit by installing tokenizers, not just its own. Check this before a legal review, not after.

Compatibility

Python
python >=3.10

Quick start

A usage example from the tokenizers README.

pythonFull README ↗
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.processors import ByteLevel

tok = Tokenizer(BPE())
tok.pre_tokenizer = Whitespace()                 # ✅ thread-safe on 3.14t
tok.post_processor = ByteLevel(trim_offsets=True)

Dependencies

Declared by tokenizers v0.23.2. Runtime dependencies are installed with the package; dev dependencies are not.

Runtime (1)

huggingface-hub <2.0,>=0.16.4

Used by

Popular packages that depend on tokenizers.

transformersPyPI

Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

↓ 0/wk◈ 0 dependents⚖ Apache 2.0 License

Relationship graph

Dependencies (left) and dependents (right) of tokenizers.

huggingface-hubtransformerstokenizers
tokenizers dependencies dependents

Frequently installed together

numpyPyPI

Fundamental package for array computing in Python

↓ 0/wk◈ 6 dependents
tqdmPyPI

Fast, Extensible Progress Meter

↓ 0/wk◈ 2 dependents⚖ MPL-2.0 AND MIT
packagingPyPI

Core utilities for Python packages

↓ 0/wk◈ 7 dependents
huggingface-hubPyPI

Client library to download and publish models, datasets and other repos on the huggingface.co hub

↓ 0/wk◈ 2 dependents⚖ Apache-2.0
regexPyPI

Alternative regular expression module, to replace re.

↓ 0/wk◈ 1 dependents
typerPyPI

Typer, build great CLIs. Easy to code. Based on Python type hints.

↓ 0/wk◈ 1 dependents
safetensorsPyPI

No description provided.

↓ 0/wk◈ 1 dependents

Release history

15 releases in the last two years, typically about 21 days apart.

Recent versions of tokenizers
VersionPublishedSizeLicense
0.23.2latest6 days ago3.0 MB—
0.23.14 months ago3.0 MB—
0.23.1rc04 months ago3.0 MB—
0.23.0rc04 months ago3.0 MB—
0.22.28 months ago3.0 MB—
0.22.2rc09 months ago3.0 MB—
0.22.111 months ago2.9 MB—
0.22.1rc011 months ago2.9 MB—
0.22.01 year ago2.9 MB—
0.22.0rc01 year ago3.1 MB—
0.21.41 year ago2.7 MB—
0.21.21 year ago2.7 MB—
0.21.2rc01 year ago2.7 MB—
0.21.11 year ago2.7 MB—
0.21.1rc01 year ago2.7 MB—

Overview

tokenizers is an PyPI package. It has 1 known dependents in the graph. The latest version is 0.23.2.

Who should use it

Teams working in the PyPI ecosystem who need this functionality and value a focused solution.

When not to use it

Consider an alternative if you need to or if a more actively-maintained option better fits your PyPI stack.

Pros

  • Lean dependency tree (1 direct dependencies).

Cons

  • No license detected — review usage terms before adopting.

Auto-generated from collected registry metadata. No external claims are inferred.

Keywords

BPENLPdeep learningtokenizertransformer

Frequently asked questions

Should I use tokenizers?

No — tokenizers should not be adopted for new projects. No license declared — you have no explicit right to use this code.

How do I install tokenizers?

Run `pip install tokenizers` to add tokenizers to your PyPI project.

How many dependencies does tokenizers have?

tokenizers declares 1 direct dependency and pulls in 18 packages in total once its runtime tree is resolved totalling about 8.8 MB on disk.

What license is tokenizers released under?

No license was detected in tokenizers's metadata; review the repository before use.

How popular is tokenizers?

PyPI does not publish download counts. Within PackageGraph's crawl, 1 known packages depend on tokenizers.

Is tokenizers still maintained?

The most recent release of tokenizers (v0.23.2) was published 6 days ago.