Claude Tokenizer
sanderland · Kit
Shows how Claude, OpenAI and Hugging Face tokenizers split your text, with token counts and characters per token side by side. Built for developers comparing t…
New
AI projects from Hugging Face
Categories and filters
Listings
TokenAtlas
prajwalpujari16 · Kit
A browser explorer that compares how many tokens the same text costs across 52 languages and several LLM tokenizers, built on the FLORES-200 benchmark. For peo…
New
Number Tokenization Blog
huggingface · Kit
A static blog page from Hugging Face on number tokenization, the way language models split numbers into pieces. It suits readers curious about how models handl…
New