The Indic Tokenizer Tax

Before an AI model reads your text, a tokenizer chops it into pieces called tokens. More tokens means higher cost, slower answers, and less text fits in the model's memory. This page measures how many more tokens 13 tokenizers need for 22 Indian languages than for the same sentences in English. 1.0× = same as English.

Pick a language

Orange = Indian-built tokenizer. Lower is better.

All 22 languages

Tokens needed relative to English (colour clipped at 10×). Hover for exact values.

Try your own sentence

Runs in your browser (Param2 is chart-only: its tokenizer uses a regex browsers don't support). The first run downloads each tokenizer (a few MB), so give it a moment.

Method

Data: AI4Bharat IN22-Gen: 1,024 sentences, each translated into all 22 scheduled Indian languages plus English.
Metric: parity = total tokens for a language ÷ total tokens for English, over the same sentences.
What it isn't: parity measures cost and efficiency, not model quality.
Prior work: Indic tokenizer comparisons already exist (arXiv 2411.12240, 2607.24276). This page adds the 2026 Indian tokenizers (Sarvam-30B, Param-1-7B, Param2-17B).
Raw data: results.csv