Pricing

The Tokenizer Tax: Why A.I. Costs More in Many Languages Than in English

co-produced by AI27. October 2026, 17:00

Same content, different price: Anyone who uses ChatGPT, Claude or Gemini in a language other than English often pays extra. The culprit is the so-called tokenizer tax, which a new study by the Coalition for Language Equity in AI Research (CLEAR) has now measured systematically. Depending on the model and the language, the same text can require several times as many tokens, and tokens are the unit A.I. providers bill by.

What Is the Tokenizer Tax?

A.I. models do not read words. They break text into tokens, or word fragments. How efficiently that works depends on which languages the so-called tokenizer saw frequently during training. English is usually split into whole words, while rarer languages and scripts are broken into many small fragments.

CLEAR illustrates the consequences with a sample sentence. The English sentence “People should benefit from AI, regardless of the language they speak” requires 13 tokens in GPT-5’s tokenizer, according to the study. The same statement in Malayalam requires 26 tokens, and in Amharic 78. An input in Amharic is therefore six times as expensive as one in English. Sentence length has little to do with it: Differences in text length between languages amount to tens of percent, according to CLEAR, while the token gaps are measured in multiples.

How the A.I. Models Rank

For the study, CLEAR used the FLORES-200 benchmark, a set of professionally translated sentences in about 200 languages, and compared the tokenizers of all the major A.I. labs. The key findings:

Top 20 languages: For the 20 languages with the biggest presence online, the models are almost fair. The most expensive language needs 1.5 to 3.1 times as many tokens as the cheapest.
200 languages: Once all languages are included, the gap balloons to as much as 17 times.
Top grade: Google’s Gemini and Gemma family performs best, with a ratio of 5.1 and a grade of B. No general-purpose frontier model does better.
Middle of the pack: Anthropic’s Claude comes in at 9.3 times and Meta’s Llama 4 at 9.5 times, both with a grade of D.
Failing grades: OpenAI’s GPT-5 (13.8 times), Mistral’s Ministral 3, Nvidia’s Nemotron 3.5 and Microsoft’s Phi-4 (about 15 times each) receive an F. Falcon 3 from Abu Dhabi comes last at 17 times.

A second measure paints a different picture, however: Counting the share of languages for which a model needs fewer than twice as many tokens as for its most efficient language, GPT-5 leads with 78 percent. Newer models are also not automatically better. Alibaba’s new Qwen 3.6 scores worse than the older Qwen 3.

And in German?

German is a Latin-script language with a large online presence and therefore gets off relatively lightly. It is not free, though. In a quick test by Trending Topics with three sample sentences, the German version required between 15 and 42 percent more tokens than the English one in OpenAI’s publicly available o200k tokenizer. Long compound words and gender-inclusive forms add noticeably to the count.

Languages with their own scripts pay the most. According to CLEAR, the tax also hits huge language groups with hundreds of millions of speakers, such as Bengali, Urdu and Punjabi. Beyond cost, speed and context are at stake: More tokens mean slower responses, and a 128,000-token context window holds only a fraction of the text in a heavily taxed language, which makes long documents and A.I. agents harder to use. A start-up in Bangalore pays a surcharge on every prompt that its competitor in Austin does not, CLEAR writes.

What CLEAR Wants

The researchers propose three steps for A.I. labs:

Transparency: Token inequality across all languages should be listed as a standard score in model cards.
Better tokenizers: The next generation of models should deliberately cover more languages efficiently.
Fair pricing: Because tokenizers can only change with new models, providers should bill by character or language instead of by token right away.

Public buyers should also require A.I. vendors to disclose how expensive a model is in different languages.

A Problem That Will Fix Itself?

Not everyone considers the problem dramatic. Dheemanth Reddy of the Indian start-up Maya Research told Rest of World that token prices are currently falling very quickly, which shrinks the extra costs in absolute terms. “Perfect equality may be unrealistic,” said Pratyush Choudhury of the investor Activate AI, who suggested treating language efficiency as a product requirement instead. CLEAR itself concedes that a one-to-one ratio is probably unattainable because languages differ in length.