Token

Tokenisers are built by learning which byte sequences occur together often enough to deserve their own vocabulary entry. Common English words usually become one token, rarer words split into several, and whitespace and punctuation attach in ways that are not obvious by inspection. The same sentence in a language under-represented in the tokeniser’s training data can cost noticeably more tokens than its English equivalent — a real and often overlooked cost and context asymmetry.

Because the model sees tokens rather than characters, a whole class of tasks is harder than it looks. Counting letters in a word, reversing a string or reasoning about precise character positions all ask the model to inspect inside units it does not natively see. This is not a reasoning failure; it is a representation mismatch, and the fix is to do that work in code and hand the model the result.

For estimation, treat any character-to-token ratio as a rough heuristic that varies by language, formatting and code content, and measure with the actual tokeniser when the number matters. JSON, deeply indented code and repeated markup are all more token-expensive than their information content suggests — which is a good argument for compact tool results.