Tokens: how AI reads text

Large language models (LLMs), the engines behind today's chat assistants, don't read letters or words the way we do. They read tokens: chunks of text that can be a whole word, part of a word, a space plus a word, a number or a single character.