A field guide to AI positions

Tokens & tokenization

AI basics.

Publication dates and source age

Sources counted: 3

Publication dates are unavailable.

Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.

Some publication dates are unknown; the newest dated source may not be the newest source overall.

Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.

Open in the glossary Reading notes · Structured record

Prepared with AI assistance. Not independently fact-checked. Original sources and qualifications are below.

In plain language

The pieces of text a language model processes, which may be words, word parts or punctuation. [1]

Reference this explanation or suggest a correction

Explanation based on the linked sources. Claim ID: claim-term-token-1a5192ba3d7825ca26b24a72. Edition 0.20.0.

Link to this explanation · Suggest a correction · How corrections work

Limits & distinctions

A token is not a fixed amount of English text. Avoid treating a token limit as an exact word or page count. [1]

Reference this explanation or suggest a correction

Explanation based on the linked sources. Claim ID: claim-term-token-09422ce4d5c74cb753a9bb98. Edition 0.20.0.

Link to this explanation · Suggest a correction · How corrections work

A fuller explanation

A tokenizer splits text into pieces and assigns each piece a number the model can use. Different tokenizers split the same text differently. Converting the output numbers back into readable text is called decoding. [1]

Reference this explanation or suggest a correction

Explanation based on the linked sources. Claim ID: claim-term-token-1e4c26398ee834b2e16dc7b8. Edition 0.20.0.

Link to this explanation · Suggest a correction · How corrections work

How it relates to the map

This helps make sense of input limits and answer lengths stated in tokens. [2] [3]

Reference this explanation or suggest a correction

Editorial interpretation or qualification. Claim ID: claim-term-token-25f52a21ad9f2e54a62ed1b2. Edition 0.20.0.

Link to this explanation · Suggest a correction · How corrections work

Share this page

https://theaiatlas.org/ideas/token/

Download a share image · Vector image

Image previews are summaries. Keep the page link so readers can check the evidence.

Sources and what we read

  1. 1. Tokenizers

    Hugging Face LLM Course · First-hand source · Published date unavailable · Material last read 2026-09-15

    Publication dates and source age

    Sources counted: 1

    Publication dates are unavailable.

    Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.

    Some publication dates are unknown; the newest dated source may not be the newest source overall.

    Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.

    Read scope is described in the source note.

    No archive check recorded.

    Read word, character and subword tokenization, encoding to numerical IDs and decoding. Used for the fact that token boundaries depend on the tokenizer; no fixed words-to-tokens conversion is assumed. Live page publication date unspecified.

    Source ID: hf-tokenizers

  2. 2. Text generation

    Hugging Face Transformers documentation · First-hand source · Published date unavailable · Material last read 2026-09-15

    Publication dates and source age

    Sources counted: 1

    Publication dates are unavailable.

    Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.

    Some publication dates are unknown; the newest dated source may not be the newest source overall.

    Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.

    Read scope is described in the source note.

    No archive check recorded.

    Read next-token generation, generation settings, temperature, sampling and prompt-format sections. Library options illustrate the process; defaults and suggested temperatures are not treated as universal chatbot behavior. Live page publication date unspecified.

    Source ID: hf-text-generation

  3. 3. Machine Learning Glossary

    Google for Developers · First-hand source · Published date unavailable · Updated 2026-04-10 · Material last read 2026-09-15

    Publication dates and source age

    Sources counted: 1

    Publication dates are unavailable.

    Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.

    Some publication dates are unknown; the newest dated source may not be the newest source overall.

    Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.

    Read scope is described in the source note.

    No archive check recorded.

    Read the artificial intelligence, deep model, context window, inference and chat entries. Used for terminology, not product performance claims. Updated date follows the earlier displayed page date; initial publication is unspecified. The chat entry was reread during the same-day beginner-content review. Also read the generalization and compute entries.

    Source ID: google-ml-glossary

Edition and machine-readable evidence

Content version 0.20.0. Evidence cutoff 2026-09-15; this does not mean every source was read on that day.

Pinned complete dataset · Complete evidence page · Agent consumption guide