Transformers & attention
Publication dates and source age
Sources counted: 4
Newest dated source: 2025-07-31
At least one source was published within the 18-month window.
Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.
Some publication dates are unknown; the newest dated source may not be the newest source overall.
Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.
Open in the glossary Reading notes · Structured record
In plain language
A neural-network design that uses attention to combine information from different parts of its input. [1]
Reference this explanation or suggest a correction
Link to this explanation · Suggest a correction · How corrections work
Limits & distinctions
'Attention' is the name of a mathematical operation. An attention diagram alone does not provide a complete explanation of why a model gave an answer. [1] [3] [4]
Reference this explanation or suggest a correction
Link to this explanation · Suggest a correction · How corrections work
A fuller explanation
Attention is a calculation that gives different amounts of influence to different pieces of information. In a Transformer, layers of these calculations help build representations of text in context. The original Transformer paper introduced the design for tasks including translation. [1]
Reference this explanation or suggest a correction
Link to this explanation · Suggest a correction · How corrections work
How it relates to the map
Useful background for reading explanations of LLM architecture: how a model is arranged. [2]
Reference this explanation or suggest a correction
Link to this explanation · Suggest a correction · How corrections work
Share this page
https://theaiatlas.org/ideas/transformer/
Download a share image · Vector image
Image previews are summaries. Keep the page link so readers can check the evidence.
Sources and what we read
1. Attention Is All You Need
Source is over 18 months old.
Publication dates and source age
Sources counted: 1
Newest dated source: 2017-06-12
Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.
Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.
Read abstract, model architecture, learned embeddings and next-token probabilities in the HTML paper; publication and revision dates checked against the arXiv abstract page. This is the original Transformer architecture, not a claim that every current LLM has its exact structure.
2. LLMs: What's a large language model?
Publication dates and source age
Sources counted: 1
Publication dates are unavailable.
Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.
Some publication dates are unknown; the newest dated source may not be the newest source overall.
Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.
Read token prediction, encoder-only and decoder-only variants, and self-attention. Used for architecture and terminology; broad performance comparisons and claims about all LLMs on the teaching page are not adopted.
3. Circuit Tracing: Revealing Computational Graphs in Language Models
Publication dates and source age
Sources counted: 1
Newest dated source: 2025-03-27
At least one source was published within the 18-month window.
Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.
Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.
Read introduction, method overview and limitations including reconstruction errors, graph complexity, global circuits and mechanistic faithfulness. The authors' replacement-model analyses reveal selected mechanisms; they do not provide a complete explanation of all behavior. Later attention-tracing work is cited alongside this paper to avoid treating its missing-attention limitation as a permanent field-wide result.
4. Tracing Attention Computation Through Feature Interactions
Publication dates and source age
Sources counted: 1
Newest dated source: 2025-07-31
At least one source was published within the 18-month window.
Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.
Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.
Direct browser-tool retrieval failed; fetched the original publisher HTML successfully and read the introduction, case-study summaries, QK-attribution method, inhibitory-effect limitation and graph-construction tradeoffs. Extends earlier attribution graphs to attention; results are selected studies with open questions, not a complete model explanation.
Edition and machine-readable evidence
Content version 0.20.0. Evidence cutoff 2026-09-15; this does not mean every source was read on that day.
Pinned complete dataset · Complete evidence page · Agent consumption guide