Data contamination / benchmark contamination
Sources are over 18 months old.
Publication dates and source age
Sources counted: 2
Newest dated source: 2023-11-16
Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.
Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.
Open in the glossary Reading notes · Structured record
In plain language
Test material appearing in training data can make an apparently new test partly familiar to a model. [1]
Reference this explanation or suggest a correction
Link to this explanation · Suggest a correction · How corrections work
Limits & distinctions
Overlap does not prove that a model memorized an answer or gained an advantage. Brown and colleagues found that effects varied, and their detection method had limits. [2]
Reference this explanation or suggest a correction
Link to this explanation · Suggest a correction · How corrections work
A fuller explanation
Overlap between training material and test questions or answers can weaken a test of performance on unseen material. Checking that overlap helps interpret a reported score. [1]
Reference this explanation or suggest a correction
Link to this explanation · Suggest a correction · How corrections work
How it relates to the map
Atlas reading question: how did an evaluator check that the test was meaningfully separate from training? [1]
Reference this explanation or suggest a correction
Link to this explanation · Suggest a correction · How corrections work
Share this page
https://theaiatlas.org/ideas/data-contamination/
Download a share image · Vector image
Image previews are summaries. Keep the page link so readers can check the evidence.
Sources and what we read
1. Investigating Data Contamination in Modern Benchmarks for Large Language Models
Source is over 18 months old.
Publication dates and source age
Sources counted: 1
Newest dated source: 2023-11-16
Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.
Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.
Read the abstract and version history. The paper studies retrieval-based overlap checks and a test-slot guessing method. Its reported scores concern particular models and benchmarks; the glossary does not treat a guessed answer alone as proof of training membership.
2. Language Models are Few-Shot Learners
Source is over 18 months old.
Publication dates and source age
Sources counted: 1
Newest dated source: 2020-05-28
Assessed at this edition's evidence cutoff: 2026-09-15. 18-month boundary: 2025-03-15.
Publication age does not tell us whether a claim is still valid. Reading an old source again does not make its publication date newer. An update date does not establish that the passage we used was updated.
Read the abstract, section 2 on zero-shot and few-shot settings, and section 4 on training-data overlap. Version 4 is dated 22 July 2020. The reported results concern GPT-3 and are not current model rankings; detected overlap did not uniformly inflate scores.
Edition and machine-readable evidence
Content version 0.20.0. Evidence cutoff 2026-09-15; this does not mean every source was read on that day.
Pinned complete dataset · Complete evidence page · Agent consumption guide