Suprabhat Rapolu
ResearchDemos
Checked against which record?The record movesFool the verifierRestated figuresErukaARESIndia record versions
BuildingAbout and CV

DemosClaims explorer

Checked against which record?

Each published claim about an official statistic, beside the record as it stood that day and as it stands now. If you think a verdict is wrong, contest it.

LiveFrom my audit of 4,669 claims on 19 US statistical releases

Try it: choose a claim, move through the record

Loading claims…

Start here: twenty claims picked to show the range, flipped ones first

    Loading the audit data

    Try this

    Three steps
    1. Pick a claim

      Start from the twenty selected, or search by text, release, verdict or series.

    2. Move through the record

      Step the claim through every vintage of the record since it was published.

    3. Read the verdict, then contest it

      See when it held and when the revision changed the verdict. Disagree? Contest it on the grounds.

    The project in depth

    What it is, and how it works

    What it is

    An audit of 4,669 claims from 334 news articles about 19 US statistical releases, each checked against the version of the record current on the day it was published.

    My role

    Sole researcher: the question, the pipeline, the analysis, and this interface.

    What it found

    • 17.8% of claims supported by the record when published are contradicted by the same record today, rising to 43.3% for monthly payroll. Source authority did not predict groundedness.
    • Outlet tier does not predict this: national and wire outlets 18.3%, aggregator and regional 17.3%. The preregistered source-authority hypothesis is not supported.

    How it works

    1. Extract claims
    2. Link each to a FRED series
    3. Find the ALFRED version current on the publication date
    4. Verdict then and now
    5. Hand verification

    Build log

    • First end-to-end pipeline run on the August jobs release, with a rule-based extractor.
    • What went wrongThe rule-based extractor produced schema-invalid claims (28 of 161 in the first batch). Extraction moved to a pinned model session with hashed prompts and few-shot examples; every batch is logged with its hashes.
    • What went wrongArticles time-stamped before a release that nonetheless reported it were being checked against the wrong vintage. A rule now moves their as-of date to the release moment.
    • Verdict rules frozen at v1.6 and run across all 19 releases; the paper's numbers are asserted by a check script.
    • This explorer: the verdict at every vintage between publication and today, re-run through the audit's own linking and verdict code. It reproduces the published verdicts for every checkable claim.

    Status and limits

    Analysis complete. The research note is under review at HKS Misinformation Review, and an extended abstract at NTTS 2027.

    Verdicts are produced by a pipeline, not by hand. It was validated by blind model adjudicators and by one human coder, the author, on a stratified subset; no second human annotator was recruited. A claim marked unverifiable means the record could not decide it, not that it is false. Outlets appear by tier only, and quotes are cut to 25 words. Contests are kept in your browser for now; they are not yet sent anywhere.

    What this does to published claims

    The tables on the record page are about the series. This is about the sentences. Across 19 US macroeconomic releases and 334 news documents, 1,862 claims were supported by the record on the day they were published. 332 of them, 17.8%, are contradicted by the record as it stands today. Nothing about the sentences changed.

    Release Supported when published Later contradicted Rate
    Employment Situation 918 304 33.1%
    … payroll-change claims only 637 276 43.3%
    CPI 781 0 0.0%

    The split is not about journalism. It follows the agency's revision schedule. Checking a payroll claim against today's data, rather than against the data that existed when it was written, mislabels correct reporting; checking a CPI claim that way does not.

    What the audit gets wrong

    The pipeline that produced those numbers was itself audited, and it is worth stating plainly what it cannot do. Every contradiction it flagged was read by independent adjudicators; 78% were the pipeline misreading the claim rather than an outlet misstating the record. After six rounds of repair the rules were frozen and tested on two releases they had never seen, with every checkable claim adjudicated blind: the contradiction verdict was right 3 times out of 18. That verdict is not a measuring instrument, and no contradiction rate is reported from it without that caveat.

    The supported verdict is a different matter: on those same unseen releases it agreed with blind adjudicators 142 times out of 153. The finding above rests on that verdict, a claim the record supported when published and contradicts now, which is why it is reported and the contradiction rate is not.

    Can a model check these claims?

    Can a small model check these claims if it gets the right version of the record? See the OE-Agent paper.

    Bar chart: share of claims the record supported when published that the latest vintage contradicts, by release and tier. Employment Situation 33% (T1) and 34% (T2); GDP advance 16% and 20%; CPI 0% and 0%.
    Right when published, wrong now. Whiskers are 95% bootstrap intervals resampling documents, not claims.
    Bar chart: precision of the pipeline's contradiction verdict. 0.38 and 0.37 on the claims the rules were tuned against; 0.40 on a held-out CPI release; 0.08 on a held-out Employment Situation release.
    What validating on your own tuning data is worth: precision of the contradiction verdict on tuning rounds and on releases the rules had never seen.
    Research note, under reviewWhat comes nextAll demos

    Next demo

    The record moves