DemosPaper in preparation
Restated figures
Companies restate figures in later filings. When a claim quotes the earlier figure, was it right? That depends on when it was made. This study asks whether language models know that.
One restated figure, twelve questions
Design · draft datasetThe figure
Western Digital Corp, revenue from contracts with customers, fiscal year ended 30 Jun 2023
Choose a cell to see exactly what the model is asked.
Each cell will show each model’s answer once the runs are complete and checked. The example comes from the draft dataset; gold labels are computed by rule from the filings shown in each cell.
How one figure becomes a judged question
Four figures, each drawn in. Click a step, or let it loop on its own.
- The figure moves between filings
A company reports a figure, then recasts it in a later annual filing. Both dates carry the number.
- The claim is dated
A claim that quotes the earlier figure was made against the record as it stood on its own date - not against today’s filing.
- Twelve ways of asking
Each figure is crossed with the date of the claim in controlled variants, so the question can be phrased with, or without, the date attached.
- The model judges each one
Each variant is judged against the record of the claim’s date. Model runs pending; each figure will show every answer once the runs are complete and checked.
What it is, and how it works
What it is
Each of 300 restated figures, from 86 companies, is asked in 12 versions crossing the figure claimed with the date of the claim. Once results are in, each cell will show what each model answered.
Models
Nine frontier models, a reasoning model and a prompted-rationale condition.
Design
- Five hypotheses, set out in the analysis plan
- Gold labels balanced within each figure
- A wild cluster bootstrap checked by simulation before any model run
Build log
- Design revised to twelve cells per figure, crossing the claimed value with the claim's date in four formats.
- What went wrongA probe of extended-thinking configurations failed its manipulation check, so the thinking arms were removed before any main run.
- The wild cluster bootstrap was calibrated by simulation before any model run; it holds its false-positive rate where the company bootstrap ran high.
Status and limits
Design and analysis plan complete; model runs pending.
Design only. The restatements are recasts in later annual filings, not amended filings, and the analysis plan is not yet frozen. No model answer is shown here until the runs are complete and checked.
Next demo