DemosFinancial claim verification
Fool the verifier
Verifiers trained on FEVER look accurate on financial text: the best scores 98.8% on claims copied from the evidence. Change one number or name, and it catches the change a third of the time or less.
Edit the claim: the paper’s word-overlap baseline scores it as you type
Loading the benchmark…Evidence, from a 10-K filing
Claim words as the baseline sees them:
The benchmark: accuracy by edit type, then every pair with each verifier’s verdict
Released predictions| Verifier | Verbatim | Number changed | Negated | Entity swapped | Evidence mismatched |
|---|---|---|---|---|---|
| DistilBERT, full FEVER | 98.8% | 32.6% | 28.0% | 2.6% | 41.8% |
| DistilBERT, 40k subset | 98.2% | 26.5% | 30.6% | 10.5% | 44.9% |
| BiLSTM | 6.0% | 1.7% | 7.0% | 5.3% | 89.5% |
| XGBoost | 41.4% | 10.9% | 43.2% | 7.9% | 36.8% |
| Random forest | 8.1% | 0.9% | 2.2% | 0.0% | 89.3% |
| Logistic regression | 2.4% | 5.2% | 11.3% | 5.3% | 94.3% |
| Linear SVM | 4.0% | 5.2% | 9.6% | 2.6% | 95.3% |
| Naive Bayes | 1.4% | 0.0% | 0.9% | 0.0% | 99.8% |
| Voting ensemble | 1.2% | 2.2% | 10.9% | 0.0% | 96.2% |
| Voting ensemble, calibrated SVM | 2.0% | 2.6% | 10.0% | 0.0% | 96.0% |
| Word-overlap baseline | 100.0% | 0.4% | 0.0% | 2.6% | 100.0% |
| Claim-only check (no evidence) | 18.5% | 10.9% | 47.6% | 10.5% | 69.3% |
The claim-only check never sees the evidence; it is there to measure annotation artifacts, not as a verifier.
Try this
Three steps- Read a pair
Start from a released pair: a 10-K filing as the evidence, and a claim built from it.
- Make one edit
Change a number, negate it, or swap the entity out - one controlled edit at a time.
- Score it, then compare
Watch the word-overlap baseline call it as you type, then check the released verifiers on every pair below.
What it is, and how it works
What it is
A benchmark of 1,489 claim and evidence pairs built from SEC 10-K filings with documented single edits, used to test 11 verifiers under one protocol.
My role
Sole author: the benchmark, the generation rules, the evaluation harness and the analysis.
What it found
98.8% on verbatim claims, 33% or less on single-token edits, 2.6% on entity swaps. On real financial claims, no learned model beat a tuned word-overlap baseline.
Build log
- Camera-ready: the repository synced with the paper, so every reported figure reproduces from it.
- What went wrongA reproduction check found a rounding drift (0.368 in the paper, 0.367 from one script) and analysis scripts with hard-coded paths. Paths now resolve through one config file and a dependency is pinned.
- This demo: the browser re-implements the paper's baseline exactly and is checked against its released predictions on all 1,489 pairs.
Status and limits
Completed. Benchmark, code and pinned harness released; every reported figure reproduces from the repository.
The edits are synthetic and single-token by design, so the benchmark measures whether a verifier checks the evidence, not how it handles natural misstatements. This page runs the word-overlap baseline in your browser and shows the other verifiers' recorded predictions; it does not run the trained models yet.
The one-sentence negative result
On real financial claims (Fin-Fact), no FEVER-trained model beats a tuned lexical-overlap heuristic (macro-F1 0.368), despite in-domain FEVER scores up to 0.841; and the strongest model, which verifies verbatim financial statements at 98.8% accuracy, detects single-token falsifications at 33% or below.
Abstract (camera-ready)
Realistic financial misinformation is typically a single-token edit of a true statement, yet no public financial dataset supports per-edit statements about which factual changes a verification system can detect. We introduce a controlled benchmark of 1,489 claim–evidence pairs derived from SEC 10-K filings via documented FactCC-style perturbations (number changes, negation flips, entity swaps, evidence mismatches), each pair tagged with its generation rule, and use it to evaluate eleven models spanning three technological generations (TF-IDF linear classifiers, tree ensembles, and neural models including a fine-tuned transformer), all trained on FEVER under one protocol. Per-edit attribution is our central result: the strongest model verifies verbatim financial claims at 98.8% accuracy but detects single-token falsifications at 33% accuracy or below, with number changes and negation flips detected at similar rates and entity swaps far the hardest (2.6%, on 38 pairs). Two diagnoses follow. A hypothesis-only probe shows the classical tier's in-domain competence is claim-surface artifact (a claim-only classifier reaches macro-F1 0.556 against 0.549 for full claim+evidence models), while our benchmark shows no such artifact under the same probe (0.358). And transfer to real financial claims collapses: on Fin-Fact, no learned model beats a tuned lexical-overlap heuristic (macro-F1 0.368), despite in-domain scores up to 0.841. We release the benchmark, generation code, and evaluation harness.
Next demo