Benchmark methodology

The 1,238 figure comes from one script.

This page defines the grid, the counting rule and the command that reproduces the figures. The number measures verdict-space coverage of the contracts, not a production drift rate.

Scope. The grid is synthetic: it enumerates evidence combinations per claim family to confirm that each contract is deterministic and reaches the verdicts it admits. It does not measure how often real claims drift; no production pilot has run. Field rates require an independently adjudicated corpus.

Headline figures

1,238total decisions
12grid claim families (of 14 types)
78.4%gated (REWRITE + REJECT)
12 / 12families reach ≥2 verdicts

Overall mix: ACCEPT 214 (17.3%) · REWRITE 126 (10.2%) · REJECT 845 (68.3%) · HOLD 53 (4.3%). Most evidence combinations are under-supported by construction, so a high REJECT share is expected; it does not describe real corpora.

Decision mix per claim family

Claim familyNACCEPTREWRITEREJECTHOLDverdicts
statistical_confidence543747439234 / 4
financial_metric_claim364292930244 / 4
exact_model_solution18210107653 / 4
universal_anchor_claim411522134 / 4
causal_mechanism_claim20131244 / 4
evidence_conflict_claim20111444 / 4
multimodal_evidence_claim2026844 / 4
systematic_review_claim20131244 / 4
programming_language_behavior_claim1613484 / 4
reproducibility_check611224 / 4
claim_transition311013 / 4
physical_accuracy310113 / 4

The grid tracks the current engine: on 2026-10-13 three payloads moved from ACCEPT to HOLD under that day’s rules. The blind rates on the front page belong to the engine frozen at the sealed tag. The current engine has no blind rate until the next sealed batch.

Coverage is nominal, not balanced: 907 of the 1,238 payloads sit in two families, and seven families have n ≤ 20 with hand-set mixes. The grid exercises each contract’s logic. It does not estimate per-family rates.

By contract design, physical_accuracy cannot reach REWRITE and claim_transition cannot reach REJECT. exact_model_solution has REWRITE paths in the engine that the grid does not exercise: a grid gap, not a design limit.

Reproduction

Every figure on this page comes from one script. The same inputs give the same counts on any machine:

$ python3 benchmarks/family_decision_mix.py
# -> outputs/family_decision_mix.json + .md
family_decision_mix: N=1238 across 12 families
overall: {'ACCEPT': 214, 'REWRITE': 126, 'REJECT': 845, 'HOLD': 53}
Provenance · schema capas-claim-payload-v3 · outputs/family_decision_mix.json sha256 fbdc004558b55d16… · checked against that file on every commit by verify_benchmark_page_matches_mix.py · grid generator benchmarks/family_decision_mix.py · regenerated in CI on every commit. The synthetic grid validates contract logic, not fraud detection from paper text or a field drift rate. Production figures follow an independently adjudicated pilot with false-reject and false-accept rates measured against domain experts.