Pilot packetDecision briefEcosystemKrenn·IQAtlas
Two-week deployment packet · first market: pharma statistical-evidence review

Claim drift measured before it
contaminates scientific datasets.

The pilot packages claim admissibility, deterministic verdicts, human confirmation, batch reporting, and provenance blockers for a technical team and a steering committee.

The pilot is a controlled operating test and certifies no truth. It measures which claims are licensed, which must be rewritten, which must be rejected, and which need more evidence before reuse.
Open Gate App Brief →
Illustrative retrospective

The gate separated retracted from replicated science in a retrospective coded by an agent that knew the outcome.

A retrospective over 28 well-known claims, all published in Nature, Science, or The Lancet: 14 later retracted (Wakefield, Surgisphere, Schön, STAP, Stapel…) and 14 independently replicated (LIGO, Higgs, RECOVERY dexamethasone, the Pfizer vaccine trial…). CAPAS decided by structure.

Illustrative
28/28 separated: 14/14 retracted not accepted, 14/14 replicated accepted, 0 false-accepts, with a coder that knew the outcome
By construction
0/28 separated by plausibility or peer review; all 28 passed peer review by construction of the corpus, so this is no baseline
Scope. Each fraud received its verdict for the structural gap it was retracted for: no controls, no independent reproduction, or unauditable data. The corpus is coded from public retraction records (Retraction Watch, journal notices), so the result validates the gate’s structural logic and not fraud detection from raw paper text. The evidence for the method is the blind corpus (53/82 and 51/63 on X1–3; 17/21 and 16/17 out of sample on X4+X5) and the pilot’s pre-registered gate below.
Delivery path

Five steps, two weeks.

1

Scope

Select corpus, owner, reviewer, and baseline review process.

2

Ingest

Bring paper text, metadata exports, theorem notes, or structured records.

3

Confirm

Human reviewer approves candidate spans before CAPAS decides.

4

Gate

Run single and batch CAPAS decisions with explicit evidence contracts.

5

Read out

Deliver decision mix, exception queue, provenance blockers, and ROI model.

Two-week operating plan

Week by week.

Week 1: Calibrate the gate
Select a controlled corpus of candidate scientific claims.
Map source material to claim categories and evidence fields.
Establish baseline reviewer workflow and decision taxonomy.
Confirm sensitive-data handling, licensing constraints, and export boundaries.
Week 2: Run, adjudicate, read out
Run single and batch gate decisions on the selected records.
Review HOLD, REWRITE, REJECT, and provenance-blocked exceptions.
Measure decision mix, reviewer agreement, and evidence gaps.
Deliver executive summary, audit packet, exception queue, and deployment backlog.
Deliverables

Six deliverables at the close of week two.

📊

Audit export

CSV/JSON with verdict, reason, schema version, record ID, evidence fields, and provenance blockers.

🔀

Exception queue

HOLD/REWRITE/REJECT records routed for expert adjudication and wording repair.

🔍

Provenance report

Open review hashes, source URL hashes, witness IDs, reviewer attestations, and RO-Crate blockers.

📈

Decision mix

ACCEPT, REWRITE, REJECT, HOLD, fine-tune-ready, and provenance-blocked counts by corpus slice.

⚠️

Claim contamination register

Examples where source evidence supported a weaker claim than the candidate record asserted.

📋

Executive readout

What can be reused, what needs rewrite, what cannot be reused, and the next operational control.

Roles and decision rights

Ownership by role.

AI Governance Lead
Owns risk framing, success criteria, and decision policy for controlled reuse.
Research Ops / Data Lead
Owns corpus selection, source access, metadata hygiene, and export workflow.
Subject-Matter Reviewer
Confirms evidence spans, adjudicates exceptions, and approves claim rewrites.
Technical Owner
Runs browser/CLI/API surfaces, preserves audit exports, and maps integration backlog.
Legal / Provenance Reviewer
Reviews licensing, confidentiality, source reuse, attestation, and sensitive-payload handling.
Executive Sponsor
Receives decision mix, contamination examples, and go/no-go recommendation for rollout.
Success criteria

Evidence that the pilot worked.

Operational success
✓Every pilot record has a traceable decision and source context.
✓HOLD/REWRITE/REJECT records produce an actionable exception queue.
✓Reviewers can explain why accepted claims are licensed for controlled reuse.
✓Exported audit artifacts are complete enough for governance review.
Commercial success
✓The buyer sees a measurable claim contamination rate on its own corpus.
✓The team identifies repeatable provenance blockers and schema gaps.
✓The pilot produces a next-step integration backlog.
✓The sponsor can decide whether to expand to CLI/API or workflow integration.
Pre-registered go / no-go (revised 2026-10-13; fixed with the sponsor before day 1)

The criteria above describe a useful pilot. The gate itself is fixed before any record is seen:

  • False-accept ≤ 30% on the bad records and false-reject ≤ 25% on the good ones, with at least 40 bad and 40 good adjudicated, Wilson 95% intervals published. Otherwise no-go. The threshold comes from the pooled blind rate: the gate accepted 33 of 103 bad claims (32%: 29/82 in X1–3, 4/21 in X4+X5) and rejected 13 of 80 good ones (16%). Power with 40 bad records: at the X4+X5 rate (19%) the gate passes 97% of pilots; at the pooled rate (32%) it passes 47%. A 20% threshold would have passed 7% of the time.
  • Third criterion, against an engine that never accepts: at least 40% of the good records end in ACCEPT, and HOLD is at most 30% of all records. Without it, an engine that returned HOLD or REWRITE on every record would score 0% false-accept and 0% false-reject and pass. Measured blind: 53 of 80 good records ACCEPTed (66%) and 6 HOLD in 183 records (3%); source outputs/ginibre_sealed_scores.json.
  • Source of bad records: the sponsor’s own history, records its reviewers later corrected or rejected. Nothing is seeded. If the first 100 hold fewer than 40 bad records, records are added until 40, or the go/no-go is deferred and the interval is published as is.
  • Mandatory blind coding arm: two coders with no access to the outcome convert the records into payloads; κ ≥ 0.70 per field. Below that threshold the pilot measures the coders, and it stops.
  • Engine frozen at a git tag before coding; verdicts sealed by hash before adjudications are opened. The same protocol as the published blind corpus.

The thresholds are the CAPAS team’s proposal, negotiated with the sponsor and frozen once fixed. The blind data to date are the research-corpus figures: 53/82 and 51/63 on X1–3, and 17/21 and 16/17 out of sample on X4+X5.

Measurement protocol
  • Two samples, two questions. An enriched sample, of records with known problems, measures how the gate responds to errors. A consecutive sample of real work estimates prevalence, workload and cost. Neither sample answers the other’s question.
  • Reporting. Errors are reported by severity. Time is measured for the whole process, including evidence preparation. HOLD resolution is tracked to closure. The sponsor’s adjudicators set the reference truth, never the gate’s own rules.
Governance model

Boundaries of the engine.

Supplied evidence only

The engine does not infer hidden evidence, perform broad paper understanding, or certify scientific truth.

Human span confirmation

A reviewer must confirm claim wording and evidence spans before the gate output is treated as operational.

External provenance verification

Active checks for hashes, attestations, witness registries, and provenance packets require CLI/API verification.