Methodology

How the Hype Gap score is computed, end to end — the pipeline, the formula, the terminology, and why each piece works the way it does.

The pipeline

Step 1
Ingest
Paste an arXiv or bioRxiv link, or upload a PDF directly. An uploaded PDF's raw bytes are handed straight to Gemini's native vision — the same way a person reads a page — instead of first running through OCR or a text-extraction library. Scanned pages, multi-column layouts, and figures reach the model as-is, not garbled by a lossy conversion step before it ever reads a word.
Step 2
Extract
Three independent, parallel AI calls read the paper — one pulls hype claims, one pulls reported statistics, one checks for self-justification. Kept independent on purpose, so one call's read can't bias another's. Every extracted quote is required to be an exact, verbatim substring of the source text — never a paraphrase or summary.
Step 3
Validate
Each call's output is checked against a locked schema before anything downstream sees it, so the formula only ever runs on well-formed, verified data — never raw, unvalidated model output.
Step 4
Score
A fixed, pure-Python formula — no LLM call, no model judgment — turns the validated extractions into a Hype score (0–10), an Evidence score (0–10), and the Gap between them (−10 to +10).
Step 5
Show the receipts
The report displays every quote that fed the score, right next to the number it produced, plus the exact arithmetic that produced it.

The formula, in full

Every number below is public, fixed, and computed in plain Python — nothing about the score itself comes from a model.

Hype Score (H, 0–10)

max_claim_severity = clamped_intensity (1–5) × claim_type_weight

pervasiveness = 0.5 × ln(1 + claim_count)

H = min(10, max_claim_severity + pervasiveness)

Evidence Score (E_raw, 0–10)

0.0 — if no statistics were reported at all

5.0 (neutral) — if statistics exist but none use a calibratable effect size

min(band score across calibratable stats) — the weakest reported statistic, if one or more exist

Evidence, adjusted (E_adjusted)

E_adjusted = E_raw × 1.25 — if the paper justifies its own effect size

E_adjusted = E_raw — otherwise

Hype Gap

Gap = clamp(H − E_adjusted, −10, +10)

Why it's built this way

The worst claim wins, not the sum. Hype severity is set by the single most severe claim, not a total across the abstract — a long, hedge-heavy paper can't out-score one severe overclaim just by making more claims.
The weakest statistic sets the evidence score. Evidence aggregates conservatively — the lowest-scoring calibratable statistic anchors the score, so a paper can't cherry-pick its single best number while quietly reporting weaker ones alongside it.
No stats and weak-fit stats are scored differently, on purpose. Reporting zero statistics floors Evidence at 0. Reporting statistics that don't fit a calibratable effect-size format defaults to a neutral 5 — so fields that report bare accuracy numbers instead of classical effect sizes (common in ML) aren't penalized for a format mismatch that isn't actually weak evidence.
Self-justification can only help, never hurt. A paper that explicitly justifies its own effect size gets a 1.25× evidence bonus; not finding one costs nothing.
Claim types are weighted by how falsifiable the overreach is. A causal claim or an unqualified generalization is weighted higher than a superlative or novelty claim, since those are the more concretely checkable kinds of overreach.

A worked example

Real numbers, from the same fixture used in the backend's own test suite: one superlative claim at intensity 3, one Cohen's d statistic of d = 0.9, and no self-justification found.

max_claim_severity = intensity 3 × superlative weight 1.0 = 3.0

pervasiveness = 0.5 × ln(1 + 1 claim) ≈ 0.35

H = min(10, 3.0 + 0.35) ≈ 3.35

d = 0.9 is at/above cohens_d's top band threshold (0.8) → 10.0

E_raw = 10.0, E_adjusted = 10.0 (no self-justification found, unchanged)

Gap = clamp(3.35 − 10.0, −10, +10) ≈ −6.65 → "Underhyped"

Terminology

Claim types & weights

Superlativeweight 1.0
Novelty claimweight 1.0
Generalizationweight 1.2
Certainty overstatementweight 1.3
Significance inflationweight 1.4
Causal overreachweight 1.5

Effect types & bands

Cohen's d0.2 / 0.5 / 0.8 — Cohen's own small / medium / large benchmarks
Correlation (r)0.1 / 0.3 / 0.5 — small / medium / large
Odds ratio1.5 / 2.5 / 4.3
Hazard ratio1.5 / 2.5 / 4.3
Percentage point3 / 10 / 20 percentage points

Statistics reported as raw_count or other aren't banded — they route to the neutral 5.0 default instead.

Self-justification types

Baseline comparison
Practical threshold
Theoretical framing

Verdict bands

Gap < 0Underhyped
0 ≤ Gap < 2Honest
2 ≤ Gap < 6Reasonable
Gap ≥ 6Overhyped

Why the number is trustworthy

Deterministic. The extraction model runs at a fixed temperature and seed, so the same paper always produces the same score — no run-to-run drift.

No hidden judgment call. The AI only extracts quotes. It never scores anything — the fixed formula above does that, the same way every time.

Verifiable. Every claim and statistic behind a score is a verbatim quote from the paper itself, shown alongside the number it produced — not a summary you have to take on faith.

Schema-gated. Extracted data is validated against a locked schema before the formula ever touches it — the score never runs on malformed or unvalidated model output.

Continuously checked for quoting accuracy. The test suite re-runs the pipeline against a fixed set of real papers and asserts that every extracted quote is an exact substring of that paper's text — not just a stated policy, but something that fails the build if it's ever violated.

Sourced honestly. That test corpus is five real, open-access arXiv papers, fetched programmatically from arXiv's own API — never hand-transcribed, so there's no chance of a misquote or cherry-picked wording sneaking into the fixtures the pipeline is tested against.

The scoring bands are principled defaults, and we're still tuning them against more real papers — so read the score as a consistent, transparent estimate, not a certified benchmark.