Back-Test Report: Rheumatoid Arthritis Drug Cohort

1. Executive Summary

The model achieved a pairwise AUC of 0.625 (passing the 0.60 threshold), meaning a randomly chosen eventual success outranks a randomly chosen failure 62.5% of the time. The phase-controlled AUC of 0.65 (target 0.55) confirms the signal holds within decision phase, controlling for the structural advantage later-stage decisions carry. Risk flag sensitivity reached 87.5% (7/8 failures flagged). At the best operating point — a PoS cutoff of 30% — the model achieved 62.5% accuracy with 66.7% precision and 50% recall. Discrimination passes, but calibration is weak at n=16: the separation gap (+8.4pp) and the false-confidence rate at the 25% cut (50%) both fail their targets (see §6). Every PoS multiplier is held to a validation gate: a factor may score the engine only if a held-out cohort with both approvals and failures can confirm it, otherwise it is demoted to a display-only flag rather than moving the number (see §2.4).

2. Methodology

2.1 Core Principle: No Future Information

The back-test simulates the decision an investor or founder would have faced at the time — using only information available at each drug's go/no-go moment. No post-hoc data (trial results, FDA decisions, commercial outcomes) leaks into the inputs. This is not a prediction of the future; it is a reconstruction of the past with the tools available today.

2.2 How the Back-Test Works

2.3 PoS Sources

The back-test uses a two-tier PoS system:

• Computed rates (primary): Drug-level transition rates from the enrichment corpus, time-gated. Used when n>=5 drugs at that phase.
• BIO/QLS 2021 benchmarks (fallback): Static academic rates for immunology. Used for early decision dates (pre-2005) or small samples.

Target validation multiplier:

Time-gated academic multipliers:

2.4 Risk Flags

Six risk flags are evaluated for each drug. Four affect PoS calculations via multiplicative adjustments; two are display-only informational flags.

2.5 Data Sources

Stage costs and durations are based on DiMasi et al. (2016) and Wouters et al. (2020) estimates, adjusted for inflation and phase-specific complexity. WACC is set at 10% (industry standard per Damodaran). Peak revenue estimates are sourced from analyst consensus at the decision date. All figures are expressed in nominal USD at the decision date.

2.6 Confidence Tiers

3. Data Enrichment Pipeline

3.1 Why Raw CT.gov Data Is Insufficient

ClinicalTrials.gov provides structured trial metadata (phase, status, enrollment, dates), but lacks the drug-level fields critical for computing transition rates: drug class, mechanism of action, molecular target, modality, published efficacy data, and FDA regulatory linkage. Intervention names are inconsistent ("Adalimumab" vs "adalimumab" vs "Humira"), and there is no way to determine which trials belong to the same drug program without domain knowledge.

3.2 Raw Data Scope

Filtering for RA (condition text matching "rheumatoid arthritis") identified 1,304 unique interventional trials across all phases.

3.3 9-Phase Enrichment Process

Each trial was enriched through a systematic, multi-tier process designed to maximize data quality while preventing hallucination.

3.4 Four Data Sources Per Trial

Confidence score = weighted coverage across sources (0–1 scale). All 679 RA trials achieved "full" enrichment level (4 sources consulted).

3.5 Survivorship Bias Verification

Of the 1,304 raw RA trials, 625 were not enriched because they lacked drug-level metadata (non-drug interventions, unmappable entries, duplicate substudies). To verify this filtering was outcome-agnostic, we compared completion-to-termination ratios:

3.6 Final Dataset

4. Drug Cohort

4.1 Approved Drugs

4.2 Failed Drugs

4.3 Selection Rationale

Drugs were selected to span the full history of RA targeted therapy (1996-2019), covering multiple modalities (small molecule, monoclonal antibody, fusion protein, nanobody) and mechanisms (TNF, IL-6, JAK, CD20, BAFF, SYK, CTLA-4). The 8/8 approved/failed split ensures balanced class representation. All drugs reached at least Phase 2 in RA (except atacicept, which entered at Phase 1), providing sufficient clinical data for reconstruction.

5. Results Summary

6. Aggregate Accuracy Metrics

The pairwise AUC of 0.625 is the headline discrimination metric: it passes the 0.60 target and measures the probability that a randomly chosen eventual success carries a higher PoS than a randomly chosen failure. The phase-controlled AUC of 0.65 confirms the signal holds within decision phase, removing the structural advantage that earlier decisions have over later ones (fewer remaining stages = mechanically higher cumulative PoS). At n=16 this is an early directional signal — Wilson 95% accuracy intervals on the conventional and optimal cuts both include chance-level performance, so discrimination is suggestive, not confirmatory.

The honest counterweight: calibration and separation are weak at this sample size. The separation gap between success and failure means is only +8.4pp against a 10pp target, and the false-confidence rate at the 25% PoS cut is 50% (5 of 10 above-threshold calls were failures) against a 20% target. Both fail. The story is discrimination passing while calibration lags — the expected shape for a directionally sound model that needs a larger, multi-indication cohort before its absolute PoS levels can be trusted.

Go/No-Go Threshold Analysis

7. Case Study: Atacicept (Model's Strongest Signal)

8. Case Study: Filgotinib (Model's Edge Case)

9. Computed Transition Rates

A central methodological choice in this back-test is replacing static BIO/QLS NDA/BLA transition rates with rates computed from the enrichment corpus. This is not a refinement — it is a fundamentally different measurement.

The BIO/QLS rate of 91% measures a near-certainty: once a company files an NDA, it almost always gets approved. But the investment decision happens before filing — often years before. The relevant question is whether a drug in Phase 3 will ever reach and pass the NDA stage. Many drugs complete Phase 3 but never file (commercial viability, safety signals, competitive landscape shifts). The computed rate captures this full attrition.

Combined with drug-level counting (tracking individual drugs across phases, not trial counts) and time-gating (only using data available at decision date), this is a central source of the model's discriminative signal — pairwise AUC 0.625 and phase-controlled AUC 0.65.

Production status (as of this writing): the computed indication-specific transition rates described in this section are a research approach. Current production uses static BIO/QLS 2021 base rates.

10. Calibration

11. Limitations

1. Sample size (n=16) — This is a proof of concept, not a powered validation study. Statistical significance requires cross-indication expansion.
2. Single indication (RA only) — Results may not generalize to oncology, rare disease, or CNS indications where PoS dynamics differ substantially.
3. Cost/revenue estimates are manual — Stage costs and peak revenue are reconstructed from public sources and analyst consensus, introducing subjectivity.
4. Class-level safety, not drug-level — The SAFETY_CLASS_SIGNAL flag captures mechanism-level risks but cannot detect molecule-specific toxicities (see: filgotinib).
5. Competitive density is count-based — The model counts competitors but does not assess differentiation, market positioning, or pricing dynamics.
6. Phase 3 cohort has only failures — Both Phase 3 decision-point drugs (filgotinib, peficitinib) failed, preventing within-phase discrimination testing at Phase 3.
7. No survivorship bias in data — Verified: completion rates are identical between the raw 1,304 and enriched 679 trial sets, confirming no systematic exclusion of failed trials.

12. Next Steps

1. Cross-indication expansion — Repeat the back-test for oncology (lung, breast), rare disease, and CNS cohorts. Target: n>=50 drugs across 4+ indications.
2. Drug commercial profiles — Integrate commercial-profile data (peak revenue, LOE dates, biosimilar entry) for automated revenue estimation.
3. Molecule-level safety signals — Incorporate FDA adverse event data (FAERS) to supplement class-level safety flags with drug-specific signal detection.
4. Prospective validation — Identify 10-15 drugs currently in Phase 2/3 and track model predictions against real-world outcomes over 3-5 years.
5. Calibration improvement — Apply Platt scaling or isotonic regression to recalibrate PoS outputs once cross-indication data provides sufficient sample size.
6. Competitive landscape integration — Replace count-based competitor density with the CT.gov landscape data (trial velocity, enrollment rates, phase distribution).