Why Predictive Models Collapse on Your Own Chemistry

A model that scores 0.66 on a public permeability benchmark and near-nothing on your own compounds has not degraded. It has been moved outside the chemistry it was fitted to, and the distance is measurable. The uncomfortable finding from the small number of published prospective industrial benchmarks is that the distance grows faster than the training set does, so adding data does not close it.

The best-documented run is Biogen's. Cheng Fang and colleagues collected 120 internal prospective data sets over 20 months across six ADME in vitro endpoints — human and rat liver microsomal stability, MDR1-MDCK efflux ratio, solubility, and human and rat plasma protein binding — and scored models against compounds that did not exist when the models were trained. Gradient boosting and deep learning consistently beat random forest. Retraining on a fixed schedule helped, and more frequent retraining helped more. Hyperparameter tuning barely moved prospective performance at all. What did not happen is the thing every vendor deck promises: predictivity did not reliably improve as the training set grew.

The mechanism is not mysterious. As the gap between retraining and prediction widens, the chemistry the model is asked to score drifts away from the chemistry it learned. In Apheris's reading of the Biogen results, train–test similarity falls from 0.79 at a one-month retraining interval to 0.68 at ten months. That is the number that predicts your model's behaviour, and it is the one almost nobody reports on a slide.

Why a random split flatters every model you have ever seen

Lead optimisation produces compounds in series. Twenty analogues of the same scaffold get registered in the same fortnight and assayed in the same plate. Split that data randomly and members of one series land on both sides of the line, so the model is partly recalling neighbours rather than predicting strangers. Robert Sheridan made the point cleanly in 2013: time-split selection gives an R2 closer to true prospective prediction than random selection, which is too optimistic, or leave-class-out, which is too pessimistic.

Thirteen years on, most published model comparisons still lead with a random or scaffold split. The reason is prosaic — public data sets rarely carry registration dates, so a temporal split cannot be constructed. Greg Landrum, working with Novartis colleagues, built the workaround: SIMPD uses a multi-objective genetic algorithm with objectives derived from an analysis of early-versus-late compounds in more than 130 Novartis lead-optimisation projects, and applies them to ChEMBL to produce 99 public data sets whose splits behave like real temporal ones. If you are benchmarking on public data and you cannot do a time split, SIMPD is the honest substitute. There is no excuse left for reporting a random-split number as evidence of prospective performance.

The permeability case: 0.663 in public, negative R2 in-house

The clearest published example of transfer failure is a Caco-2 study from Zhejiang University with Shanghai Qilu Pharmaceutical, published in the Journal of Cheminformatics on 10 January 2025. The team curated 5,654 non-redundant Caco-2 permeability records from an initial 7,861 compounds, and an XGBoost model on a combined molecular representation reached R2 0.663 with RMSE 0.479 on the held-out public test set. Applied to 67 in-house compounds from the industrial partner, R2 went negative — meaning the model predicted worse than simply returning the training mean.

That headline is worth slowing down on, because it is routinely told as "near-zero predictivity on proprietary scaffolds" and that is not quite what happened. The same models held Pearson r around 0.70 and Spearman rho around 0.72 on the industrial set. The ranking survived; the calibration did not. A model that ranks well but is miscalibrated is still useful for triage and useless for a threshold. If you are choosing which ten of two hundred designs to synthesise, rank correlation is the metric you need. If you are asking whether a compound clears a permeability cut-off before it goes into an assay queue, negative R2 means the prediction cannot carry that decision.

The same paper is unusually candid about applicability domain. Compounds sharing similarity above 0.70 with training data gave more reliable predictions, and the authors warn that relying on the applicability domain can itself mislead when the domain is defined by distance-based approaches over simple fingerprints. Applicability domain is a confidence heuristic, not a guarantee, and the 67-compound sample size means the industrial result is a signal rather than a settled effect.

What a naive baseline does to a benchmark leaderboard

Before asking whether a model generalises, ask what it beats. Pat Walters ran that test on Biogen's own released solubility data, 2,173 measurements taken by chemiluminescent nitrogen detection at pH 6.8. The dynamic range is about three logs and 58 per cent of the LogS values sit between −4 and −3.5. Guessing −4 for every compound gives a mean absolute error of 0.46 logs. Any model reporting 0.45 on that data has demonstrated nothing.

The same post makes a sharper point about the benchmarks the field ranks itself on. Public solubility sets spanning ten or twelve logs make correlation cheap: on the Therapeutics Data Commons solubility set, LogP alone gets a Pearson r of 0.77. A leaderboard built on that data measures how well a method recovers lipophilicity, not how well it will help a chemist choose between two analogues that differ by half a log. This is why the industry-led critique of drug discovery benchmarking, published in Nature Machine Intelligence in 2024 by Wognum, Ash, Aldeghi, Rodríguez-Pérez and Walters, argued for datasets and evaluation protocols owned by the people who have to act on the predictions.

Does AI target identification generalise any better?

It has the same problem in a different coat, and one more honest piece of evidence.

The strongest published intervention on target choice is still human genetics. Minikel, Painter, Dong and Nelson estimated in Nature on 17 April 2024 that the probability of success for drug mechanisms with genetic support is 2.6 times greater than for those without, up from the roughly two-fold figure the same group reported a decade earlier. Relative success varies by therapy area and development phase, rises with confidence in the causal gene assignment, and is largely unaffected by genetic effect size, minor allele frequency or year of discovery. Note what 2.6-fold means against a base rate where only about 10 per cent of clinical programmes reach approval: it is a meaningful shift in a portfolio, not a filter that makes an individual programme safe.

A June 2026 preprint from the University of Edinburgh — single-author, not peer reviewed, and to be read as such — pushes on that number in a way worth knowing about. Analysing 26,278 target–disease pairs from Open Targets and ChEMBL, Victoria Paterson reports a pair-level odds ratio of 3.25 for any genetic association, falling to 2.79 at target level once the non-independence of pairs sharing a gene is corrected for, because the 26,278 pairs derive from only 1,434 genes. Oncology's pair-level odds ratio of 6.72 attenuates to 2.71 on the same correction. The enrichment replicates on post-2015 approvals.

The instructive part is the ablation. Literature mining alone accounts for most of the classifier's performance, AUPRC 0.099 against 0.109 for all six evidence types combined, which the author attributes to temporal leakage from publications that appear after approval. Genetic evidence alone contributes about a 1.0 percentage-point absolute AUPRC gain, and the best model has a Brier score of 0.20 with poor calibration. In other words: the genetic association is real and replicated, and a target-prioritisation classifier built on Open Targets evidence scores has limited practical predictive value. Both statements are true at once, and only one of them appears in pitch material.

The one clinical data point worth citing is rentosertib, a TNIK inhibitor where the target came from Insilico Medicine's generative platform. The phase 2a trial published in Nature Medicine on 3 June 2025 randomised 71 patients with idiopathic pulmonary fibrosis across three dose arms and placebo for 12 weeks. The primary endpoint was the proportion of patients with at least one treatment-emergent adverse event, similar across arms; forced vital capacity was a secondary endpoint, with a mean change of +98.4 mL (95% CI 10.9 to 185.9) in the 60 mg once-daily arm against −20.3 mL on placebo. Insilico sponsored the study. This is a genuine milestone for an AI-nominated target, and it is a 71-patient, 12-week safety trial with an encouraging secondary endpoint, not efficacy evidence. Discontinuations were driven by liver toxicity and diarrhoea.

Why the virtual cell benchmarks are the clearest warning

Perturbation prediction is where the generalisation question is being asked most publicly, and the answers are recent and blunt.

Arc Institute released State on 23 June 2025, trained on observational data from nearly 170 million cells and perturbational data from over 100 million cells across 70 cell lines, and described it as the first model to consistently beat simple linear baselines. Six months later, Arc's own wrap-up of the 2025 Virtual Cell Challenge, published 6 December 2025, reported that perturbation prediction models are not yet consistently outperforming naive baselines across all metrics, and that almost all models performed worse than baseline on mean absolute error. The challenge drew over 5,000 registrations across 114 countries, more than 1,200 teams submitting and over 300 final submissions, scored on perturbation discrimination, differential expression and MAE. Real gains appeared on discrimination and differential expression. The absolute-error metric went the other way.

That is the organisation with the strongest incentive to declare progress publishing the result that a mean-shift baseline still wins on a core metric. It is also the most useful thing anyone has said about this field in two years. The 2026 challenge sharpens it further: models must predict CRISPRi knockdown responses in six cell lines they have never seen perturbed, with no challenge-specific training set, using only the unperturbed state and a gene list. Validation and submissions opened on 20 August 2026, the final test set is released on 22 October and final submissions are due 5 November. Zero-shot transfer to an unseen cell context is exactly the ADMET problem — a model asked to extrapolate past the region it was fitted to — restated in transcriptomics.

Where the regulators actually sit on a discovery model

Mostly, they do not sit on it at all, and that is the answer a discovery lead should be able to give without hedging.

The EMA's reflection paper on the use of artificial intelligence in the medicinal product lifecycle, adopted by the CHMP and CVMP and published on 9 September 2024, is a reflection paper and not binding guidance. It says that where AI is used in drug discovery and the results form part of the evidence submitted for regulatory review, the principles for non-clinical development apply, while acknowledging that the risk to a marketing authorisation applicant at that stage may be low. The trigger is submission, not modelling.

In the United States, FDA published Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products as a draft guidance in the Federal Register on 7 January 2025, docket FDA-2024-D-4689, with the comment period closing on 7 April 2025. A Federal Register search of FDA documents on 30 August 2026 returns no notice announcing a final version, so the draft and its risk-based credibility assessment framework organised around a defined context of use remain non-binding recommendations. Separately, FDA issued a request for information on an AI-Enabled Optimization of Early-Phase Clinical Trials Pilot Program on 29 April 2026, docket FDA-2026-N-4390, comments closing 29 May 2026, with principles aligned to the NIST AI Risk Management Framework. That is a proposed pilot at the RFI stage, not a programme you can enrol in today.

On the EU GMP side, the point that trips people up: on 30 August 2026 the binding text on computerised systems is still the 2011 Annex 11. A revised Annex 11, a revised Chapter 4 and a new Annex 22 covering artificial intelligence were published for public consultation on 7 July 2025, the consultation closed on 7 October 2025, and none of them has been adopted. Anyone telling you that Annex 22 requires a particular validation approach for your ADMET model today is describing a draft clause as law, and in any case a discovery model that never touches a GMP activity is outside Annex 11's scope in either version.

What this means in practice

Rebuild your internal benchmark as a temporal split this week. You already have registration dates in the corporate compound registry; the work is joining them to the assay table and re-scoring the models you currently trust. Expect the numbers to fall. That fall is not a regression, it is the removal of an accounting error.

Report train–test similarity next to every metric and treat it as a release gate. The Biogen result only becomes actionable once similarity is a monitored quantity: 0.79 and 0.68 are not incidental, they are the explanation. Pick a similarity floor below which the model returns "out of domain" rather than a number, and accept the Caco-2 authors' caution that a distance-based domain over simple fingerprints will sometimes tell you the wrong thing.

Set retraining cadence from measured decay rather than from the calendar. Biogen found more frequent retraining generally better; a lead-optimisation case study from Nested Therapeutics and Inductive Bio, published in ACS Medicinal Chemistry Letters on 25 July 2024, found weekly retraining on human liver microsomal stability holding a Spearman R of 0.65 against 0.55 for the same model one month later, and that models trained on both curated external data and project-specific data beat either alone across all four ADME properties tested. Weekly is a pipeline problem, not a science problem.

Score against a naive baseline before anything else, and put the baseline in the report. Predict the training mean. Predict from LogP. Predict from the nearest neighbour by fingerprint. If the model does not beat all three on a temporal split of your chemistry, you do not have a model, you have a summary statistic.

Separate the two decisions the model is being asked to support and buy metrics for each. Triage — which designs get made — needs rank correlation and can tolerate miscalibration. Gating — does this compound clear a threshold — needs calibration and an honest R2 on held-out prospective data. Most ADMET deployments quietly promote a triage model into a gating role somewhere between the pilot and the second budget cycle, and that is where the credibility is lost.

Finally, name the cost. A temporal benchmark requires that you keep running the assay on compounds the model has already scored, which is the expense everyone hopes prediction will remove. Budget it as a permanent line, not a validation phase. The comparison that matters is not model against assay, it is the cost of the assays you keep against the cost of the decisions you would otherwise make on a number with a negative R2 behind it. On the evidence published so far, keeping the assay running on a sampled fraction of predictions is the cheaper of the two.