How to Validate a Non-Deterministic LLM Without Pretending It Is Deterministic

Run the same prompt through the same model at temperature zero a thousand times and you can still get eighty different answers. Horace He and Thinking Machines Lab published the experiment on 10 September 2025: 1,000 completions of "Tell me about Richard Feynman" from Qwen3-235B-A22B-Instruct-2507, greedy decoding, identical for the first 102 tokens, then splitting at token 103 where 992 completions said "Queens, New York" and 8 said "New York City". Eighty unique completions in total. Nothing was broken. The batch size on the serving node varies with load, and the standard attention and matrix-multiply kernels are not batch-invariant, so the arithmetic itself changes underneath you.

That is the fact a validation protocol has to survive. An operational qualification script is built on a sentence — enter this input, observe this expected output — that has no stable meaning here. And it does not become meaningful by turning the temperature down, because determinism is a property of the inference stack you control rather than of a sampling parameter. When the same team rebuilt the kernels to be batch-invariant, all 1,000 completions came back identical. Almost nobody buying an LLM through a hosted API controls that layer.

So the answer to the title is that you do not validate the model's output space at all. You validate the deterministic layer wrapped around it, and you qualify the model only as a component whose statistical performance is measured, bounded and monitored. That is not a rhetorical dodge. It is the position ISPE's GAMP community has converged on, and the July/August 2026 issue of Pharmaceutical Engineering puts numbers against it.

Why 1,000 identical prompts gave 80 different answers

The common explanation for LLM nondeterminism — floating-point addition is not associative, and GPU threads finish in an unpredictable order — is mostly wrong for a transformer forward pass, which uses fixed-order reduction trees. The Thinking Machines analysis locates the cause in batch invariance: run the same tokens through the same weights in a batch of 4 and a batch of 32 and the numerics differ, and the batch you land in depends on what other users were doing at that millisecond.

This matters for validation in a way that is easy to miss. It means the nondeterminism you see in production is not a property of "AI" in the abstract that a quality argument can wave at. It is a specific, located, fixable engineering property of the serving infrastructure, and whether you can fix it depends entirely on your deployment model. Self-hosted weights on infrastructure you own: fixable, at a throughput cost. A multi-tenant API endpoint: not fixable, not observable, and not contractually guaranteed.

That distinction belongs in the validation plan, in writing, because it changes the evidence you can generate. If you cannot pin the numerics, a repeat-run test is not a pass/fail check on identical text; it is a measurement of output variance, and you must decide in advance which part of the output you require to be stable. Almost always that is the extracted decision, not the prose. If a model reads a deviation record and returns a severity category and a structured rationale, the requirement is that the severity is stable across runs, not that the rationale is word-identical. State that in the user requirements, set the stability acceptance criterion on the field, and the repeat-run test becomes something you can actually pass.

What draft Annex 22 says, and what it does not

The clause everyone now quotes is 9.2, and it is worth reading precisely rather than in paraphrase. The consultation text says:

"Models used to predict or classify data should have an appropriate threshold setting to ensure predictions or classifications are made only when suitable. If the confidence score is very low, it should be considered whether the model should flag the outcome as 'undecided', rather than making potentially unreliable predictions or classifications."

Two things follow that are routinely overstated in vendor material. First, the threshold is expressed as a "should", and the "undecided" label is softer still — something that "should be considered". Second, and more important, Annex 22 does not apply to your LLM at all if the application is critical, because the scope section excludes it: the annex applies to static models with deterministic output, and states that generative AI and large language models "should not be used in critical GMP applications". Clause 9.2 governs the in-scope classifiers, not the excluded generative systems.

What the scope section does give you is the sentence most people skip. For non-critical GMP applications, it says qualified personnel should always be responsible for ensuring the outputs are suitable — a human in the loop — and that "the principles described in this document may be considered where applicable". That is an invitation to borrow the annex's machinery, not an obligation to comply with it. Borrowing it is still the right call, because an inspector who has read the draft will recognise the shape of your package.

The status question decides how you cite any of this.

The deferral is settled: the Digital Omnibus on AI was published in the Official Journal on 24 July 2026 and entered into force on 27 July 2026, moving standalone Annex III high-risk obligations from 2 August 2026 to 2 December 2027 and product-embedded Annex I systems to 2 August 2028. Article 50 was deliberately left out of that deferral and began to apply on 2 August 2026. The prohibited-practices ban and the Article 4 AI literacy duty have applied since 2 February 2025, and the GPAI model obligations since 2 August 2025.

Annex 22 itself has not moved. EMA held a multistakeholder workshop on 30 June and 1 July 2026 specifically to gather expert input on "possible control and mitigation measures such as guardrails" and on how adaptive and probabilistic models might be accommodated, with the second day closed to allow the drafting group to review contributions. No revised text or policy position has been published since. Which instrument binds you and which merely signals intent is the subject of a fuller guide to what inspectors actually cite in 2026. If yours is a drug-sponsor system rather than a device, note also that computer software assurance is device guidance and citing it as your rule is a scoping error — although CDER's 2026 guidance agenda, reported by RAPS on 16 July 2026, does add a planned supplemental guideline on computer software assurance for AI-based systems in drug manufacturing and clinical investigations.

Bounded nondeterminism: the actual unit of validation

The phrase that resolves the problem is bounded nondeterminism, and the clearest statement of it is in Robert Stoop and Gaurav Walia's article in the July/August 2026 Pharmaceutical Engineering: "Instead of attempting to validate every possible AI output, guardrails establish validated boundaries within which AI systems can operate safely."

That sentence relocates the validated object. What you claim is fit for intended use is not the model and not its answers. It is a pipeline in which a stochastic component sits inside components that are individually deterministic, specifiable and testable against a fixed expected result, so that the model's contribution is bounded by what those components will let through.

Victor Bechmann and Peter Albrechtsen set out the layered version in the January/February 2026 issue: seven control layers — input guardrails, domain knowledge through retrieval-augmented generation, LLM selection and capabilities, fine-tuning, output guardrails, system monitoring, and explainability and transparency. Stoop and Walia's output guardrails are five specific checks: citation validation against validated source documents, rules-based compliance checking against a regulatory requirements database, sensitivity screening for confidential and personal information, confidence thresholding, and tone analysis.

Look at what each of those is from a testing point of view.

One row is statistical. The other six are ordinary computerised-system validation, and they are where most of your evidence will come from. This is the practical meaning of validating a generative system: six-sevenths of the package is not about the model.

The numbers, and what kind of numbers they are

Stoop and Walia's framework is the most concrete published proposal for how to set the boundaries, applied to root-cause analysis and CAPA. It classifies each investigation into one of four categories and attaches a confidence floor to each.

The input side is a weighted fitness score over reliability, relevance, representativeness and abundance, with default weights of 0.35, 0.30, 0.20 and 0.15 — reliability and relevance weighted higher, in the authors' words, "due to GxP criticality". Where a machine-learning component needs training data, they call for a minimum of 100–500 instances per category depending on complexity. Category 0 events carry a timeline expectation of investigation initiation within 24 to 48 hours and preliminary findings within five to seven days.

Now the honesty part, because this is where articles on this subject go wrong. These are two named authors' proposal in an ISPE member magazine. They are not a regulator's numbers, they are not in Annex 22, and no inspector will accept "ISPE said 90%" as a justification. The article itself is explicit that guardrail design offers "a pathway toward regulatory acceptance and industry standardization in an area where formal regulatory requirements do not yet exist."

What is transferable is the mechanism, and the mechanism is genuinely useful: risk band determines confidence floor, data quality gates entry rather than being noted afterwards, and a floor exists below which the system refuses to run at all. Adopt the structure. Derive your own numbers from your own hold-out data, and write the derivation down, because the derivation is the thing that survives challenge.

What an abstention threshold actually costs

Confidence thresholding is the guardrail everyone specifies and few operate, because the cost of it lands somewhere the business case did not model.

Selective prediction has a fixed shape: raising the threshold raises accuracy on the answers you keep and lowers coverage. The errors do not disappear, they change category. An error becomes an abstention, and an abstention becomes a human's work. If a triage system handles 400 records a shift at a 90% confidence floor and abstains on a quarter of them, you have created 100 items of manual work per shift and you need a named person with the time to do it. A threshold set without that staffing analysis will be quietly lowered within a month, and the lowering will not go through change control unless you made the threshold a controlled configuration item.

There is a harder problem underneath. Abstention is a model capability, not a switch, and it is not a solved one. AbstentionBench, from Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri and Samuel Bell at Meta's FAIR, evaluated 20 frontier models across 20 datasets of unanswerable, underspecified, false-premise and stale-data questions. Two findings should change how you write a revalidation trigger. Scaling the model does not reliably help. And reasoning fine-tuning degrades abstention by 24% on average, including on the mathematics and science domains those models are explicitly trained for.

Read that against clause 9.2. The draft asks for a model that can decline. The industry's dominant upgrade path over the last eighteen months — swap the instruction-tuned model for its reasoning-tuned successor because it scores better on the task benchmark — makes that capability worse on average, and does so invisibly, because task accuracy on answerable inputs goes up while the ability to recognise an unanswerable input goes down. A validation package that tested only accuracy would sign that upgrade off.

A related trap: a number the model states about its own certainty is not a calibrated probability, and treating a verbalised "95% confident" as the confidence score in clause 9.2 is not defensible. The score has to come from a measured mechanism — token-level log probabilities, an ensemble or self-consistency spread, a separately trained verifier, or a retrieval-grounding check — and whichever you pick, you owe a calibration curve on your own hold-out set showing that the score means what the threshold assumes it means.

The test strategy, end to end

Three test populations, not one, and they answer different questions.

Guardrail qualification. Scripted, deterministic, with pre-defined expected results and deliberate negative cases. This is the bulk of the protocol and it is conventional work. The negative cases are what an auditor will read: fabricated citations that the citation validator must reject, malformed structures the schema check must catch, records below the fitness gate that must be refused, inputs sitting exactly at the confidence floor that must route to "undecided".

Model performance qualification. Statistical, on an independent hold-out set, with metrics and criteria fixed before the run. Draft Annex 22's clauses here are unusually specific and worth mining even though the annex does not formally reach your generative system. Clause 5.2 requires the test set and each of its subgroups to be "sufficient in size to calculate the test metrics with adequate statistical confidence" — a confidence interval, not a point estimate. Clause 3.2 requires the sample space to be divided into subgroups, so you report per-subgroup performance and can see the rare defect class you actually deployed the thing to catch. Clause 5.6 says generating test data or labels with generative AI "is not recommended" and any use must be fully justified, which rules out the convenient shortcut of having a model label its own evaluation set.

Clause 6 is the one that catches organisations out. Independence is not just a random split: 6.2 requires that staff involved in development never had access to the test data, that it sits behind access control with an audit trail, and that no copies exist outside that repository. Clause 6.5 goes further, requiring procedural or technical controls to stop anyone who has seen the test data from working on training, and where a small team makes that impossible, allowing it only in pairs with a colleague who has not — an explicit four-eyes principle applied to data science staffing.

And clause 4.3, the one nobody costs into the plan: acceptance criteria "should be at least as high as the performance of the process it replaces", which "implies that the performance should be known for the process which is to be replaced". If you are automating human deviation triage, you need the humans' error rate, measured, before you can set the model's. Most organisations have never measured it. Budget for that study; it is often the longest pole in the tent, and it produces a number the business case needs anyway.

Stability testing. Repeat runs of the same input, measuring variance on the decision field rather than on the text, with a pre-declared stability criterion. This is the test that acknowledges the batch-invariance problem instead of pretending it away.

Revalidation triggers that actually fire

Draft Annex 22 clause 10.1 puts the model, the system and the whole process it assists under change control before deployment, requires any change to be evaluated for whether retesting is needed, and requires any decision not to retest to be "fully justified". Clause 10.3 requires regular performance monitoring against the qualification metrics; clause 10.4 requires monitoring of whether inputs are still inside the characterised sample space, with drift metrics defined.

For a generative system, the list of things whose change should trigger that evaluation is longer than a classical system's, and several items on it are outside your control:

• The model version or checkpoint, including a silent point release from the vendor
• The system prompt, and any prompt template or tool definition
• The retrieval corpus, its chunking, its embedding model, and any re-index
• Guardrail configuration, above all the confidence floors and the fitness gate
• Inference parameters, tokenizer, and the serving stack or hosting region
• Evidence from monitoring that the input distribution has moved
• A sustained change in human override rate, in either direction

The last one is the most informative signal you will have and the least commonly instrumented. Overrides falling towards zero usually means the reviewers have stopped reviewing, not that the model improved. The version-control and change-control mechanics behind this — pinning, retirement dates, what a vendor's deprecation notice does to your validated state — are the subject of a separate piece on why a validated model has a retirement date, and the monitoring side, including how to instrument abstention and drift in production, is covered in the operating plan a deployed GxP model needs.

What this means in practice

On Monday, four things.

Write the criticality determination first, in half a day. Critical under Annex 22's functional test means direct impact on patient safety, product quality or data integrity. If your generative system is critical, the draft's answer is that you should not be using one, and no amount of guardrail engineering changes that while the draft stands. Redesign so the model's output feeds a human decision, or accept that you are ahead of the guidance and document that at quality-unit level.

Make every threshold a controlled configuration item. Confidence floors, the fitness gate, the retrieval top-k, the temperature. If these live in a YAML file that an engineer can edit, you have no validated state. If they live in a change-controlled configuration record with a justification traceable to hold-out data, you have one.

Measure the human baseline before you set the acceptance criterion. Clause 4.3's logic holds whether or not the clause ever binds you: a system that performs worse than the process it replaces has failed, and you cannot know that without the baseline.

Staff the abstentions before go-live. Model the abstention rate from the hold-out set at your chosen floor, convert it to items per shift, name the person, and put the number in the operational SOP. This is the single most common reason a well-designed threshold gets quietly turned down in month two.

The costs, honestly stated: guardrail qualification is ordinary CSV effort and behaves like it. Two items blow up plans — the independent test set, which under a strict reading of clause 6 means data your development team has never seen, held separately, with staffing controls, and the human baseline study. The signatures are a process SME on the intended-use description and the acceptance criteria, which clauses 3.1 and 4.2 assign explicitly, plus QA on the change control.

What goes wrong is rarely the model. It is a threshold nobody owns, a review step nobody has time for, and a vendor model update that arrived on a Tuesday without a notification. All three are governance failures with technical symptoms, and all three are visible in an audit trail if you built one that records what the model was, what it said, how confident it was, and what the human did next.