Seventy Percent Recall Is What an MLR Agent Actually Delivers

A team at AWS published the internals of a multi-agent medical, legal and regulatory review system on 11 June 2026, and with them a number almost nobody else discloses: "The multi-agent system achieves over 70% recall across different issue types in our proof-of-concept testing." The same post reports that "the fleet of reviewer agents requires on average 2 minutes to process a batch of 1-3 pages", and states the design choice behind the recall figure without hedging — "in regulated content review, failing to detect a genuine issue carries significantly greater risk than surfacing a false positive, therefore the system was optimized to prioritize recall over precision" (AWS Industries blog, Kozodoi, Taleb, Walia Bhullar, Zinovyeva and Eigner).

Read that as a buyer rather than as an engineer and it says something uncomfortable. Just under three in ten genuine issues are not surfaced. Because that is true, the reviewer's own read of the asset cannot be removed — you cannot skip a page the agent has cleared, because a cleared page is 70% cleared. And because the system is deliberately biased towards flagging, the reviewer now has a flag list to adjudicate on top of the read they were already doing. The correctly built MLR agent increases the volume a human must work through before it reduces anything.

That is not a criticism of the architecture. It is the architecture working as specified. But it relocates where the return comes from: not from reviewer hours per asset, which go up first, but from review rounds per asset, which is a different metric with a different owner and a different reporting line.

What the architecture actually does, step by step

The published design runs six stages, and each one is a place where the recall figure is either earned or lost.

A multimodal model first extracts text and visual descriptions from the PDF. Pages are then grouped: "a multimodal LLM processes pages in sliding windows of up to 5 pages at a time and groups them into semantically coherent batches based on content continuity", because "batching keeps each review call focused on a manageable amount of content while preserving the context needed for accurate assessment". Statements are extracted as distinct claims with normalised full context. A claims verification agent "cross-references the extracted statements against a database of pre-approved claims from previously published content", flagging deviations from approved wording and statements lacking substantiation.

Then the fleet runs, in three kinds. Fixed-context reviewers carry stable references in the system prompt — code of practice and product information agents. Dynamic-context reviewers use retrieval-augmented generation over Bedrock Knowledge Bases, so a scientific references reviewer can verify a claim against the clinical studies behind it. Model-grounded reviewers carry no external reference at all; a spell-checker is the example given. They run in parallel, so "the total review time is determined by the slowest agent rather than the sum of all agents".

Finally an editor agent "collects the outputs from all expert reviewers, deduplicates overlapping findings, assigns confidence scores on a 1–100 scale calibrated through ten few-shot examples that illustrate different severity levels", and emits a structured report. The reviewer sees a spreadsheet: page number, exact quote, issue description, relevant reference, suggested fix, source guideline, confidence score.

Two details in that paragraph deserve more attention than they usually get. The confidence score is the routing variable — it is what any deployment will use to decide which flags a human sees first, or at all. And it is calibrated on ten examples.

Why you cannot filter your way out of 70%

The instinct on reading "tuned for recall" is to fix the noise later: set a confidence threshold, suppress everything below it, hand the reviewer a shorter list.

That trade only runs one way. Every flag you suppress is either a false positive you were right to drop or a genuine issue you have just hidden, and at the point of suppression you do not know which. Filtering lowers the flag count and lowers recall. The 70% is the ceiling of what the system found, not the floor of what you will see after routing. A deployment that suppresses everything below, say, 60 confidence is running at some recall below 70% that nobody has measured, on a scale anchored by ten few-shot exemplars.

The post, as published on 11 June 2026 and as read on 30 August 2026, reports recall and does not report a precision figure. That is the single most consequential gap for a buyer, because precision is what determines the adjudication load. Recall tells you how much of the risk the agent absorbs. Precision tells you how much work it creates. You get one of the two numbers in public and it is not the one that sets your staffing.

A second published data point suggests the trade is real and steep. An i.AI, University of Liverpool, Oxford and Imperial team evaluated LLM medication safety reviews against NHS Cheshire and Merseyside records, publishing on 24 August 2026 (arXiv:2512.21127). Running the same model at three reasoning efforts moved the pair almost mechanically: precision 0.679 and recall 0.477 at low effort, 0.689 and 0.573 at medium, and 0.616 each at high. Buying recall cost precision, point for point. Their summary judgement is the one an MLR buyer should internalise: "the System reliably detects that something is wrong, yet correctly identified what is wrong and what to do about it in fewer than half of cases." Different task, different domain, same shape.

What a 1-to-3 page batch structurally cannot see

Batching is what makes the latency figure possible and it is also the source of a specific, predictable class of miss.

The US standard that generates most enforcement is not a per-sentence standard. Fair balance under 21 CFR 202.1(e)(5)(ii) is assessed across the whole piece, and OPDP's operative concept is net impression — what the material conveys taken together, including layout, imagery, prominence and what is adjacent to what. An agent reasoning over pages 4 to 6 in isolation cannot form a view about whether the benefit language on page 2 is balanced by the risk presentation on page 19.

The 2026 enforcement record splits cleanly along that line. OPDP's January 2026 untitled letter on Anktiva challenged survival and cystectomy-avoidance messaging derived from a single-arm trial and concluded that disclaimers did not cure the misleading overall impression — a whole-document judgement no batch sees. By contrast, the July 2026 letter to Sanofi over Beyfortus turned on communications referring to "RSV disease" where the approval covers "RSV lower respiratory tract disease", and the July 2026 letter to Viatris on Tobi Podhaler turned on convenience claims ("in the car", "anywhere") contradicted by the approved Instructions for Use. Those two are exactly what a claims verification agent with the approved label in its reference corpus should catch: string-level divergence from approved wording, and a claim contradicted by a document the agent holds. The FDA Law Blog counted the Sanofi and Viatris letters as OPDP's 20th and 21st untitled letters of 2026 as of 28 July 2026; the letters themselves sit on FDA's untitled letters index.

The volume context matters. Gardner Law records that on 9 September 2025 alone OPDP sent 46 untitled letters, "more than double the number sent in the previous 5 years combined", against a pre-September 2025 baseline that Covington puts at five untitled letters and one warning letter for the year (Covington, 2025 year in review). The environment your agent is being asked to pre-empt changed by an order of magnitude inside one month, and it skewed towards direct-to-consumer material, where net impression judgements dominate and per-batch checks help least.

So the 70% is not evenly distributed. It will be higher on label divergence, reference mismatch and spelling, and lower on the whole-document judgements that generate the letters people lose their jobs over. The published post reports recall "across different issue types" without a per-type breakdown, and that breakdown is the thing you most need.

The numbers in public, and what each is worth

The gap between rows one and four is the whole commercial problem. A published architecture with a stated tuning rationale and an admitted limitation reports just over 70%. A product page reports 98% with no denominator. That is a 25 to 28 point spread between a measured number and a marketed one, and the marketed one is what lands in the business case.

Two caveats, in fairness. The AWS post is written by the vendor of the underlying platform, so it is not independent either; what distinguishes it is that it publishes a recall figure with the tuning rationale attached and states plainly that "actual performance will vary based on document complexity, reference material quality, and prompt design". And Veeva launched Falcon MLR alongside its acquisition of Copli on 23 June 2026, describing an agentic system that checks content against approved labels and local regulations (Veeva press release). The mechanism is credible; the unsourced percentages beside it are not evidence.

The "1.3 additional compliance gaps per already-approved page" figure is worth pausing on even though it is unmethodologied, because it points at the right variable. If a system surfaces even one flag per page on material that has already cleared MLR, a 24-page detail aid arrives at the reviewer's desk with roughly two dozen items to adjudicate that did not exist before. That is the workload the recall tuning buys.

The arithmetic nobody puts in the business case

Take a 24-page detail aid, and treat this as an illustrative model rather than a benchmark, because no public source supports the per-flag adjudication times that would make it a costing.

At 1-3 pages per batch the piece decomposes into roughly 8 to 24 review calls. If those batches run in parallel, wall-clock time is close to the 2-minute figure. If they run serially, it is 16 to 48 minutes. The post states that agents parallelise within a batch — "the total review time is determined by the slowest agent rather than the sum" — and does not state whether batches themselves are parallelised. That unstated detail is the difference between a same-minute pre-check and a queue, and it is the first question to put to any vendor.

Now the human side. The reviewer receives a flag list and must clear each item: agree, reject, or escalate. Then they read the asset anyway, because 70% recall means the unflagged pages carry residual risk. Reviewer time per asset therefore goes up, not down, in the first pass.

The return has to come from somewhere else, and it does. Indegene's manual-state benchmark — vendor-published and directional, not audited — puts the baseline at about three review rounds per asset over roughly 21 days at $2,500 to $5,000 per asset. Rounds are where the money is. If a pre-check catches the label divergences, the missing references and the unapproved wording before the committee convenes, the committee's first pass is cleaner and round two may not happen. Falcon's own published claim is a 22% reduction in review cycles, which is a far more plausible shape of benefit than 98% accuracy, and notably more modest.

That reframes the metric set. If your pilot reports reviewer hours saved, it will report a loss and you will kill a system that was working. Report rounds per asset, first-time-right rate, and rework volume.

Which rules actually bind this, on 30 August 2026

There is a habit in pharma of reaching for the GxP validation playbook whenever a model touches a document. Here it mostly does not apply, and being precise about that saves real money.

Promotional review is not a GMP activity. The binding EU instrument for computerised systems remains the 2011 Annex 11 to EudraLex Volume 4, and it governs systems used in GMP-regulated activities. The revised Annex 11 and the new Annex 22 on artificial intelligence were published for consultation on 7 July 2025, that consultation closed on 7 October 2025, and no final text of either has been published as of 30 August 2026; EMA held a workshop on 30 June and 1 July 2026 to inform the further development of Annex 22 (EC stakeholder consultation page). Draft Annex 22 is not enforceable, and even when final it will not reach a promotional review tool.

The EU AI Act position is similarly narrower than the discourse suggests. A promotional content review agent does not sit in any Annex III category of Regulation (EU) 2024/1689, so the high-risk obligations — and their deferral to 2 December 2027 for Annex III systems and 2 August 2028 for Annex I embedded systems, enacted by the Digital Omnibus on AI, Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026 and in force since 27 July 2026 — do not attach. What does apply already: the Article 5 prohibitions and the Article 4 AI literacy duty, in application since 2 February 2025, and the GPAI obligations since 2 August 2025.

The instrument that actually binds is the one that has always bound. In the US, promotional material must satisfy 21 CFR 202.1, and specimens must be submitted on Form FDA 2253 at the time of initial dissemination under 21 CFR 314.81(b)(3)(i). A named human signs that submission. No recall figure changes who is accountable when OPDP writes.

What this means in practice

Build the ground truth before you build anything else. Your own MLR system holds years of reviewer comments against assets — that is a labelled dataset nobody else has, and it is the only way to get the per-issue-type recall the published figures withhold. Score the agent against 100 to 200 historical assets and report recall and precision by issue type: label divergence, reference substantiation, unapproved claim wording, fair balance, spelling and style.

Set the confidence threshold empirically and write down what it costs. Whatever number you pick, calculate the recall below it on your labelled set and put that figure in the validation record, because that is the miss rate you have chosen to accept.

Change the pilot's success metric before it starts. Rounds per asset, first-time-right rate and rework volume. Not reviewer hours, which will worsen.

Staff the adjudication. If the system produces even one flag per page on approved content, someone clears those flags, and in most organisations that person is the reviewer you were trying to unburden. Budget for a triage role in the first six months.

And keep the signature where it is. The system, on its own authors' account, "accelerates the initial review pass", and "the final decisions on the content must remain with qualified experts". At 70% recall that is not a governance nicety. It is arithmetic.