From hours to seconds on CLO contract review with 94% end user acceptance

A top-10 US bank's portfolio managers and risk teams needed fast, trustworthy answers from CLO (Collateralized Loan Obligation) contracts up to 500 pages long, but manual searches took hours and an initial GPT-4 prototype was rated acceptable in only 25% of expert reviews due to nested definitions, cross-referenced terms, and layered conditions. Snorkel's engineers worked with subject-matter experts to programmatically encode expert judgment into training data, generate synthetic edge cases, redesign contract processing around semantic similarity and document structure rather than fixed token windows, apply custom classifiers for metadata tagging, and fine-tune embedding models, building an eval set that supported 40+ experiments. SME acceptance rose from 25% to 94% (Sprint 1 hit 79%, Sprint 2 hit 89%, Phase 2 reached 94%) within three weeks, eliminating hallucinations across 48 high-value topics and cutting response time from hours to 15 seconds.

The customer is anonymised in Snorkel AI’s own case study — not withheld by us.

Snorkel AI is a data-development and evaluation company whose programmatic labeling tools build training and benchmark datasets used to stress-test whether a model approach is viable before a full build.

This sits in the Design & Stress-Test stage of the AI use-case lifecycle: Named examples from the tools pharma teams use to scope, prototype and benchmark an AI use case before committing to build it — data-catalog, sandbox and evaluation platforms.