How an F500 telecom uses Snorkel AI to measure and improve virtual assistant CX

A Fortune 500 U.S. telecom's digital support agent handles millions of interactions a year, but the company struggled to measure customer satisfaction since post-interaction surveys were sparse and skewed negative, and manual evaluation was slow and costly. Snorkel built a conversation-level satisfaction model and a turn-level agent-quality model, using an SME-aligned LLM-as-a-judge that improved alignment from 54.8% to 67.7%, plus a synthetic-preference-pair reward model built on a 1.5B-parameter Qwen model. The system delivered 7x faster evaluation cycles, a 79 F1 conversation-rating score (39 points above baseline), and 81% accuracy predicting turn-level satisfaction, enabling retroactive scoring of all conversations and real-time quality monitoring for A/B testing.

The customer is anonymised in Snorkel AI’s own case study — not withheld by us.

Snorkel AI is a data-development and evaluation company whose programmatic labeling tools build training and benchmark datasets used to stress-test whether a model approach is viable before a full build.

This sits in the Design & Stress-Test stage of the AI use-case lifecycle: Named examples from the tools pharma teams use to scope, prototype and benchmark an AI use case before committing to build it — data-catalog, sandbox and evaluation platforms.