In a previous blog, we made the case that hallucinations aren’t a glitch to be patched out of a model, but a predictable consequence of how these systems are trained. We closed that piece with a claim that deserves more scrutiny: that Weave measures quality through a combination of rigorous experimentation and real-world user behavior, rather than relying on a model’s ability to sound convincing.
This piece goes one level deeper: what does it take to measure the quality of AI-generated content for regulatory submissions, and how can the tools the rest of machine learning relies on be adapted for this use case?
Adapting standard LLM quality metrics for regulatory submission content
The standard toolkit for evaluating classification by a machine learning model is precision and recall. Did the model correctly flag what needed flagging? Did it miss anything? These metrics work beautifully when a model’s job is to make a discrete call: verify this data point, tag this document, flag this anomaly. There’s a ground truth to check against. The answer is right or it isn’t.
Long-form content generation is more complex. When a model drafts a clinical overview section or synthesizes a study summary, there’s no single correct answer to check against — there’s a space of acceptable outputs, and a much larger space of outputs that are subtly, unacceptably wrong. “Was this generated section correct?” isn’t a yes/no question the way “was this data point tagged correctly?” is.
To evaluate generation tasks, NLP experts have historically used graded rather than binary metrics: n-gram overlap measures like ROUGE and BLEU, embedding-based similarity like BERTScore, and structured human rubrics with inter-annotator agreement as the reliability check. Other off-the-shelf evaluation frameworks, such as RAGAS, include metrics for faithfulness, relevance, and adherence.
Deploying generative AI in Weave’s customer environments required adapting these and other metrics to measure the quality of regulatory submission content. In drug development and other regulated industries, the discipline of evaluating whether an AI system is good at its job is still being invented, in real time, by the teams building on the frontier. We encourage our peers in the space to share their feedback on our framework and continue openly circulating their ideas.
Breaking AI quality into measurable dimensions: faithfulness, adherence, and more
The first step was refusing to treat “quality” as one fuzzy, holistic judgment. A single up-or-down vote from a human reviewer tells you almost nothing about why something went wrong or how to fix it. And when that vote comes from an LLM grading its own kind, it can tell you less than nothing. Each problem has a different answer.
We address the first by decomposing quality into the distinct, independently measurable dimensions, each targeting a specific failure mode we’ve a) observed in production, and b) validated with regulatory writers to confirm that they are in fact valuable to measure.
We address the second with independence: our evaluators come from a different model family than the models doing the writing, and they run at zero temperature — so a change in score reflects a change in the system, not a reroll of the dice.
A few examples of what we track and why each one matters in regulatory writing:
- Faithfulness asks whether generated content actually matches its source material. This is the throughline from our hallucination piece: faithfulness errors happen when a model has the right source in front of it but still distorts it. We evaluate this at a granular level, sentence by sentence and, for tables, cell by cell, rather than asking for one verdict on an entire document. A single unfaithful sentence buried in an otherwise accurate ten-page section is exactly the kind of error a document-level check would miss — and exactly the kind that matters most in a regulatory filing.
- Prompt adherence asks whether the model did what it was actually asked to do. In regulatory authoring, instructions are rarely simple. So we don’t ask an evaluator to render one verdict on a complex instruction. We break the instruction into its component parts and check each one independently. The result isn’t a single adherence score — it’s a diagnosis of exactly which part of the instruction was missed.
- Non-redundancy and efficiency ask whether the draft repeats itself, and whether it says what it needs to in as few words as possible. This isn’t a style preference. Every redundant sentence is a sentence a reviewer has to read, question, and reconcile against everything else in the dossier. Measuring efficiency is measuring how much unnecessary surface area for inconsistency we’ve left in a document.
- Structural integrity checks whether tables render properly and whether they’re titled. Do they have titles? Do the outputs merely look complete on a scan, or are they actually complete? These are the unglamorous checks that rarely make it into a product demo but reliably make it into a real production incident if they’re skipped.
More recently, we began experimenting with metrics for technical writing quality. These have not yet been validated by specific individuals, but reflect general feedback we’ve received from users.
- Clarity asks whether a regulatory reviewer can understand each sentence on first reading, and whether the prose is lean: short direct sentences carrying one or two thoughts, active voice where the actor is known, paragraphs that open with a topic sentence and follow a single line of reasoning, and no filler (“it should be noted that”, “due to the fact that”).
- Emphasis asks whether the right things are prominent. It scores whether the most pivotal data gets proportional space and position, led in topic sentences rather than buried mid-paragraph, so the reader comes away with an accurate reading of the pivotal messages.
None of these dimensions substitutes for the others. A section can be faithful to its source and still redundant. It can be efficient and still miss half the instruction. True quality assessment means tracking all of these continuously.
Who grades the grader? Validating LLM evaluators against human judgment
The harder problem underneath all of this: most of these dimensions can’t be measured with a simple rule-based check. Faithfulness, adherence, and redundancy all require judgment, which means we’re using an LLM to evaluate the output of another LLM. We don’t take that arrangement on faith.
An LLM evaluator is only useful if it agrees with what a knowledgeable human would conclude. We validate evaluator judgments directly against human assessments of the same content. And that validation isn’t a one-off annotation exercise. Reviewers confirm or flag AI-verified content at the same granularity the evaluator works at — sentence by sentence, cell by cell — and a human decision always overrides an automated one.
Every one of those actions is a human judgment on a machine judgment about the same words, accumulating continuously, on real submissions. Every time we make a significant change to the system — a model upgrade, a change to a pipeline that feeds the model context — we rerun the evaluation. Quality measurement is not a certification we earn once. It’s a discipline we repeat, because the alternative is discovering a regression after it’s already shipped.
Beyond the lab: measuring AI quality with real-world usage signals
Experimentation measures performance under controlled conditions. It doesn’t measure performance once real users, with real deadlines and real documents, are actually using the system. That’s why we also track user actions directly in the product: does a reviewer accept or reject an AI-suggested revision? Add or delete a data tag the model proposed? Flip a verification flag the model set?
Each of these is a small, real signal about whether the system is doing its job. Unlike a controlled experiment, it’s happening continuously, in production, on real content, for as long as the feature is live.
We’re also direct about the limits of this signal. A user not overriding a suggestion doesn’t mean the suggestion was correct. It might just mean no one looked closely. These signals are structurally better at catching what the model got wrong, which a user can flag, than what the model should have done but didn’t, since there’s nothing on the screen to react to. Any vendor telling you their usage metrics capture the full picture of quality either hasn’t thought carefully about this or isn’t being straight with you. We track it anyway, because an imperfect real-world signal is still more valuable than no real-world signal — but we treat it as one input among several, not a verdict.
“Why can’t I just use my own ChatGPT for this?”
This question comes up often, and it deserves a real answer.
A frontier LLM accessed through a general-purpose chat interface can absolutely produce fluent, plausible regulatory content.
What that interface doesn’t give you is any of the above. No faithfulness scoring against your source documents. No sentence-level or cell-level QC. No calibration discipline verifying that the system’s self-assessment tracks human judgment. No instrumentation showing you where, across your team’s real usage, the system is quietly underperforming. You get an output. You don’t get a measurement system telling you whether to trust it, and you don’t get a feedback loop that improves it based on how your team actually works.
The model itself is table stakes, and it’s becoming less differentiated by the month as frontier labs converge on similar capability. What doesn’t come bundled with API access is the infrastructure to know, with evidence, whether a piece of output belongs in your regulatory filing. That infrastructure isn’t a feature we added. It’s the product.
Rigor over marketing: why AI quality measurement matters in drug development
In drug development, the standard is whether the output holds up under intense scrutiny from a reviewer, an auditor, a health authority. One unsupported claim can trigger a review cycle. There is no version of “close enough” that survives that environment.
We measure quality this way because our customers can’t afford for us to guess, and because at our core, we are scientists too. It means something to put your name behind a claim, and part of that is showing your work.
To learn more about how leading regulatory teams are applying quality frameworks in practice, register for our webinar on August 19. We’ll draw on Weave’s real-world collaboration with Takeda to share the playbook for scaling AI in drug development without sacrificing rigor.
