A large language model (LLM) can summarise a record, code a free-text answer or score an interview. When it is wrong, it writes with the same confidence as when it is right. A team at Geneva University Hospitals and the University of Geneva proposes a method to spot these errors and measure their severity before go-live: 
FMECA for LLM clinical summaries. It deals with clinical summaries, but its logic applies to any project where a model's output becomes data.
A well-written error goes unseen
The record mentions a severe penicillin allergy. The summary says "no known drug allergies". The text is fluent and consistent; nothing prompts a check. An overall quality score or the word "hallucination" tells you neither where the error sits nor what it may cost.
A proven method, adapted to LLMs
FMECA, failure mode, effects and criticality analysis, has long been used for medical devices and pharmaceutical manufacturing. You describe what can fail, then rank each failure. Lydie Bednarczyk, Jamil Zaghir, Julien Ehrsam, Christian Lovis and their colleagues applied it to discharge-letter summaries produced by an open model hosted on site. A panel of eight experts listed 78 possible failures, then kept only those a reviewer can observe in the text.
14 observable failures, in six families
Following the instructions5
- Wrong overall structure
- Information in the wrong section
- Out-of-context information added
- Inadequate level of detail
- Vocabulary unsuited to the context
Readability4
- Ambiguous or contradictory wording
- Response in the wrong language
- Repeated words
- Repeated medical information
Ethics1
- Stigmatising or discriminatory vocabulary
Faithfulness to the source2
- Fact that contradicts the source
- Information absent from the source, invented or interpreted
Completeness1
- Omission of information from the source
Technical1
- No summary produced
Two choices make the framework usable. The model and the instructions stay fixed during the assessment, because a structural or level-of-detail error only exists relative to the instructions. And the framework describes what can be seen in the text, without speculating about technical causes.
Two scores: severity and detectability
Each error found receives two scores from 1 to 5. The first measures how severe its effects would be if it went through. The second measures the likelihood that a professional notices it while reading. This second score is the most useful contribution for AI: an impossible date is obvious, an allergy left out of a consistent summary slips through.
Two scores for every error
Severity measures the effect if the error goes through. Detectability measures the chance that a professional notices it while reading.
Spotting is hard, scoring is reliable
Three reviewers annotated 36 summaries in two rounds. Once the definitions were clarified, their agreement on whether an error was present improved, without becoming high. Once an error was found, however, they gave it close scores.
Spotting is hard, scoring is reliable
Agreement between reviewers, from 0 (chance) to 1 (perfect agreement).
| Spotting an error, Round 1 | 0.15 |
|---|---|
| Spotting an error, Round 2 | 0.33 – 0.45 |
| Scoring an error once found, Severity | 0.72 |
| Scoring an error once found, Detectability | 0.73 |
The authors remain cautious: four patients, one hospital, one language, and reviewers who had helped build the framework. The study does not measure how often errors occur and does not qualify the model tested. It provides a method, to be validated on larger volumes.
Rank by severity first
The classic formula multiplies frequency, severity and detectability. It can push a rare but severe error behind frequent, harmless defects. The authors propose starting from severity, then raising the priority when the error is frequent or hard to see. They present these thresholds as provisional, to be adapted to the context.
Severity first
Severity sets the priority. A frequent or hard-to-see error moves up a level.
- Severity 5In every case
- Severity 4Very high if frequent or hard to see, otherwise high
- Severity 3High if frequent or hard to see, otherwise moderate
- Severity 2Moderate if very frequent and very hard to see, otherwise low
- Severity 1In every case
- Very highfix before go-live
- Highfix before go-live
- Moderatefix recommended
- Lowmonitor
Frequent: at least one output in ten. Hard to see: it takes careful reading or a check against the source.
Beyond clinical summaries
The same logic applies whenever a model turns free text into data: coding open-ended answers, analysing patient feedback, scoring research interviews. In patient feedback, a sentence about a medication error filed under "reception" is a severe omission that is hard to see. Irony read as a compliment skews an average without alerting anyone. Error families change with the task; the two questions remain: what happens if the error goes through, and who would see it?
In practice, for a project that scores LLM outputs
- Freeze the version. A dated model, versioned instructions, and the parameters stored with each result. Any change restarts the assessment.
- Define the errors before measuring. A task-specific list, observable in the text, with one example per type.
- Require evidence. Each value keeps the source excerpt that justifies it. A value without an excerpt flags itself.
- Prefer a blank to a guess. When the source says nothing, the field stays empty.
- Have a reference set annotated. Two or three domain reviewers, and a measure of their agreement before judging the model. A second model can then do a first screening, provided it is validated against those reviewers.
- Rank by severity. A severe error blocks go-live, even if it is rare.
- Replay without overwriting. A new model produces a new series of results, compared with the previous one using the same framework.
- Make errors visible. The interface shows the source next to the result; detectability also depends on the screen.
- Measure in service. Each human correction records the type of error, and the real frequency builds up with use.
Our approach
We build applications in which AI produces data used by clinical and research teams. For the overall framework of such a project, see what a clinical study sponsor should prepare. If your project needs to assess a model's outputs, write to us: we start with the error framework and the reference set.
Sources and references
3 sourcesBednarczyk L., Zaghir J., Ehrsam J. et al., Lovis C., Development and refinement of a FMECA framework for identifying patient safety risks in LLM-generated clinical summaries · webstore.iec.ch · doi.org
- Bednarczyk L., Zaghir J., Ehrsam J. et al., Lovis C., Development and refinement of a FMECA framework for identifying patient safety risks in LLM-generated clinical summariesA methodological pilot study. BMC Medical Informatics and Decision Making, 2026, article in press. Geneva University Hospitals and University of Geneva; 36 summaries of French discharge letters, 4 patients, 3 reviewers.doi.org
- webstore.iec.chIEC 60812:2018, Failure modes and effects analysis (FMEA and FMECA), edition 3.0, 10 August 2018. International FMECA standard; official listing accessed 14 November 2026.webstore.iec.ch
- doi.orgAsgari E. et al., A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine, 8:274, 13 May 2025. Closest framework cited by the Geneva team; measured rates depend on the task, model and instructions.doi.org
Swiss Made Software