npj Digital Medicine ·
Human-in-the-loop validation of a sequential multi-LLM medical education pipeline
Pa Hong: co-corresponding author
As checked on 14 October 2026, the publisher provides an early, peer-reviewed accepted manuscript. It is citable and will be replaced by the final Version of Record after further editing.
Research question
Can a multi-LLM education pipeline replace human verification?
An automated PASS label is not equivalent to human-assessed safety. This study examined a seven-stage multi-LLM radiology education pipeline through resident and attending evaluations of blocking errors and rating changes around automated feedback.
Study design
The pipeline generated 6,000 flashcards and 833 infographics. Nine residents and eleven attending radiologists evaluated 1,284 flashcards across eleven subspecialties. Generated items, evaluated cards and evaluation-level observations are distinct denominators.
Key findings
- Across 1,100 evaluations of 980 text-level PASS cards, the evaluation-level false-negative rate for blocking errors was 1.00% (95% CI 0.50–1.78%), above the prespecified 0.3% safety threshold.
- Fifth-stage automated feedback was associated with more blocking-error flags and lower technical-accuracy and educational-quality ratings. These changes do not isolate a causal feedback effect.
- A 19-item image pilot found critical errors. Table-versus-infographic comparisons were underpowered and non-significant after multiple-comparison correction, so the image findings remain exploratory.
Interpretation and limits
Safety estimates are conditional on the study's selection and PASS criteria. Reference ratings had low reliability, with rater- and workload-dependent patterns. The single-arm design cannot separate useful cueing from anchoring or re-review effects. The reported error rate should not be generalized to all models or educational domains.
Article and research materials
The open article and official supplementary materials are available through the publisher.
Original publisher abstract
The abstract below is reproduced from the source article. The research note above is not a translation of the abstract or a new publication.
Large language models (LLMs) can generate medical educational content at scale, but multi-dimensional human validation of multi-LLM pipelines has not been performed at scale. We evaluated a 7-stage sequential multi-LLM pipeline implemented exclusively with the Gemini family (Gemini 3 Flash and Pro for text; Gemini 3 Pro Image, the Nano Banana Pro preview, for images), producing 6000 flashcards and 833 infographics for radiology board preparation. Nine residents and eleven attending radiologists evaluated 1284 flashcards across 11 subspecialties in a two-phase design around fifth-stage (S5) feedback. Among 1100 evaluations of 980 text-level PASS cards, the evaluation-level false-negative rate (FNR) for blocking errors was 1.00% (95% CI 0.50–1.78%), exceeding the pre-specified 0.3% safety threshold; card-level (1.12%) and majority-vote (0.82%) analyses converged. S5 feedback was associated with increased blocking-error flags (39–54 evaluation-level observations, McNemar p = 0.003) and decreased technical accuracy and educational quality scores (both p < 0.001). In unadjusted analyses, attending radiologists identified more errors than residents (odds ratio 4.52, 95% CI 2.38–8.57), although workload-matched sensitivity analyses attenuated this gap. A preliminary image pilot of 19 items showed critical errors in both tables and infographics, with between-format comparisons underpowered and not significant after Holm correction; this image-related observation should be interpreted as exploratory. The conditional safety estimates, low-reliability reference standard, and rater- and workload-dependent patterns do not support fully automated deployment; automated feedback was associated with sharpened evaluator scrutiny, though the single-arm design cannot isolate useful cueing from anchoring or re-review effects.
Source: npj Digital Medicine · CC BY-NC-ND 4.0. Abstract copyright remains with the original authors. Original figures are not redistributed.
Citation
Yoojin Nam, Taein An, Sung Il Hwang, Chawoong Jeon, Ji Won Jeong, Juhyun Jeong, Dong Yeong Kim, Pyeong Hwa Kim, So Yeon Kim, Sungkwan Kim, Yangwon Kim, Kwang Hwi Lee, Taehee Lee, Hee Sang Oh, Joon Hyung Park, Seunghyeon Shin, Yongsik Sim, Jeong Min Song, Seunghyun Song, Pa Hong, Namkug Kim. Human-in-the-loop validation of a sequential multi-LLM medical education pipeline. npj Digital Medicine. 2026. https://doi.org/10.1038/s41746-026-02978-8
Related research
Metadata and abstract checked: · Check the source article for subsequent updates and corrections.