BMC Medical Education ·

Feasibility and psychometric evaluation of large language model–generated radiology learning materials for board examination preparation: a prospective pilot study

Yoojin Nam, Taein An, Sung Il Hwang, Chawoong Jeon, Ji Won Jeong, Juhyun Jeong, Dong Yeong Kim, Pyeong Hwa Kim, So Yeon Kim, Sungkwan Kim, Yangwon Kim, Kwang Hwi Lee, Taehee Lee, Hee Sang Oh, Joon Hyung Park, Seunghyeon Shin, Yongsik Sim, Jeong Min Song, Seunghyun Song, Pa Hong, Namkug Kim

Pa Hong: co-corresponding author

The online publication date is 15 June 2026. The publisher separately lists the Version of Record date as 26 August 2026.

Research question

What needs checking before an efficacy trial of AI-generated learning materials?

Distributing learning materials is not the same as being ready to test educational efficacy. This pilot provided AI-generated materials to radiology residents preparing for board examinations and evaluated retention, voluntary adoption and the survey instruments intended for a later trial.

Study design

Forty-one residents enrolled in the prospective observational pilot; 26 completed the post-examination survey, and 24 had complete educational change outcome data and were included in the analysis. Among the 24 analysed completers, 15 used the materials and nine did not. Use was self-selected rather than randomized.

Key findings

  • Complete-case analytical retention was 58.5%, below the prespecified 80% threshold. Voluntary adoption among analysed completers was 62.5%.
  • After reverse-worded-item correction, trust-in-AI internal consistency was 0.583, below the conventional 0.70 threshold. The extraneous cognitive load scale showed acceptable reliability.
  • Between-group estimates for cognitive load and self-efficacy were small with wide confidence intervals. They were presented for future study planning, not as evidence for or against educational benefit.

Interpretation and limits

A self-selected comparison of 15 users and nine non-users cannot establish a causal educational effect. Self-reported outcomes, small samples, attrition and short pre-examination exposure limit interpretation. Reports of not encountering errors are not a validated accuracy estimate.

Article and research materials

The article states that de-identified data are available from the corresponding author on reasonable request. This is not presented as a permanently public data or code repository. Supplementary materials are available through the article.

Original publisher abstract

The abstract below is reproduced from the source article. The research note above is not a translation of the abstract or a new publication.

Background

Large language models (LLMs) can generate structured educational content at scale, yet their role in postgraduate radiology training remains untested.

Methods

This pilot study evaluated the feasibility of deploying LLM-generated radiology board preparation materials, the psychometric properties of adapted survey instruments, and exploratory effect-size estimates for sample-size planning of a future multicentre trial. We conducted a prospective observational pilot among 41 radiology residents preparing for the 2026 Korean Radiology Board Examination. A governed seven-stage LLM pipeline generated 6,000 Anki flashcards, 833 infographic summaries, and presentation slides. Materials were distributed freely, and 24 completers self-selected into users (n = 15) and non-users (n = 9). Outcomes included enrollment yield, retention rate, voluntary adoption rate, internal consistency (Cronbach’s alpha) of adapted cognitive load and satisfaction scales, and change scores in extraneous cognitive load (ECL) and self-efficacy (SE), compared using Mann–Whitney U tests with Cohen’s d and 95% confidence intervals.

Results

Of 114 registered candidates, 44 were recruited (38.6%); 41 were enrolled (yield 93.2%); retention was 58.5% (24 of 41), below the prespecified 80% threshold; and voluntary adoption was 62.5%. The ECL scale showed acceptable reliability (alpha = 0.87 baseline, 0.80 post-examination). After reverse-coding one reverse-worded item (E3), the corrected three-item trust-in-AI alpha was 0.583 (bootstrap 95% CI 0.151 to 0.746), below the conventional 0.70 threshold. Between-group differences were non-significant for ECL (d = − 0.28; 95% CI − 1.50 to 0.59) and SE (d = 0.26; − 0.62 to 1.24). Among users, educational quality was rated 3.33 of 5.00, and 86.7% reported encountering no factual inaccuracies. A medium-effect scenario (d = 0.50) would require 160 participants.

Conclusions

Retention (58.5%), the trust-scale alpha of 0.583, and the − 40 Net Promoter Score together indicate that the data-collection protocol, the AI-trust instrumentation, and the user-experience design all require modification before an efficacy trial is justified. Operational deployment was feasible at the levels of enrollment yield and voluntary adoption, but not retention. Adapted cognitive load and satisfaction scales reached acceptable reliability; the trust-in-AI and single-item self-efficacy outcomes did not meet conventional thresholds. Exploratory between-group point estimates were small and unstable across sensitivity strata; they are reported as inputs for sample-size planning and should not be interpreted as evidence for or against educational benefit.

Source: BMC Medical Education · CC BY-NC-ND 4.0. Abstract copyright remains with the original authors. Original figures are not redistributed.

Citation

Yoojin Nam, Taein An, Sung Il Hwang, Chawoong Jeon, Ji Won Jeong, Juhyun Jeong, Dong Yeong Kim, Pyeong Hwa Kim, So Yeon Kim, Sungkwan Kim, Yangwon Kim, Kwang Hwi Lee, Taehee Lee, Hee Sang Oh, Joon Hyung Park, Seunghyeon Shin, Yongsik Sim, Jeong Min Song, Seunghyun Song, Pa Hong, Namkug Kim. Feasibility and psychometric evaluation of large language model–generated radiology learning materials for board examination preparation: a prospective pilot study. BMC Medical Education. 2026;26:1369. https://doi.org/10.1186/s12909-026-09704-8

10.1186/s12909-026-09704-8

Metadata and abstract checked: · Check the source article for subsequent updates and corrections.