Complementary Error Profiles of Human Readers and Large Language Models in Detecting AI-Generated Chest Radiographs: A Comparative Reader Study
Source record
lee-2026-ai-chest-radiographs· 2026 · Journal of Imaging Informatics in Medicine
Evidence tier: B (peer-reviewed-journal) · Verification: verified-abstract (compiler confidence: high)
Bottom line: Expert and non-expert physicians and frontier LLMs could not reliably tell AI-generated chest X-rays from real ones: the best human (radiologist, 75.8% balanced accuracy) and best LLM (77.9%) were similar but made different errors, radiologists called 38-45% of real images fake, and a strong frequency-domain detector collapsed on a held-out generator (12% sensitivity).
Citation
Suji Lee, Jei Youn Park, Yongsik Sim, Youho Myong (2026). Complementary Error Profiles of Human Readers and Large Language Models in Detecting AI-Generated Chest Radiographs: A Comparative Reader Study. Journal of Imaging Informatics in Medicine. https://doi.org/10.1007/s10278-026-02339-7
Verification entry (identifiers, resolvers, claims with source locations): /data/citations.json#lee-2026-ai-chest-radiographs
Study design
| Field | Value |
|---|---|
| Design | Reader study: 2 radiologists, 2 nonradiologist physicians and 3 frontier LLMs (GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8) classified chest radiographs as real or AI-generated; LLMs re-run 3 months later and with laterality markers/borders masked; frequency-domain baseline classifier for comparison |
| Modality | image-general |
| Stimuli | 160 posteroanterior chest radiographs: 40 real (NIH ChestX-ray14) and 120 synthetic (diffusion and text/LLM-based generation pipelines); 75% synthetic prevalence |
| Participants (N) | 4 |
| N detail | 4 human readers (2 radiologists, 2 nonradiologist physicians) plus 3 LLM evaluators |
| Topics | human-detection, human-vs-machine, expertise |
Results
| Metric | Value | Note |
|---|---|---|
| Top evaluator (Gemini 3.5 Flash) balanced accuracy | 77.9% | specificity 100%, sensitivity 55.8% |
| Best human reader (radiologist RAD1) balanced accuracy | 75.8% | sensitivity 55.0%, specificity 96.7%; paired difference −2.1 points [95% CI −5.8, 1.2] |
| Human readers: diffusion-generated images detected | 82-100% | all four human readers |
| Radiologists: real images misclassified as synthetic | 38-45% | |
| Nonradiologist physicians | near chance | |
| LLMs: LLM-generated images detected | 32-72% | |
| LLM specificity with markers/borders masked | Gemini 62.5%; GPT-5.5 85.0% | vs 90-100% in same-day unmasked runs |
| Agreement among LLMs (Fleiss' kappa) | 0.213 | low interevaluator agreement |
| Frequency-domain baseline | balanced accuracy 98.3% in-distribution; 12% sensitivity on a held-out generator |
Moderators
- Generator type: diffusion images were caught by humans far more often than LLM/text-pipeline images were caught by LLMs.
- Expertise: radiologists outperformed nonradiologists but at the cost of many false alarms on real images.
Limitations
- Four human readers
- Medical images only
- Values from abstract; full text not read
- Fixed 75% synthetic prevalence
Cited in
- MR-0001: Human Detection of Deepfakes: Accuracy, Bias, Confidence, and Training
Verification
| Field | Value |
|---|---|
| Status | verified-abstract |
| Compiler confidence | high |
| Checked on | 2026-10-19 |
| Via | OpenAlex abstract record W7221727670 for doi:10.1007/s10278-026-02339-7 (publisher page could not be fetched) |
Values on this page have not all been checked against the full text. Confirm them against the primary source before high-stakes use.