--- type: source id: lee-2026-ai-chest-radiographs title: 'Complementary Error Profiles of Human Readers and Large Language Models in Detecting AI-Generated Chest Radiographs: A Comparative Reader Study' authors: - Suji Lee - Jei Youn Park - Yongsik Sim - Youho Myong year: 2026 venue: Journal of Imaging Informatics in Medicine volume: null pages: null doi: 10.1007/s10278-026-02339-7 pmid: '42867583' pmcid: null arxiv_id: null openalex_id: W7221727670 oa_url: https://link.springer.com/content/pdf/10.1007/s10278-026-02339-7.pdf peer_review: peer-reviewed-journal evidence_tier: B design: 'Reader study: 2 radiologists, 2 nonradiologist physicians and 3 frontier LLMs (GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8) classified chest radiographs as real or AI-generated; LLMs re-run 3 months later and with laterality markers/borders masked; frequency-domain baseline classifier for comparison' modality: - image-general stimuli: '160 posteroanterior chest radiographs: 40 real (NIH ChestX-ray14) and 120 synthetic (diffusion and text/LLM-based generation pipelines); 75% synthetic prevalence' n_participants: 4 n_note: 4 human readers (2 radiologists, 2 nonradiologist physicians) plus 3 LLM evaluators topics: - human-detection - human-vs-machine - expertise summary: 'Expert and non-expert physicians and frontier LLMs could not reliably tell AI-generated chest X-rays from real ones: the best human (radiologist, 75.8% balanced accuracy) and best LLM (77.9%) were similar but made different errors, radiologists called 38-45% of real images fake, and a strong frequency-domain detector collapsed on a held-out generator (12% sensitivity).' results: - metric: Top evaluator (Gemini 3.5 Flash) balanced accuracy value: 77.9% note: specificity 100%, sensitivity 55.8% source_location: abstract - metric: Best human reader (radiologist RAD1) balanced accuracy value: 75.8% note: sensitivity 55.0%, specificity 96.7%; paired difference −2.1 points [95% CI −5.8, 1.2] source_location: abstract - metric: 'Human readers: diffusion-generated images detected' value: 82-100% note: all four human readers source_location: abstract - metric: 'Radiologists: real images misclassified as synthetic' value: 38-45% note: '' source_location: abstract - metric: Nonradiologist physicians value: near chance note: '' source_location: abstract - metric: 'LLMs: LLM-generated images detected' value: 32-72% note: '' source_location: abstract - metric: LLM specificity with markers/borders masked value: Gemini 62.5%; GPT-5.5 85.0% note: vs 90-100% in same-day unmasked runs source_location: abstract - metric: Agreement among LLMs (Fleiss' kappa) value: '0.213' note: low interevaluator agreement source_location: abstract - metric: Frequency-domain baseline value: balanced accuracy 98.3% in-distribution; 12% sensitivity on a held-out generator note: '' source_location: abstract moderators: - 'Generator type: diffusion images were caught by humans far more often than LLM/text-pipeline images were caught by LLMs.' - 'Expertise: radiologists outperformed nonradiologists but at the cost of many false alarms on real images.' limitations: - Four human readers - Medical images only - Values from abstract; full text not read - Fixed 75% synthetic prevalence verification: status: verified-abstract confidence: high checked: 2026-10-19 via: OpenAlex abstract record W7221727670 for doi:10.1007/s10278-026-02339-7 (publisher page could not be fetched) url: null cited_in: - MR-0001 updated: '2026-10-19' --- # Complementary Error Profiles of Human Readers and Large Language Models in Detecting AI-Generated Chest Radiographs: A Comparative Reader Study > **Source record** `lee-2026-ai-chest-radiographs` · 2026 · Journal of Imaging Informatics in Medicine > **Evidence tier:** B (peer-reviewed-journal) · **Verification:** verified-abstract (compiler confidence: high) > **Bottom line:** Expert and non-expert physicians and frontier LLMs could not reliably tell AI-generated chest X-rays from real ones: the best human (radiologist, 75.8% balanced accuracy) and best LLM (77.9%) were similar but made different errors, radiologists called 38-45% of real images fake, and a strong frequency-domain detector collapsed on a held-out generator (12% sensitivity). ## Citation {#citation} Suji Lee, Jei Youn Park, Yongsik Sim, Youho Myong (2026). Complementary Error Profiles of Human Readers and Large Language Models in Detecting AI-Generated Chest Radiographs: A Comparative Reader Study. *Journal of Imaging Informatics in Medicine*. https://doi.org/10.1007/s10278-026-02339-7 Verification entry (identifiers, resolvers, claims with source locations): [/data/citations.json#lee-2026-ai-chest-radiographs](/data/citations.json#lee-2026-ai-chest-radiographs) ## Study design {#design} | Field | Value | |---|---| | Design | Reader study: 2 radiologists, 2 nonradiologist physicians and 3 frontier LLMs (GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8) classified chest radiographs as real or AI-generated; LLMs re-run 3 months later and with laterality markers/borders masked; frequency-domain baseline classifier for comparison | | Modality | image-general | | Stimuli | 160 posteroanterior chest radiographs: 40 real (NIH ChestX-ray14) and 120 synthetic (diffusion and text/LLM-based generation pipelines); 75% synthetic prevalence | | Participants (N) | 4 | | N detail | 4 human readers (2 radiologists, 2 nonradiologist physicians) plus 3 LLM evaluators | | Topics | human-detection, human-vs-machine, expertise | ## Results {#results} | Metric | Value | Note | |---|---|---| | Top evaluator (Gemini 3.5 Flash) balanced accuracy | 77.9% | specificity 100%, sensitivity 55.8% | | Best human reader (radiologist RAD1) balanced accuracy | 75.8% | sensitivity 55.0%, specificity 96.7%; paired difference −2.1 points [95% CI −5.8, 1.2] | | Human readers: diffusion-generated images detected | 82-100% | all four human readers | | Radiologists: real images misclassified as synthetic | 38-45% | | | Nonradiologist physicians | near chance | | | LLMs: LLM-generated images detected | 32-72% | | | LLM specificity with markers/borders masked | Gemini 62.5%; GPT-5.5 85.0% | vs 90-100% in same-day unmasked runs | | Agreement among LLMs (Fleiss' kappa) | 0.213 | low interevaluator agreement | | Frequency-domain baseline | balanced accuracy 98.3% in-distribution; 12% sensitivity on a held-out generator | | ## Moderators {#moderators} - Generator type: diffusion images were caught by humans far more often than LLM/text-pipeline images were caught by LLMs. - Expertise: radiologists outperformed nonradiologists but at the cost of many false alarms on real images. ## Limitations {#limitations} - Four human readers - Medical images only - Values from abstract; full text not read - Fixed 75% synthetic prevalence ## Cited in {#cited-in} - [MR-0001](/metareports/human-detection-accuracy.md): Human Detection of Deepfakes: Accuracy, Bias, Confidence, and Training ## Verification {#verification} | Field | Value | |---|---| | Status | `verified-abstract` | | Compiler confidence | high | | Checked on | 2026-10-19 | | Via | OpenAlex abstract record W7221727670 for doi:10.1007/s10278-026-02339-7 (publisher page could not be fetched) | Values on this page have not all been checked against the full text. Confirm them against the primary source before high-stakes use.