Human detection of political speech deepfakes across transcripts, audio, and video
Source record
groh-2024-political-speech· 2024 · Nature Communications
Evidence tier: B (peer-reviewed-journal) · Verification: verified-fulltext
Bottom line: Accuracy rose with each added modality (transcript 57% → silent video 64% → audio 81% → video+audio 85-86% in Exp 1a); people relied more on how something was said than on what was said, and TTS fakes were harder to spot than voice-actor fakes.
Citation
Matthew Groh, Aruna Sankaranarayanan, Nikhil Singh, Dong Young Kim, Andrew Lippman, Rosalind Picard (2024). Human detection of political speech deepfakes across transcripts, audio, and video. Nature Communications, 15, 7629. https://doi.org/10.1038/s41467-024-51998-z
Verification entry (identifiers, resolvers, claims with source locations): /data/citations.json#groh-2024-political-speech
Study design
| Field | Value |
|---|---|
| Design | 5 pre-registered experiments; real vs fabricated political speeches shown as transcript, silent video (± subtitles), audio (± subtitles), or video+audio (± subtitles) |
| Modality | text, audio, video, audiovisual |
| Stimuli | Real and fabricated political speeches; fabricated audio from state-of-the-art TTS or voice actors |
| Participants (N) | 2215 |
| N detail | 2,215 recruited participants across 5 experiments, plus 41,313 non-recruited participants in Exp 1b (Exp 1a recruited n=501) |
| Topics | human-detection, modality, voice |
Results
| Metric | Value | Note |
|---|---|---|
| Exp 1a (recruited, n=501): transcript | 57% | content alone ≈ slightly above chance |
| Exp 1a: silent video | 64% | |
| Exp 1a: audio | 81% | with or without subtitles; Discussion reports 80.5% |
| Exp 1a: video + audio | 85% (with subtitles), 86% (without) | |
| Exp 1b (non-recruited, n=41,313): transcript / silent video / audio / video+audio | 43% / 65% / 74% / 82-83% | |
| Exp 3: transcript / subtitled video / audio / video+audio | 58% / 62% / 65% / 74% | same ordering at lower levels |
| Exp 2: TTS vs voice-actor deepfakes identified | 72% vs 83% | state-of-the-art TTS harder to detect |
| Cue reliance | delivery (audio-visual) > content | authors' conclusion |
Moderators
- Participants struggled to distinguish a speaker's real audio from TTS trained on that speaker.
Limitations
- Political figures only (familiar speakers may aid detection)
- Participants knew fakes were present
Cited in
- MR-0001: Human Detection of Deepfakes: Accuracy, Bias, Confidence, and Training
- MR-0003: Voice Clones and Synthetic Speech: How Well Do People Detect Them?
Verification
| Field | Value |
|---|---|
| Status | verified-fulltext |
| Compiler confidence | high |
| Checked on | 2026-10-07 |
| Via | nature.com article page (open access); author list on page truncated, middle authors from compiler knowledge |