Warning: Humans cannot reliably detect speech deepfakes
Source record
mai-2023-speech-deepfakes· 2023 · PLOS ONE
Evidence tier: B (peer-reviewed-journal) · Verification: verified-fulltext
Bottom line: Listeners were 70.35% accurate on single clips (73% hit rate on fakes); familiarisation added only ≈3.8 points; crowd-aggregated judgments rivalled in-domain detectors, while out-of-domain detectors failed.
Citation
Kimberly T. Mai, Sergi Bray, Toby Davies, Lewis D. Griffin (2023). Warning: Humans cannot reliably detect speech deepfakes. PLOS ONE, 18(8), e0285333. https://doi.org/10.1371/journal.pone.0285333
Verification entry (identifiers, resolvers, claims with source locations): /data/citations.json#mai-2023-speech-deepfakes
Study design
| Field | Value |
|---|---|
| Design | Online listening experiment (Prolific); unary (single clip) and binary (pick the fake from a pair) tasks; with vs without familiarisation (5 example deepfakes); English and Mandarin |
| Modality | audio |
| Stimuli | VITS text-to-speech deepfakes vs genuine speech (LJSpeech; Chinese Standard Mandarin Speech Corpus); 50 sentences per language, 100 clips, 2-11 s |
| Participants (N) | 529 |
| Topics | human-detection, voice, training, human-vs-machine |
Results
| Metric | Value | Note |
|---|---|---|
| Unary accuracy (single clip) | 70.35% | deepfakes identified 73%; genuine 67.78% |
| Binary accuracy (pick fake from pair) | 85.59% | |
| Familiarisation effect | +3.84 points (p=0.001) | unary +3.76, binary +3.85 |
| Confidence-weighted AUROC | 73.83% → 75.54% with familiarisation | |
| Crowd-aggregated AUROC | 95.51% (94.04% with familiarisation) | comparable to in-domain detectors |
| Automated detectors AUROC | in-domain 100%; out-of-domain 25.31% | out-of-domain flagged genuine speech as fake |
| Mandarin vs English listeners | +1.79 points, n.s. (p=0.202) |
Moderators
- Reference audio of the same sentence helped significantly (p<0.001) but is rarely available in practice.
- Extra time and repeated listening did little to improve performance.
Limitations
- Older TTS (VITS), not current voice cloning
- Short clips
- Participants knew fakes were present
Cited in
- MR-0001: Human Detection of Deepfakes: Accuracy, Bias, Confidence, and Training
- MR-0003: Voice Clones and Synthetic Speech: How Well Do People Detect Them?
Verification
| Field | Value |
|---|---|
| Status | verified-fulltext |
| Compiler confidence | high |
| Checked on | 2026-10-07 |
| Via | journals.plos.org article page |