Deepfake detection by human crowds, machines, and machine-informed crowds
Source record
groh-2022-crowds-machines· 2022 · Proceedings of the National Academy of Sciences (PNAS)
Evidence tier: B (peer-reviewed-journal) · Verification: verified-abstract (compiler confidence: high)
Bottom line: Ordinary human crowds and the leading DFDC model were similarly accurate but made different mistakes; model-informed participants beat either alone on average, but inaccurate model predictions often lowered their accuracy.
Citation
Matthew Groh, Ziv Epstein, Chaz Firestone, Rosalind Picard (2022). Deepfake detection by human crowds, machines, and machine-informed crowds. Proceedings of the National Academy of Sciences (PNAS), 119(1), e2110013119. https://doi.org/10.1073/pnas.2110013119
Verification entry (identifiers, resolvers, claims with source locations): /data/citations.json#groh-2022-crowds-machines
Study design
| Field | Value |
|---|---|
| Design | Two online studies comparing human crowds with the leading DFDC model; machine-informed condition; face-processing disruption manipulations |
| Modality | video |
| Stimuli | Deepfake Detection Challenge (DFDC) videos |
| Participants (N) | 15016 |
| N detail | total across two studies |
| Topics | human-detection, human-machine-teaming, face-processing |
Results
| Metric | Value | Note |
|---|---|---|
| Crowd vs leading model | similarly accurate | different kinds of mistakes |
| Machine-informed crowd | more accurate than either alone | inaccurate model predictions often decreased accuracy |
| Face-processing disruptions | hurt humans, mostly not the model | implicates specialised face processing |
Limitations
- Self-selected online samples
- 2019-2020 face-swap deepfakes
Cited in
- MR-0001: Human Detection of Deepfakes: Accuracy, Bias, Confidence, and Training
- MR-0002: The Uncanny Valley and Synthetic Media: Does It Exist, What Causes It, and Have Deepfakes Crossed It?
Verification
| Field | Value |
|---|---|
| Status | verified-abstract |
| Compiler confidence | high |
| Checked on | 2026-10-07 |
| Via | arXiv 2105.06496 abstract (journal ref PNAS 119(1)) |
Values on this page have not all been checked against the full text. Confirm them against the primary source before high-stakes use.