A Synthetic Prediction Market for Estimating Confidence in Published Work - Rajtmajer, Sarah; Griffin, Christopher; Wu, Jian; Fraleigh, Robert; Balaji, Laxmaan; Squicciarini, Anna; Kwasnica, Anthony; Pennock, David; McLaughlin, Michael; Fritton, Timothy; Nakshatri, Nishanth; Menon, Arjun; et al. (Proceedings of the AAAI Conference on Artificial Intelligence, 2022)
Explainably estimating confidence in published scholarly work offers opportunity for faster and more robust scientific progress. We develop a synthetic prediction market to assess the credibility of published claims in the social and behavioral sciences literature. We demonstrate our system and detail our findings using a collection of known replication projects. We suggest that this work lays the foundation for a research agenda that creatively uses AI for peer review.