APSA 2025: Validation and Generative AI in Political Science Working Group

This working group is co-organized by Political Scientists from BYU’s AI and Social Science lab, including Josh Gubler, Lisa Argyle, and Ethan Busby. It’s intended as a step towards developing a set of validation practices and standards for working with generative AI tools in political science. Work using generative AI tools in political science has dramatically increased, but researchers still lack a set of best practices to validate and justify the use of generative AI tools in their many applications. This leaves scholars, reviewers, funders, and the discipline at large unsure how to evaluate the quality of research using generative AI in political science. There are some isolated discussions of these ideas (mostly focused on model development and selection, see Spirling 2023, Ollion et al. 2024, Lyman et al. 2025, Argyle et al. forthcoming) but as of yet, there is no coordination in the discipline about how to best approach the use of these tools to promote better scientific inquiry. We intend for the scope of this conversation to include best practices and evaluation standards related, but not limited, to: extent and disclosure of AI use in the research process, transparency and replicability, metrics and benchmarks for task-specific performance, and model selection. While we expect such an effort to require an ongoing and multi-pronged conversation, we view this working group as an early step towards constructing such practices, promoting them in the discipline, and constructing a community to work through and develop these standards and ideas.