By Sayash Kapoor, Mihir Kshirsagar, and Arvind Narayanan
Our response to the National AI Research Resource RFI highlights the significance of supporting a research infrastructure that is designed to independently test the validity of the claims of AI performance. In particular, we draw attention to the widespread phenomenon of the industry peddling what we call “AI snake oil” — promoting an AI solution that cannot work as promised. Relatedly, we highlight how AI-based scientific research is often plagued by overly optimistic claims about its results and suffers from reproducibility failures. We also offer suggestions on how the NAIRR can promote responsible data stewardship models. We recommend that the Task Force’s implementation roadmap include establishing a public infrastructure that can critically evaluate AI performance claims as that is vital to ensuring that AI research serves our shared democratic values. Note, the AI Task Force has extended the deadline for submitting public responses to October 1, 2021.
