The cost of reproducibility in artificial intelligence.

Abstract

Insufficiently documented publications can make independent reproduction an unnecessarily laborious task. We define the cost of reproducibility as the resources required to reproduce the findings of a study and we propose five dimensions of cost, using objective characteristics such as code and data. We review 918 publications between 2022 and 2024 at top AI venues to evaluate the variation in reproducibility costs, with the hypothesis that publications at venues with stricter standards are less costly. Papers published at machine learning conferences are less costly to reproduce than publication from other venues, 12.91% less costly compared with journals, and 16.52% less costly compared with general AI conferences. Award-winning papers, however, are not less costly to reproduce than regular papers, and 6.72% of implementation repository links were broken. These findings suggest that reproducibility costs can be significantly reduced through compliance-based mechanisms-provided they receive community support. We recommend that venues require the inclusion of reproducibility checklists in their publications, permit appendices for comprehensive documentation, and adopt low reproducibility cost as a key criterion for awards.