‘Papers with Code’ / NeurIPS Guidelines for Publishing Research Code: A Role Model For Recommender-Systems?

The recommender-system community is facing a reproducibility crisis. This has recently been demonstrated by the authors of the paper Are we really making much progress? A worrying analysis of recent neural recommendation approaches (Maurizio Ferrari Dacrema, Paolo Cremonesi, Dietmar Jannach). However, the crisis is not new, and has been recognized (at least) a decade ago by Ekstrand et al. (2011) who criticized that “it is currently difficult to reproduce and extend recommender systems research” and that evaluations are “not handled consistently”. Konstan and Adomavicius (2013) added that many research papers “contribute little to collective knowledge”, primarily due to the difficulty of reproducing the results. They concluded

“[T]he Recommender Systems research community is facing a crisis where a significant number of papers present results that contribute little to collective knowledge […] often because the research lacks the […] evaluation to be properly judged and, hence, to provide meaningful contributions.”

Konstan and Adomavicius (2013)

The machine learning community faces similar problems, but in contrast to recommender systems, the machine learning community has some guidelines that potentially alleviate the problem. Particularly noteworthy, in my opinion, are the Tips for Publishing Research Code by ‘Papers with Code’. These tips are now also the official guidelines at the NeurIPS conference, one of the major machine learning conferences (formerly NIPS). Even though NeurIPS is not requiring authors to follow the guidelines, they strongly recommend it.

Frankly, while my group usually publishes its source code on GitHub, we haven’t been following any particular guidelines, and haven’t been too careful about it. Nevertheless, the Tips for Publishing Research Code by ‘papers with code’ are an important step towards more reproducibility, and I believe the recommender-system community should pay more attention to it (including myself).

What do you think? Are such guidelines on publishing code important? Should they be mandatory e.g. for ACM RecSys submissions? Let us know your thoughts in the comments and vote in the poll.

 Loading ...