When will the ACM RecSys Conference have a rebuttal phase (again)?

Update 2024-02-26: I just learned that the ACM RecSys Conference 2024 will also have a rebuttal period. Awesome!

NeurIPS has it, the AutoML Conference has it, CHI has it (kind of), and many other conferences as well — a ‘rebuttal’ or ‘author’s response’ phase. The rebuttal phase of a conference takes place after the reviewers make their comments and suggestions regarding acceptance or rejection. The rebuttal phase allows authors to respond to the criticisms, questions, and suggestions raised by reviewers during the evaluation of their research papers. During this phase, authors can defend their methodology, clarify misunderstandings, and provide additional data or context, if necessary, to address the reviewers’ comments. Sometimes, authors are allowed to submit only a single response to the reviewers, other conferences allow a real discussion, e.g., via OpenReview.

The ACM RecSys conference, and other major conferences for recommender systems research, like SIGIR, WSDM, KDD, or CIKM, typically don’t have a regular rebuttal phase to the best of my knowledge. A rebuttal phase is occasionally discussed by the steering committees (e.g., in 2016 for SIGIR). I believe ACM RecSys 2021 in Seattle also had a discussion phase as the first and only RecSys conference, but a rebuttal was never introduced to the review process regularly. Criticism of a rebuttal phase includes the increased workload for reviewers or that “They never make a difference, and they add to the frustration of all. Reviewers just don’t change their minds.”

I disagree, and I would like to argue why I believe a rebuttal phase is a positive thing. And, why the ACM RecSys Conference should introduce it to the review process on a regular basis. I am sure I am not the first one to suggest this, and I am sure at least some will disagree with my statements. Tell me in the comments what you think.

I would like to base my argument on an example of two papers that a Ph.D. student of mine submitted to the AutoML Conference 2023 (I was a co-author). One of the papers initially received an overall borderline recommendation, and the other was reviewed negatively. After the rebuttal, several reviewers changed their recommendation, and eventually, both papers were accepted.

The reasons for the initially negative reviews were manifold.

Some criticism was related to shortcomings in our writing. For instance, two reviewers did not see the main contribution of our work and failed to understand parts of the methodology. However, this issue was quickly addressed by an explanation to the reviewers and by adjusting the manuscript.

Other criticism related to relatively minor issues — issues that justify a rejection but that we could fix very quickly. For instance, we had forgotten to provide confidence intervals in one chart. And, we had not provided results for a particular metric that one reviewer wanted to have. We could fix these issues within minutes.

Some criticism, by one reviewer in particular, was just plain wrong. The rebuttal phase gave us the opportunity to make it clear to the Senior PC (or ‘area chair’) that the criticism was not justified. Without our response, the Senior PC member, who was supposed to write a meta-review and make a final recommendation, would not have been able to identify that the criticism was wrong (unless he had read the entire paper himself, which most Senior PCs won’t do and shouldn’t do).

I also don’t believe that a rebuttal phase increases the workload — in the long run. If our paper had been rejected because of issues that are easy to fix, we would have re-submitted the manuscript to another conference. Then, three or even four, reviewers would have had to review the entire manuscript again. In contrast, the rebuttal increased the workload only slightly for the existing reviewers.

In the end, the rebuttal phase improved the quality of our manuscript, improved our satisfaction with the review process and the conference, and, in the long run, decreased work load for reviewers.

What do you think about a rebuttal? Let us know in the comments.