Advice on Reviewing for EMNLP — and ACM RecSys? [EMLP Blog]

The intention for this post is to provide some advice to reviewers, such that we can identify the best research to be presented in the conference, and provide constructive feedback in order for authors to further improve their papers. We recognize and appreciate the amount of efforts reviewers have contributed, and hope to make that more beneficial. We want all the authors to feel the delight when they read the peer reviews for their papers.

This is not the first attempt to educate reviewers. Many major conferences have included advice to reviewers, in NLP and in other fields, and there’s also plentiful advice relating to journal reviewing. Within the field of NLP, we would highlight:

* Discursive advice in ACL 2017 from leading lights in the field: Mirella Lapata, Marco Baroni, Yoav Artzi, Emily Bender, Joel Tetreault, Ani Nenkova, and Tim Baldwin

* Two example good reviews from NAACL 2018 presented in their reviewing form

* A podcast by Noah Smith about peer reviews

Please take the time to look through these excellent resources.

We hope reiterating some dos and don’ts here can help reviewers as well as authors.

First, evaluate the paper’s contributions. This is where you will use your NLP domain knowledge. We advise that you should not accept papers just because their reported results are better, or that they appear to be mathematically sophisticated. These are not sufficient or necessary to constitute contributions. And we advise that you should not reject papers just because their results are not better than state-of-the-art. In the previous *ACL conferences, some reviewers placed too much emphasis on SOTA performance, giving low scores to any systems that failed to reach that. While we aim to publish the very best work, a more constructive question to ask is “state of which art?“. As discussed in this blog post, a paper could offer a step forward in terms of efficiency, generalizability, interpretability, and many other criteria. A convincing contribution of any kind should not be rejected only for not topping the leaderboards.

https://2020.emnlp.org/blog/2020-05-17-write-good-reviews