Moving to a Revise and Resubmit Review Process for CHI 2022

Moving to a Revise and Resubmit Review Process for CHI 2022

As an effort to improve paper quality, increase equity, and reflow the reviewing process, CHI 2022 Papers Chairs, Technical Program Chairs, and General Chairs have replaced the paper rebuttal mechanism with a Revise and Resubmit (R&R) reviewing process. R&R is a two-round reviewing process. At the end of a first review round, each article will receive one of the three following decisions: Accept with minor changes, Revise and Resubmit, or Reject. A R&R decision gives authors the chance to revise their submission and go through a second review round. We intend to improve the quality of CHI publication by giving authors an explicit revision cycle within the review process. R&R is also based on informed decision making and a balance of responsibilities for a fairer review process.

Before detailing the rationale for such a change, here is a short reminder of what the review process has been for previous years. Authors submit a paper. The relevant subcommittee chairs (SC) assign this paper to two associate chairs (AC), who become the primary AC (1AC) and the secondary AC (2AC). The 1AC assigns two external reviewers. These external reviewers, as well as the 2AC, write a review anonymously. The 1AC summarizes these assessments in their meta-review. Authors then write a rebuttal, answering reviewers’ concerns and making proposals for possible changes in a revision. At the PC meeting, both the 1AC and the 2AC know the authors’ identities when they decide: accept, reject, or accept with shepherding if the committee asks for more substantial changes. All authors of accepted papers revise their final submission. Shepherded papers have an early deadline for shepherds to check on substantial changes.

As detailed below, moving to a Revise and Resubmit process aims at further establishing CHI as a premier, high-quality venue and at guaranteeing more equity for authors in the reviewing process.

High-Quality Content

CHI articles are considered journal-level publications. An R&R process better matches journals’ reviewing practices. Authors benefit from feedback at the end of the first review round and get a real chance to propose a revised and better presentation of their work. We expect the process will lead to high-quality and impactful contributions.

Informed Decision Making

Possibilities for authors to address concerns raised in their submission used to rely on rebuttals and, sometimes, shepherding.

Rebuttals were promises made to the program committee in a non-anonymous way (the committee knows authors’ identities when they decide at the PC meeting). The committee had to estimate and speculate on how requested changes would be implemented in papers. Here, implicit and explicit biases (e.g., regarding authors submitting for the first time) can quickly shift a conversation. Seeing the changes already implemented by the authors provides the committee with a better ground for making informed decisions; the changes can be judged for what they are.

Shepherding relied on individual ACs. Shepherding a paper means taking on significant responsibility and committing to spend time with the authors on their revision. Some ACs might be more inclined/available to do so than others, meaning that authors’ chances partly rely on ACs’ personalities and own availability at the appropriate time. With an R&R process, evaluation of requested changes is a collaborative effort between ACs and external reviewers.

Balanced Workload and Responsibilities

Evaluation of post-rebuttal changes relied exclusively on the program committee (most often on the 1AC). With R&R, we expect the responsibility and workload of ACs to be better balanced, as the decision and evaluation of the revision are more evenly distributed across committee members and external reviewers. In particular, the second reviewing cycle includes external reviewers back in the loop. Those external reviewers are often the most experts to judge the revision in light of their initial concerns. That way, we can broaden our recruitment strategies for prestigious roles on the committees and enable more people to participate in these roles.

Some Questions

• What kind of revisions are good candidates for a second round? As opposed to a journal reviewing process, R&R for CHI is implemented on a limited time frame. In particular, this means that reviewers are not expected to provide a second full review but should quickly evaluate minor changes. This includes: improving presentation, rewriting a discussion, moderate reframing, providing more details, expanding a bit on existing analyses. However, changes that could significantly change the contribution, such as running an additional study, are considered too significant.
• Will ACs’ load be higher? We expect not, as the reviewing load is distributed among all reviewers, including external ones. We estimate the load to be equivalent to reading rebuttals and engaging discussions about what the final paper might look like based on the rebuttal.
• Will there be papers rejected at the second round? Yes, there will. The R&R process aims at increasing the quality of submissions and fairness to authors, not at augmenting the volume of CHI papers (which is already very difficult to accommodate in a conference program). One might argue that a revision is more effort than a rebuttal and that rejection might be more challenging to handle for authors. This might be the case indeed. However, it is probably preferable to the frustration of not having the chance to demonstrate potential changes. Furthermore, our shared goal as a community is to build high-quality literature. A revised paper will make a better contribution for any venue, including CHI, next CHI, or another HCI venue.
• Will there still be a rebuttal somewhere in the process? No, there is no rebuttal anymore. As mentioned above, no more promises, actual changes.
• Will there still be a PC meeting? Yes, there will be a short PC meeting at the end of each of the two rounds. At the end of the first round, the committee will make one of the three decisions: Accept with Minor Revisions, Revise and Resubmit, Reject. At the end of the second round, the committee will discuss revised submissions and make one of the two decisions: Accept with Minor Revisions, Reject.
• Can people get a visa to attend the conference with this timeline? For the round one accepted papers (a.k.a. fast-tracked), authors will have an extra month from the previous years’ timeline since their accepts will arrive early December. For the R&R papers, there will be more of a rush. To help offset that time crunch, we will be working on some new methods to expedite visa letters to help everyone speed up their paperwork process. We will rely on virtual and hybrid presentations and interactions for all authors who have visa difficulties.
• How were the timeline and deadlines chosen? The ACM publication process takes an incompressible time, which determines the latest deadline in the process. Starting from there, we considered each phase in the process and set its duration by taking into account the workload of authors, reviewers, and chairs. Each individual deadline was chosen with the Equity and Justice chairs to avoid conflicts with major holidays around the world.