Equitable Reviewing Guide

When reviewing, it is your responsibility to our community to provide a fair and equitable review.

Start with the guide to reviewing at CHI on the CHI 2022 website.
A good publication is accessible. Check out this link on the CHI reviewing website. Make sure to let your authors know if they should make their papers, including the figures, tables etc., supplementary materials, videos, previews, and other content more accessible in your review.
Reviewing fairly also means reviewing equitably. If you want a refresher on matters of equity, we recommend the video playlist curated by the CHI 2021 diversity chairs on Youtube (external link).
Many communities within SIGCHI have invaluable specialized guides (e.g. Mobile HCI, SIGCHI Accessibility Guide, Unofficial [crowdsourced] Guide to Reviewing for CHI, Transparency in Qualitative Research: Increasing Fairness in the CHI Review Process).

Equitable reviewing might be a new concept for some of us in the CHI community. As such, here are some quick tips:

Contributions, contributions, contributions! Don’t get caught up in form over function: focus on the core of the work.

The CHI Guide to Reviewing Papers (2022) tells us it is all about contributions including replication!
Many authors are writing a CHI paper in a second or third language, which is a huge feat and has no bearing on the HCI work that has been done, which should be the focus of the review.
The conference is international, contributions are important, English grammar and writing style points are minor review points.

Questions to consider:

Is the writing able to communicate the main ideas?
Can the authors benefit from a revision cycle for writing issues you identified?
Are your writing critiques based on preferences of Western academic standards (e.g. the Oxford comma, grammar minimalism, parallel writing structure)?

Consider the importance of culture as it affects a paper’s communication.

Does the paper only apply to populations in the author’s country? If so, suggest the authors specify their scope, rather than implying that what they found in their region is generalizable to the whole world.

Is the group studied specialized?
Is there a difference of privilege within the scope of the population surveyed?

Transparency of methodology is important!

As a reviewer, look for transparency instead of debating the value of the methodology. Don’t “block” a contribution on the ground of epistemological differences; rather, verify that the authors are clear and precise in presenting their approach(es).

Is the methodology clear?
Could you replicate the study based on what was communicated?

Be honest about your expertise!

If a paper uses a methodology outside of your everyday experiences or expertise, then acknowledge what you do or do not know: both authors and reviewers have positionality! It is important to not discredit or unfairly score a paper based on our own preferences in methodological approaches.

Am I an expert in this area? How specifically?
Do I have a particular stance on the value of qualitative or quantitative research that may impact my ability to fairly review either approach?

Reflect on and communicate your personal biases!

Can you allude to your positionality as it shapes your perspective on the review?
Just like when we are running a study, we must consider all the confounding factors brought to the table when completing a reviewing task.

What does my academic school of thought consider, disregard, and remain unaware of?
Is the paper I am reviewing making assumptions from the perspective of an academic school of thought or with bias?
Which assumptions do I make about the paper that might just be grounded in a lack of information?

Are you concerned about the content of a paper? ASK YOUR AC.

Do not take it upon yourself to “desk reject” a paper.
Identify your concerns and be clear about them.
Remember, you can always add comments to be seen only by the committee.

Does the paper unfairly misrepresent a group?
Is there a group of people whose representation is unfactual or tainted with bias?

Are you concerned about the ethics of the work you are reviewing?

If you have potential concerns on ethics, please contact your SCs and Papers chairs. Papers chairs will further escalate.
The Research Ethics Committee does not advise whether a paper should be rejected or accepted, but rather hopes that such information can contextualize reviewer concerns, and that our feedback can help authors clarify their research choices. One way to think about their role is bringing in an additional reviewer with research ethics expertise.

While the reviewers should point out any ethical concerns and discuss with SCs and Papers chairs (and the Research Ethics Committee), their reviews can focus on judging the magnitude of the contributions and the validity of the work.

Equity is an ongoing issue that affects us all. Even this resource only benefits from community input! Are we forgetting anything or missing any resources? Reach out to us at allyship@chi2022.acm.org!