On the Process of Area Chair Selection

The goal of this post is to share with you the outcome of our experiment on
area chair crowdsourcing experiment we conducted this year. There were
several reasons for trying the new model. First, the recent report from the
ACL exec about serious diversity issues in our community was an eye opener.
I believe that many of the concerning points in that report  were related to
the standard practice of nominating people with whom we are personally
familiar. While there is nothing wrong with this concept per se, it leads to a
coterie of a privileged few that all know each other.  AC is selected
this way tend to be very responsive, since there is a personal relationship at stake. But it is overly limited in its reach in finding qualified and passionate area chairs. I have to admit that  this was my guiding principle for selecting  ACs for EMNP 2011, which I co-chaired. As our community grows, this practice introduces significant selection bias. Second, as a
community we are blessed with many people who will be excellent ACs.  The
only thing that was missing is the formal means for bringing these
candidates to the attention of the program chairs. While emailing the program committee is  one solution, in practice very few people exercise it. In fact, we only received
two emails with AC nominations before we issued the call. These among other
reasons motivated us to start our experiment.

I am happy to report that the results exceeded our expectations. We received
192 nominations.  From this strong pool, we selected 44 (22%) candidates, which
constitute 72% of our current AC chairs. In addition, Min and I nominated additional area chairs, depending on the need of specific areas.  Our main criteria for selecting area chairs was technical excellence and reviewing experience.  We also aimed to take into
account geographical and gender diversity, degree of seniority, academic vs
industry background, and prior AC experience. It was a complex optimization
problem to solve, but we are very satisfied with the results. The attached
tables illustrate the distribution of ACs according to these different axes.

Screen Shot 2017-01-14 at 1.35.06 PM.png

Thank you all for  participating in this experiment!   We are glad with outcomes, but let’s
wait until the show is over to judge its impact on the scientific program. We would like to thank all the community members who put the nominations, and to express our gratitude to area chairs in advance for the hard work that comes in coordinating the scientific program!

9 thoughts on “On the Process of Area Chair Selection

  1. Thanks for the new process and for sharing these data! One question about the graphs: does “accepted” refer to your acceptance of a nomination, or to the nominee’s acceptance of the job? That is, do the blue bars show the distribution of invited ACs, or the distribution of ACs who are actually serving?

    Liked by 1 person

    1. “Accepted” shows the distribution of ACs who are actually serving! Jason, thank you for catching it.

      Like

  2. “Our main criteria for selecting area chairs was technical excellence and reviewing experience.”
    These factors are so crucial that they deserve to be described in slightly more details…

    Like

    1. Dear anonymous,

      There is indeed a subjective element when one assesses these qualities. At minimum, the candidate needs to have papers in top NLP conferences, be active in his/her area, and have significant reviewing history. For areas that I coordinated, I went over recent publications of the candidates. We also aimed to make sure that area chairs have complementary expertise, so that they can handle a wide variety of papers from the area.

      Like

      1. Thank you for your reply and the great efforts!
        Hopefully the potential bias from a few inexperienced area chairs can be kept at its minimum.

        Like

      2. Actually I really like the idea for promoting diversity, but there exists a slight concern that quite a number of selected ACs have not published any ACL/TACL paper that belongs to the area he/she will be co-chairing. It might be better for program chairs to spend more time on such areas during the final decision process, to ensure the quality of judgements.

        Like

      3. We appreciate your candid feedback. We will be definitely putting much effort to ensuring the rigorousness of the program. We believe we have carefully vetted the AC pool and have full confidence in their ability to make the right decisions.

        Liked by 1 person

Comments are closed.
https://acl2017.wordpress.com/2017/01/14/on-the-process-of-area-chair-selection/
acl2017.wordpress.com
crawl=CC-MAIN-2022-21
96678
application/http; msgtype=response
sha1:CFWQGB63RGPP4X5UJ7YYTINWTEMM4JM2
<urn:uuid:85f328e0-346d-4811-8372-3878412252ee>
2022-05-19T08:22:19Z
192.0.78.12
text/html
sha1:36JZPENEGT5QIFDFZ3PSD5XQMJGQLQWK
<urn:uuid:332205d9-0977-4d9d-a3ba-70c48e144632>
https://acl2017.wordpress.com/2017/01/14/on-the-process-of-area-chair-selection/
response
<urn:uuid:565982e2-0b6a-4f28-8193-17e2894632bc>
isPartOf: CC-MAIN-2022-21
publisher: Common Crawl
description: Wide crawl of the web for May 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-190
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/