August 2023.- Since 1974, experts in data science, machine learning, and natural language processing, among other areas, have gathered at the European Conference on Artificial Intelligence (ECAI). The 2023 edition of this event, considered one of the three largest conferences in the world in its field, will take place in Krakow, Poland, and will feature the presentation of various papers, including one developed by researchers from the Institute of Mathematical and Computational Engineering (IMC) at the Pontifical Catholic University of Chile and researchers from the Millennium Institute for Data Foundations.

The accepted paper for ECAI 2023 – to be held between September 30th and October 5th – is titled “No Agreement Without Loss: Learning and Social Choice in Peer Review.” Its authors are Pablo Barceló – director of IMC UC and researcher at the Millennium Institute for Data Foundations (IMFD) – Mauricio Duarte – Ph.D. in Mathematics from U. Andrés Bello – Cristóbal Rojass – academic at IMC UC and researcher at the National Center for Artificial Intelligence (Cenia) – and Tomasz Steifer, international postdoctoral researcher at IMC UC and IMFD.

As the researchers state in the paper presentation, in the peer review systems through which scientific studies pass before being published, reviewers are often asked to evaluate various characteristics of the works, such as technical quality or novelty. A score is assigned to each of the predefined characteristics, and based on these, the reviewer has to provide an overall quantitative evaluation. Tomasz Steifer, a doctor in computer science from the Institute of Computer Science at the Polish Academy of Sciences, explains that different factors, known as “biases” in some cases, influence this process and introduce an element of arbitrariness.

For example, he indicates that the so-called commensuration bias arises when reviewers “are asked to evaluate different and incomparable characteristics, such as novelty and technical correctness, and then combine those incomparable scores into a general score. This final score often serves as a recommendation for acceptance or rejection.” In this sense, the arbitrariness stems from the fact that reviewers “differ in terms of the importance they assign to different characteristics.” One could have the misfortune of receiving a low overall score because their reviewers prefer function B over function A. Nevertheless, we would like to believe that the review process is not about luck but solely about the quality of an article. That is why frameworks like the one proposed by Noothigattu, Shah, and Procaccia offer a very tempting promise to improve the review system, making it less arbitrary, fairer for authors, and better at selecting good articles.”

Fuente: Comunicaciones IMC UC