The Magnitude of Truth: On Using Magnitude Estimation in Truthfulness Assessment in Fact-Checking

The School of EECS is hosting the following guest seminar:

The Magnitude of Truth: On Using Magnitude Estimation in Truthfulness Assessment in Fact-Checking

Speaker: Professor Stefano Mizzaro, University of Udine, Italy
Host: Prof Gianluca Demartini

Abstract

Assessing the truthfulness of information is a critical task in fact-checking. It is typically performed using binary or coarse-grained ordinal scales with limited levels, usually from 2 to 6, although fine-grained scales, such as those with 100 levels, have been explored. Magnitude Estimation (ME) takes this approach further by allowing assessors to assign any value ranging from 0 (excluded) to infinity. However, ME introduces challenges, including the aggregation of assessments from individuals with varying interpretations of the scale. Despite these challenges, its successful applications in other domains suggest its potential suitability for truthfulness assessment.

I will describe a crowdsourcing study that we conducted to assess the effectiveness of ME in the truthfulness assessment context. We collected assessments from non-experts on claims sourced from the Politifact fact-checking organization. Results show that while aggregation methods significantly impact assessment quality, optimal aggregation strategies yield accuracy and reliability comparable to traditional scales. Moreover, ME allows to capture subtle differences in truthfulness, offering richer insights than conventional coarse-grained scales.

Bio

Stefano Mizzaro is professor at the Department of Mathematics, Informatics, and Physics of the University of Udine, Italy. He has been working for more than 30 years on information retrieval, mainly focusing on effectiveness evaluation. More recently he has also worked on crowdsourcing, artificial intelligence, and misinformation assessment. On these topics he has published more than 150 scientific papers in national and international venues, he has received some grants and awards, and he is currently coordinating the national project “MoT - The Measure of Truth: An Evaluation-Centered Machine-Human Hybrid Framework for Assessing Information Truthfulness”.

Acknowledgements. This research has been partially supported by Progetto PRIN 2022 - The Measure of Truth: An Evaluation-Centered Machine-Human Hybrid Framework for Assessing Information Truthfulness - Codice n. 20227F2ZN3 CUP n. G53D23002800006 ”Finanziato dall'Unione Europea – Next-Generation EU – PNRR M4 C2 I1.1" RS Mizzaro