Crowdsourcing Relevance Magnitudes

Magnitude estimation is a psychophysical scaling technique for the measurement of sensations, where observers assign numbers to stimuli in response to their perceived intensity. I will report some results of a study on the use of magnitude estimation for judging the relevance of documents in the context of information retrieval evaluation. This large scale study involved collecting more than 50,000 magnitude estimation relevance judgments by means of crowdsourcing.
The analysis shows that, perhaps surprisingly, crowdsourcing did work, allowing us to gather good quality data. On average, magnitude estimation judgments are rank-aligned with ordinal judgments made by expert relevance assessors. An advantage of magnitude estimation is that users can chose their own scale for judgments, allowing deeper investigations of user perceptions than when categorical scales are used: I will also talk about applying magnitude estimation for calibrating gains in gain-based effectiveness metrics, like nDCG and ERR, directly from user-reported perceptions of relevance. Finally I will also anticipate some results on analyzing free text comments provided by the workers.

Stefano Mizzaro is Associate Professor at the University of Udine, Italy, since 2006. His main research interests for the last 20 years have been (and still are) information retrieval evaluation, mobile and context-aware retrieval, and scholarly publishing and peer review. After publishing is 100th paper, he stopped counting them. He is co-founder of MoBe Ltd., a spinoff company dealing with software and services for mobile devices. In 2014 he spent his sabbatical at RMIT, Melbourne; now he is back home enjoying the good old world