How to benchmark objective quality metrics from paired comparison data?

Philippe Hanhart (1) , Lukáš Krasula (2, 3) , Patrick Le Callet (3) , Touradj Ebrahimi (1)

1 Multimedia Signal Processing Group
2 CTU - Czech Technical University in Prague
3 IRCCyN - Institut de Recherche en Communications et en Cybernétique de Nantes

The procedures commonly used to evaluate the performance of objective quality metrics rely on ground truth mean opinion scores and associated confidence intervals, which are usually obtained via direct scaling methods. However, indirect scaling methods, such as the paired comparison method, can also be used to collect ground truth preference scores. Indirect scaling methods have a higher discriminatory power and are gaining popularity, for example in crowdsourcing evaluations. In this paper, we present how the classification errors, an existing analysis tool, can also be used with subjective preference scores. Additionally, we propose a new analysis tool based on the receiver operating characteristic analysis. This tool can be used to further assess the performance of objective metrics based on ground truth preference scores. We provide a MATLAB script with an implementation of the proposed tools and we show one example of application of the proposed tools.

Domaines: Traitement du signal et de l'image [eess.SP]; Traitement des images [eess.IV]; Performance et fiabilité [cs.PF]; Multimédia [cs.MM]

Philippe Hanhart, Lukáš Krasula, Patrick Le Callet, Touradj Ebrahimi. How to benchmark objective quality metrics from paired comparison data?. 8th International Conference on Quality of Multimedia Experience (QoMEX), Jun 2016, Lisbon, Portugal. ⟨10.1109/QoMEX.2016.7498960⟩. ⟨hal-01395449⟩