IEEE Transactions on Multimedia (Early Access), 21 July 2021
Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)
doi: 10.1109/TMM.2021.3098450Pablo Perez; Lucjan Janowski; Narciso Garcia; Margaret H. Pinson
Abstract: Recent studies have shown that it is possible to characterize subject bias and variance in subjective assessment tests. Apparent differences among subjects can, for the most part, be explained by random factors. Building on that theory, we propose a subjective test design where three to four team members each rate the stimuli multiple times. The results are comparable to a high performing objective metric. This provides a quick and simple way to analyze new technologies and perform pre-tests for subjective assessment.
Keywords: subjective quality evaluation; video quality; experiment design; video quality assessment
For technical information concerning this report, contact:
Margaret H. Pinson
Institute for Telecommunication Sciences
(303) 497-3579
mpinson@ntia.doc.gov
To request a reprint of this report, contact:
Lilli Segre, Publications Officer
Institute for Telecommunication Sciences
(303) 497-3572
LSegre@ntia.gov
Disclaimer: Certain commercial equipment, components, and software may be identified in this report to specify adequately the technical aspects of the reported results. In no case does such identification imply recommendation or endorsement by the National Telecommunications and Information Administration, nor does it imply that the equipment or software identified is necessarily the best available for the particular application or uses.