Bounds on Agreement Between Subjective and Objective Measurements

Jaden Pieper () and Stephen D. Voran ()

Abstract:

Objective estimators of multimedia quality are typically judged by comparing estimates with subjective “truth data,” most often via Pearson correlation coefficient (PCC) or mean-squared error (MSE). But subjective test results contain noise, so striving for a PCC of 1.0 or an MSE of 0.0 is neither realistic nor repeatable. Numerous efforts have been made to acknowledge and appropriately accommodate subjective test noise in objective-subjective comparisons, typically resulting in new analysis frameworks and figures-of-merit. We take a different approach. By making only the most basic assumptions, we are able to derive bounds on the PCC and MSE that can be expected for a subjective test. Consistent with intuition, these bounds are functions of subjective vote variance. When a subjective test includes vote variance information, the calculation of the bounds is straight-forward, and we say the resulting bounds are “fully data-driven.” We derive these data-driven PCC and MSE bounds for 18 subjective tests. We also provide two options for calculating bounds in cases where vote variance information is not available. One option is to use vote variance information from other subjective tests. The second option is to use a model for subjective votes, and we introduce a binomial-based model for this purpose. We use both approaches to derive bounds and compare them with the data-driven bounds. These results set expectations for the achievable PCC and MSE for any subjective test, even those where vote variance information is not available.

Keywords: correlation; subjective test; objective estimation; objective estimator; mean-squared error; binomial distribution; rating model; voting model

Disclaimer:

For questions or information on this or any other NTIA scientific publication, contact the ITS Publications Office at ITSinfo@ntia.gov or 303-497-3572.

Back to Search Results