Institute for Telecommunication Sciences / Research / Quality of Experience / Video Quality Research / No Reference Metrics / New Statistical Methods

New Statistical Methods

By Margaret Pinson, July 2025

The same question arises during every ITU discussion when deciding which metrics (if any) to include in international standards. How good is good enough?

None of the traditional statistical methods answer this question (e.g., Pearson correlation, root mean square error, and outlier ratio). Pragmatically, we typically compare new metrics to peak signal to noise ratio (PSNR). International experts agree that PSNR cannot be justified as a baseline, since PSNR is not a perceptual metric, but no better option exists. The Motion Picture Experts Group (MPEG) relies on PSNR during video codec development, so the advantages and flaws of PSNR are well understood. 

This journal article from 2023 defines new statistical methods that measure and compare the decisions reached by subjective tests and metrics that assess media quality. The article provides three statistical methods that characterize the expected precision and repeatability of subjective tests. Expected values for these statistics become the basis of three statistical methods that equate the performance of metrics to subjective tests. 

The conceptually simplest of these new statistical methods compares an NR metric to an equivalent number of people. For example, an NR metric is good enough when it has an error rate equivalent to one person.