Margaret H. Pinson

Abstract: This report describes a methodology that measures the precision of objective metrics that assess image quality, video quality, speech quality, or the overall audiovisual quality. We assess the confidence intervals of 60 subjective tests and use a confusion matrix to classify the conclusions reached when two subjective test labs perform the same experiment. This allows us to compute the metric’s confidence interval and, when confidence intervals are used to make decisions, to prove whether the metric performs similarly to a subjective test with 15 or 24 subjects. When confidence intervals are not used, the metric’s precision is likened to a certain number of people in an ad-hoc quality assessment. The methods in this report are developed and evaluated using speech quality, video quality, image quality, and audiovisual quality datasets.

Keywords: video quality; speech quality; image quality; audiovisual quality; metric; subjective test; confidence interval

Disclaimer: Certain commercial equipment, components, and software may be identified in this report to specify adequately the technical aspects of the reported results. In no case does such identification imply recommendation or endorsement by the National Telecommunications and Information Administration, nor does it imply that the equipment or software identified is necessarily the best available for the particular application or uses.

For questions or information on this or any other NTIA scientific publication, contact the ITS Publications Office at ITSinfo@ntia.gov or 303-497-3572.

Back to Search Results