March 2023 | Journal Article
Why No Reference Metrics for Image and Video Quality Lack Accuracy and Reproducibility
Cite This Publication
Margaret H. Pinson, “Why No Reference Metrics for Image and Video Quality Lack Accuracy and Reproducibility,” IEEE Transactions on Broadcasting 69, no. 1 (March 2023): 97–117. https://doi.org/10.1109/TBC.2022.3191059
Margaret H. Pinson 
Abstract:
This article provides a comprehensive overview of no reference (NR) metrics for image quality analysis (IQA) and video quality analysis (VQA). We examine 26 independent evaluations of NR metrics (previously published) and analyze 32 NR metrics on six IQA datasets and six VQA datasets (new results). Where NR metric developers claim Pearson correlation values between 0.66 and 0.99, our measurements range from 0.0 to 0.63. None of the NR metrics we analyzed are accurate enough to be deployed by industry. Performance evaluations that indicate otherwise are based on insufficient data and highly inaccurate. We will examine development strategies, tools, datasets, root cause analysis, and our baseline metric for collaboration, Sawatch.
Keywords: video quality; image quality; root cause analysis (RCA); no reference (NR); metric; Sawatch
Related Links:
For technical information concerning this report, contact:
Margaret H. Pinson
Institute for Telecommunication Sciences
(720) 601-7314
mpinson@ntia.gov
For funding information concerning this report, click this link.
Funding Information
Performing Agency
National Telecommunications and Information Administration
Institute for Telecommunication Sciences
<p>325 Broadway</p><p>Boulder, CO 80305</p>
https://ror.org/00mj5bc69
Disclaimer:
Certain commercial equipment, components, and software may be identified in this report to specify adequately the technical aspects of the reported results. In no case does such identification imply recommendation or endorsement by the National Telecommunications and Information Administration, nor does it imply that the equipment or software identified is necessarily the best available for the particular application or uses.
For questions or information on this or any other NTIA scientific publication, contact the ITS Publications Office at ITSinfo@ntia.gov or 303-497-3572.