Institute for Telecommunication Sciences / Research / Quality of Experience / Video Quality Research / No Reference Metrics / No Reference Metrics

No Reference Metrics

No-reference (NR) metrics are algorithms that predict the quality of an image or video, using only the pixel values. NR metrics are extremely difficult to develop. Most NR metric researchers want to predict compression impairments. The problem is that aesthetics and camera capture influence the mean opinion score (MOS) training data. Thus, aesthetics and camera capture are confounding factors. 

The other types of video quality metrics eliminate aesthetics and camera capture from the training by either 1) using a reference video and a double stimulus subjective test method, or 2) using heuristic, such as the likely quality of H.264 at 1 Mbps. This simplifies the research problem considerably. 

Why No Reference Metrics for Image and Video Quality Lack Accuracy and Reproducibility

By Margaret Pinson, published February 2023

This journal article provides a comprehensive overview of no reference (NR) metrics for image quality analysis (IQA) and video quality analysis (VQA). We examine 26 independent evaluations of NR metrics (previously published) and analyze 32 NR metrics on six IQA datasets and six VQA datasets (new results). Where NR metric developers claim Pearson correlation values between 0.66 and 0.99, our measurements range from 0.0 to 0.63. None of the NR metrics we analyzed are accurate enough to be deployed by industry. Performance evaluations that indicate otherwise are based on insufficient data and highly inaccurate. We will examine development strategies, tools, datasets, root cause analysis, and our baseline metric for collaboration, Sawatch.

NR Metric Framework and NR Metric Sawatch

NRMetricFramework is an open software framework for collaborative development of No Reference (NR) metrics for Image Quality Analysis (IQA) and Video Quality Analysis (VQA). This framework includes the support tools necessary to begin research and avoid common mistakes. The vision is a series of NR-VQA metrics that build upon each other to industry requirements for scope, accuracy, and capability. The code is written in MATLAB®.

  • Support tools that support NR metric developments
  • Looping structures to run NR metrics over multiple datasets of images and videos 
  • Tutorials and documentation 
  • Links to openly available datasets  
  • Independent analyses of NR metrics developed by other organizations
  • The NR metric Sawatch

NR Metric Sawatch was developed by ITS. Sawatch is a series of NR metrics that provide RCA, open source, and fast run speed. The intention is that Sawatch will be updated regularly instead of remaining a fixed, static algorithm. Sawatch is intended for a broad range of modern camera systems and video content. Sawatch assesses image quality and video quality but not transmission errors.

The Sawatch mountain range in central Colorado contains eight of the 20 highest peaks in the Rocky Mountains. Similarly, the Sawatch metric is a collection of NR metrics and RCA algorithms. Mountain climbers tackle increasingly difficult mountains. Similarly, NR metric development is a difficult challenge, and our goal is steady improvement until we achieve the highest levels of performance. ITS welcomes collaboration on improving Sawatch.

2023, New Statistical Methods for Assessing Subjective Tests and NR Metrics 

This white paper describes the motivations behind the new statistical methods for analyzing video quality subjective tests and metrics in this journal article from 2023. 

2025, Missing Factor in NR Metrics: Object Size and Artistic Intent

This white paper describes how NR metrics that analyze artistic intent could be used to identify confounding factor that confound MOSs and limit the accuracy of NR metrics. 

Circa 2020, ITS Resources for No Reference Metric Development

This white paper describes resources provided by ITS to support NR metric development, and briefly explains the motivation for NR metrics. 

Datasets for NR Metric Research

The following datasets were designed specifically for NR metric research. Some of these datasets focus on first responder use cases. 

Publications with Insights on NR Metrics

These publications present ITS findings on best practices for NR metric development and the accuracy of NR metrics published by other researchers. Additional white papers can be found in the documentation folder of the NRMetricFramework.