Institute for Telecommunication Sciences / Research / Quality of Experience / Video Quality Research / Reduced Reference Metrics / History of RR Metrics
A Quick History of ITS Research on RR Metrics
By Margaret Pinson, July 2025
The video quality research project began as a line item in the U.S. Department of Commerce 1989 budget. Television broadcasters were assessing analog video links by pointing their camera at static prints, like the Porta-Pattern resolution chart shown below. These static images did not work for the nascent digital video technology.
The research proposal was inspired by two prior ITS research projects. The study of data transmission quality began circa 1973 and culminated in the publication of "User-oriented performance evaluation of data communication" and the associated measurement software. The study of audio quality began circa 1981 and continues to address open questions in digital speech and audio quality assessment, enhancement, compression, and transmission.
The initial research goal was to develop replacement metrics that could be used real-time, for in-service systems. Therefore, research focused on reduced reference (RR) metrics. Most people used these RR metrics as full reference (FR) metrics, due to the difficulty of simultaneously accessing both the original and impaired video streams at the same time.
ITS research on RR metrics is best known for the NTIA General model (2002), which is referred to as Video Quality Model (VQM) in literature. In 2011, ITS released an updated video quality model with variable frame delay (VQM_VFD). The VQM software repository provides open source software for these and several other RR and FR metrics.
Nearly half of ITS research on RR metrics was devoted to detecting and removing impairments that people cannot see and, as such, do not impact subject ratings. Examples include dynamic changes to temporal alignment, spatial shifts up to 10%, spatial scaling up to 10%, additive changes to luma gain, multiplicative changes to luma offset, and invalid pixels at the edge of the video that are hidden in a television's overscan region. These calibration impairments, if not removed, will cause RR and FR metrics to produce random values.
Early video codecs (1988 to around 2010) were particularly prone to calibration problems. These issues impacted all video codecs, from software-only to hardware video coders with analog elements. We cannot comment on the likelihood of calibration problems in modern video codecs, since ITS research on RR metrics was discontinued in 2011.
For more insights into calibration, see this report. However, be aware that these early codecs exhibited very different temporal patterns than modern video codec. Today, video delay is typically constant, with large but infrequent changes due to rebuffering events. When the algorithms in this report were developed, the video system delay and the frequency of frame updates both changed dynamically in response to changes in the video's motion and spatial complexity. These delay patterns can be observed by downloading the T1A1 dataset or the Video Quality Experts Group (VQEG) Full Reference Phase II dataset from the Consumer Digital Video Library (CDVL).
Early ITS research on video quality also included sponsoring VQEG, sponsoring CDVL, developing the spatial information (SI) and temporal information (TI) metrics in ITU-T Rec. P.910 (see this ITU Contribution), involvement in VQEG metric validation tests (both as a metric proponent and as an independent lab), and supporting international standards development.