Institute for Telecommunication Sciences / Research / Quality of Experience / Video Quality Research / Objective Metrics
Objective Measures of Transmission Quality
Propagation modeling, interference mitigation, and spectrum sharing schemes all speak to ensuring reliable radio channels between nodes so that users may successfully communicate data, speech, video, or other media signals. But layers of protocols that facilitate sharing, robustness, encoding, and decoding often mean that users’ perceptions of the communications experience vary significantly from what physical measurements of the condition of the radio channel would lead one to expect.
Objective measures of transmission quality move the measurement paradigm away from the transmission channel and towards the user. Objective measures of transmission quality can provide useful indications of the quality, intelligibility, or usability. Objective measures thus provide critical feedback so that systems can be designed around users’ expectations, rather than somewhat arbitrary engineering thresholds.
In general, humans have no problem judging received media signals without seeing or hearing the transmitted media signals for comparison purposes. But for objective measures, the analogous task remains a serious challenge. Objective metrics are classified into categories based on the extra information required. This paper summaries the advantages and disadvantages of each approach.
No Reference Metrics
The most popular supplemental information is a pristine original version of the media and the transmission bit-stream. This extra information simplifies the development process but severely limits metric’s scope. The original media is not available for most industrial applications; and metrics that rely on the transmission bit-streams can only be applied to the specific use-case for which they were developed.
The ITS video quality research focuses on no reference (NR) algorithms. NR metrics only use the media itself, so they can be deployed at any point in the video product or service. NR algorithms would open up an entire world of light-weight, in-service, real-time endpoint monitoring, fault detection, and optimization. The most compelling use case for NR metrics is to provide root cause analysis (RCA), which will allow communication systems to detect specific impairments that hinder communication and deploy diverse mitigation strategies.
NR metric development is very tricky. Despite decades of research, none of the available metrics meets the high performance standards of US industry. International experts agree on the need for collaboration to build reliable NR metrics. ITS contributions include:
- Experiment design: innovation on subjective test designs and statistical methods for NR metrics
- Training data: subjective datasets available freely for R&D on the Consumer Digital Video Library (CDVL)
- Software framework: an open software framework with all of the knowledge, tools, and data needed to begin NR metric research
- Reports: independent analyses that assess whether published NR metrics are suitable for consumer applications
- Baseline metric: this NR metric, “Sawatch,” provides a starting point for a series of NR metrics that provide root cause analysis (RCA)
For more information, see the NR Metric Framework.
ITS provides leadership for the Video Quality Experts Group’s (VQEG) No Reference Metric (NORM) working group. NORM is an open collaboration for developing NR metrics and methods for monitoring use case specific visual service quality. Related VQEG efforts include Quality Assessment for Computer Vision Applications (QACoViA) and Quality Assessment for Heath applications (QAH).