Institute for Telecommunication Sciences / About ITS / 2021 / New Interactive Demo: Estimating Video Quality
New Interactive Demo: Estimating Video Quality
September 16, 2021
The Video Quality Research Program has launched an interactive web demo of ITS-developed software that estimates the quality of images or short videos displayed over a network. It’s one more step towards providing video service providers robust and trusted tools to enable them to deliver better quality video over less bandwidth.
Video service providers rely heavily on Quality of Service (QoS) metrics that use network parameters to infer the quality of delivered video in real time. These tools excel if all the quality problems are associated with compression bitrate or network issues. But the performance of QoS metrics drops rapidly when the media entering the network already has quality problems. Most user generated content falls into this category—videoconferencing, smartphone videos, internet video surveillance, 360° videos, etc. Similarly, QoS metrics cannot detect some quality problems that broadcasters struggle with—like camera configuration errors and LED flicker artifacts.
The missing tool is a no reference (NR) metric: an algorithm that assesses the quality of an image or video, using only the pixels displayed to the user. Half the human brain is devoted to processing visual information, yet U.S. industry needs lightweight algorithms that do the same thing and operate in real-time. This makes NR metric research very difficult. Feedback from industry indicates none of the available NR metrics meet their needs around scope, accuracy, and features.
To test this claim, ITS is evaluating NR metrics and reporting on their performance to stimulate an open exchange of ideas, information, and research. Industry needs robust and trusted NR metrics to more efficiently use increasingly crowded bandwidth. Objective and factual information on the performance of existing NR metrics will help developers make the best decisions on where to focus future research.
In 2020, ITS published in-depth analyses of NR metrics, first in a formal technical memorandum and then in a GitHub repository, NRMetricFramework, that provides an open software framework for collaborative development of NR metrics. For each metric analyzed, the GitHub repository presents a metric summary, performance analysis and rating, and source code. The metric’s performance is rated by comparing its quality rating to the rating given by a six-person pilot test across 10+ datasets.
The NRMetricFramework repository includes support tools needed to begin research and avoid common mistakes, as well as versions 1 and 2 of the ITS-developed NR metric Sawatch. Sawatch is designed to assess images produced by a broad range of modern camera systems. Responding to U.S. industry feedback, it is a lightweight, open-source NR metric that provides limited root cause analysis (RCA) as well as overall quality estimation. RCA will allow communication systems to detect specific impairments that hinder communication and deploy mitigation strategies to improve quality in real time.
To demonstrate Sawatch’s potential, ITS developed this limited interactive demo. Users can run the underlying algorithm on a media file uploaded from their computer before diving into the technical details of ITS’s open-source software on the GitHub repository.
Sawatch is named after the Sawatch mountain range in Colorado, which contains eight of the highest 20 peaks in the Rocky Mountains. Like climbing a mountain, the goal is steady improvement towards a robust and trusted NR metric that provides video service providers the ability to deliver better quality video over less bandwidth.