Video Quality Documents

This memorandum provides technical details for the image quality experiment COCRID: A Challenging Optical Character Recognition Dataset. The design goals of the COCRID dataset are (1) to train no-reference metrics that track the quality of recognized...

This article provides a comprehensive overview of no reference (NR) metrics for image quality analysis (IQA) and video quality analysis (VQA). We examine 26 independent evaluations of NR metrics (previously published) and analyze 32 NR metrics on six...

This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence ...

This memorandum provides technical details of the VCRDCI dataset. This dataset was designed similarly to a video quality subjective experiment, but the VMAF metric was used to create simulated subjective data. The VCRDCI dataset contains 130 scenes t...

This technical memorandum provides details for the image quality subjective experiment ITSnoise. This experiment includes 24 scenes, each digitally photographed using 12 different Image Capture Methods (ICMs) for a total of 288 images. The ICMs were ...

Recent studies have shown that it is possible to characterize subject bias and variance in subjective assessment tests. Apparent differences among subjects can, for the most part, be explained by random factors. Building on that theory, we propose a ...

This report describes a methodology that measures the precision of objective metrics that assess image quality, video quality, speech quality, or the overall audiovisual quality. We assess the confidence intervals of 60 subjective tests and use a con...

We need data sets of images and subjective scores to develop robust no reference (or blind) visual quality metrics for consumer applications. These applications have many uncontrolled variables because the camera creates the original media and the im...

This paper describes ongoing work within the video quality experts group (VQEG) to develop no-reference (NR) audiovisual video quality analysis (VQA) metrics. VQEG provides an open forum that encourages knowledge sharing and collaboration. The VQEG n...

This paper analyzes the performance of eight no-reference (NR) metrics. Seven assess image quality (BRISQUE, CurveletQA, IL-NIQE, NIQE, OG-IQA, QAC and SSEQ) and one assesses video quality (VIIDEO). The challenge we address in this paper is moving fr...

This technical memorandum provides technical details for the video quality subjective experiment its4s4. This experiment includes 196 videos, each 4 seconds in duration, depicting camera pans at different speeds. Other impairments are avoided where p...

The conventional video subjective test design, in which subjects view and rate multiple versions of each source video sequence, was used for decades. New technology, like adaptive streaming, makes it almost impossible to use this design since much lo...

This technical memorandum provides technical details for the video quality subjective experiment its4s3. This experiment includes 594 videos, each 4 seconds in duration. The experiment consists of six sessions, each portraying a different public safe...

This technical memorandum provides technical details for the image quality subjective experiment its4s2. This experiment contains 1,473 still images that were presented to subjects for 4 seconds each, as per still video. The goals are (1) to train no...

Video surveillance systems, designed to provide security, act like a guard in a watchtower-a security solution that has been well understood for thousands of years. Systems are designed to meet security needs of the customer who installs the system. ...

This report describes the video quality subjective test its4s, including the experiment design and footage attribution. Subjective experiment its4s includes 813 unique video sequences, each four seconds in duration. No video sequences are repeated. T...

A video is not always an accurate representation of what a first responder sees at an incident. New research suggests that first responders would benefit from a camera that removes airborne obscurants and shadows before the video is compressed. Curre...

This report identifies camera technology gaps that impact first responders. These technology gaps were identified by interviewing first responders about images, video, and camera systems in general. This is a working document that is intended to fost...

This paper presents a subjective image quality experiment design that specifically targets understanding consumer perception of visual quality. With this consumer-oriented goal in mind, images were captured from a large number of representative consu...

The new video coding standard, MPEG-H Part 2 High Efficiency Video Coding (HEVC) or H.265, was developed to be roughly twice as efficient as H.264/AVC—meaning H.265/HEVC could deliver the same quality as H.264/AVC using roughly half the bitrate. In t...

How accurately are people able to use the absolute category rating (ACR) 5-level scale? Put another way, how repeatable are an individual subject’s scores? Several subjective experiments have asked subjects to rate the same sequences a couple of time...

Traditional 35mm film cameras are no longer the main devices today’s consumers use to capture images. Though the dominant technology has shifted to digital cameras and displays that differ widely in pixel count and resolution, our understanding of th...

The Video Quality Experts Group (VQEG) is an international and independent organisation of technical experts in perceptual video quality assessment from industry, academia, and government organisations. The main goals of VQEG are to advance the field...

Consumers demand video delivery to a wide variety of multimedia capable playback devices in dissimilar contexts. Monitoring the quality of the network provides a preliminary indication of the video quality end-users can expect. However, more advanced...

This paper examines color from multiple perspectives. A theory is proposed that the current color spaces are poorly designed for video quality and video coding purposes. The implication is that objective video quality metrics could be improved by the...

This article describes how to perform a video quality subjective test. For companies, these tests can greatly facilitate video product development; for universities, removing perceived barriers to conducting such tests allows expanded research opport...

During 1993–1994, the T1A1 subcommittee of the American National Standards Association (ANSI) accredited Alliance for Telecommunications Industry Solutions (ATIS) conducted an objective video quality metric validation test focused on video teleconfer...

This paper introduces ITU-T Recommendation P.913, a new subjective video quality testing standard approved in January 2014. This Recommendation focuses on the evaluation of flat screens, laptops, and mobile devices. P.913 emphasizes flexibility of en...

We announce a new Video Quality Model (VQM) that accounts for the perceptual impact of variable frame delays (VFD) in videos with demonstrated top performance on the Laboratory for Image & Video Engineering (LIVE) Mobile Video Quality Assessment (VQA...

We propose a model for rating behavior based on subject bias and subject error. Evidence for subject bias can be found in freely available subjective experiments. When subject bias is removed from ratings, the sensitivity of statistical comparisons b...

This report provides full technical details for the video quality subjective test AGH/NTIA. Analyses of this dataset appear in separate publications. The purpose of this document is to provide design details that are beyond the scope of a conference ...

The VQEG eletter publishes up-to-date technical advances on video quality related topics. The VQEG eLetter also makes available limited distribution publications that contain valuable technical information, such as out-of-print ATIS contributions. Th...

An immersive subjective test method is proposed in which subjects view each source stimulus only once. In order to encourage a subject’s engagement with test content, longer stimuli are used. Distractor questions are used in addition to the tradition...

The Web-Enabled Subjective Test software allows researchers to conduct subjective tests on multiple devices with aggregated data collection and reporting. The system leverages modern web technologies and infrastructure to deliver test stimuli to typi...

This paper describes objective video quality validation efforts conducted in the past two decades. Validation efforts to be examined include a validation test performed by the T1A1 committee in the early 1990's; five rounds of validation testing perf...

This paper presents recommended techniques for choosing video sequences for subjective experiments. Subjective video quality assessment is a well-understood field, yet scene selection is often driven by convenience or content availability. Three-dime...

In 2011, the Video Quality Experts Group (VQEG) ran subjects through the same audiovisual subjective test at six different international laboratories. That small dataset is now publically available for research and development purposes.

The difficulty of finding and getting rights to use the high quality video sequences needed for video quality research has hampered research in this area for many years. This article describes the Consumer Digital Video Library (CDVL) website (www.cd...

This memorandum describes the Fast Low Bandwidth model. This summary is intended to help the reader understand the model from an algorithmic standpoint. Some knowledge of prior NTIA objective video quality models is necessary for the understanding of...

This paper explores the utility of visual acuity as a video quality metric for public safety applications. An experiment has been conducted to track the relationship between visual acuity and the ability to perform a forced-choice object recognition ...

This paper presents recommended techniques for choosing video sequences for subjective experiments. Subjective video quality assessment is a well understood field, yet scene selection is often limited by content availability. The Consumer Digital Vid...

Traditionally, audio quality and video quality are evaluated separately in subjective tests. Best practices within the quality assessment community were developed before many modern mobile audiovisual devices and services came into use, such as inter...

We explore the quality impact when audiovisual content is delivered to different mobile devices. Subjects were shown the same sequences on five different mobile devices and a broadcast quality television. Factors influencing quality ratings include v...

The perceived quality of an audiovisual sequence is heavily influenced by both the quality of the audio and the quality of the video. The question then arises as to the relative importance of each factor and whether a regression model predicting audi...

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Windows 7® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as ou...

Time varying delays of the output (or processed) video frames with respect to the input (i.e., the original or reference) video frames present significant challenges for Full Reference (FR) video quality measurement systems. Time alignment errors bet...

How much bandwidth is required for good quality video for a given screen resolution? Data acquired during two Video Quality Experts Group (VQEG) projects allow at least a partial answer to this question. This international subjective testing produced...

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol—IP), and a video decoder can produce pauses in the video presentation, after which the video may continue with or without skippin...

The quality perception of an audiovisual sequence is heavily influenced by the quality of the audio, the quality of the video, and the audiovisual time synchronization. The questions then arise: what is the relative importance of each factor, and can...

The intent of H.264 (MPEG-4 Part 10) was to achieve equivalent quality to previous standards (e.g., MPEG-2) at no more than half the bit-rate. H.264 is commonly felt to have achieved this objective. This document presents results of an HDTV subjectiv...

We present the design, preparation, and analysis of a subjective experiment on typical HDTV sequences and scenarios. This experiment follows the guidelines of ITU and VQEG in order to obtain reproducible results. The careful selection of content and ...

The shortage of high quality video sequences has hampered video quality research for 20 years. Every Video Quality Experts Group (VQEG) test has encountered difficulty in obtaining quality source content. For some validation tests, this caused a mult...

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol – IP), and a video decoder can produce pauses in the video presentation, after which the video may continue with or without skipp...

For industry, the need to access accurate and reliable objective video metrics has become more pressing with the advent of new video applications and services such as mobile broadcasting, Internet video, and Internet Protocol television (IPTV). Indus...

Peak Signal to Noise Ratio (PSNR) has been used as a benchmark to evaluate new objective perceptual video quality metrics. For example, PSNR has been used as a benchmark for both the Multimedia (MM) and Reduced Reference Television (RRTV) test progra...

Study Group 9 has been working on finalizing Draft New Recommendation J.redref which specifies Reduced Reference (RR) methods for performing video quality measurements of digital television systems. The U.S. National Telecommunications and Informatio...

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol – IP), and a video decoder can produce pauses in the video presentation that result from dropped or repeated video frames. For ex...

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Red Hat Linux® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., a...

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Vista® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as output...

This report presents techniques for evaluating objective video quality models using overlapping subjective data sets. The techniques are demonstrated using data from the Video Quality Experts Group (VQEG) Multi-Media (MM) Phase I experiments. These r...

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol – IP), and a video decoder can produce pauses in the video presentation that result from dropped or repeated video frames. For ex...

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Red Hat Linux® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., a...

This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network ...

The Public Safety Statement of Requirements (PS-SoR) for Communications and Interoperability focuses on the needs of first responders to communicate and share information as authorized, when it is needed, where it is needed, and in a mode or form tha...

This paper summarizes results from an experiment whose goal was to assess whether the NTIA General Video Quality Metric (VQM) is an acceptable objective metric for measuring High Definition TV (HDTV) video quality. The HDTV subjective test that was p...

The purpose of this handbook is to provide a user’s manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicatin...

This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network ...

The purpose of this handbook is to provide a user’s manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicatin...

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® operating system. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as output by a video...

This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network ...

The Video Quality Metric (VQM) software was developed by ITS between 2002 and 2010. The VQM algorithms estimate how people perceive video quality. These VQM algorithms compare the processed video (output) with the original video (input). These models...

The purpose of this handbook is to provide a user’s manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicatin...

This paper presents a new reduced reference (RR) video quality monitoring system that utilizes less than 10 kbits/s of reference information from the source video stream. This new video quality monitoring system utilizes feature extraction techniques...

Digital video compression algorithms are being deployed that spatially stretch or shrink the video picture. Although small changes in spatial scaling are not usually noticeable to viewers, objective video quality measurement systems may be adversely ...

The National Telecommunications and Information Administration (NTIA) General Model for estimating video quality and its associated calibration techniques were independently evaluated by the Video Quality Experts Group (VQEG) in their Phase II Full R...

This memorandum compares subjective video quality test results from a professional cathode ray tube (CRT) television monitor with that of a consumer liquid crystal display (LCD) video phone monitor. The CRT monitor supported the full ITU-R Recommenda...

Video quality metrics (VQMs) have often been evaluated and compared using simple measures of correlation to subjective mean opinion scores from panels of observers. However, this approach does not fully take into account the variability implicit in t...

This document discusses a method for correcting inaccurate color output by digital still and video imaging systems. The method uses a known reference image together with a least–squares algorithm to estimate the optimal color channel mixing matrix th...

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. Some of these test methods are double stimulus where viewers rate the qu...

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. In addition to displaying the video sequences in different ways, subject...

A multiple data set fitting problem often arises in conjunction with the development of objective estimators of perceived audio or video quality. In such development work, we often seek the best linear relationship between a set of objective audio or...

Objective metrics for measuring digital video performance are required by Government and industry for specification of system performance requirements, comparison of competing service offerings, service level agreements, network maintenance, and opti...

The purpose of this handbook is to provide a user’s manual for the video quality metric (VQM) tool. The VQM software tool performs automated batch processing of video files. Program VQM runs under the UNIX operating system and uses a control file to ...

Over the past few years, there have been a number of proposals aimed at introducing different levels of service in the Internet. One of the more recent proposals is the Differentiated Services (Diff-Serv) architecture, and in this paper we explore ho...

This paper presents objective-to-subjective correlation results for a reduced-reference, in-service, video quality monitoring system. This reduced-reference system utilizes quality parameters that are computed by comparing features extracted from spa...

Many organizations have focused on developing digital video quality metrics which produce results that accurately emulate subjective responses. However, to be widely applicable a metric must also work over a wide range of quality, and be useful for i...

This contribution presents new techniques for estimating temporal delays between input and output video streams from video teleconferencing systems. The discussion in this contribution considers "variable alignment," defined as an alignment process t...

This contribution presents a new technique for estimating temporal delays between input and output video streams from video teleconferencing systems. The discussion in this contribution is limited to "constant alignment," defined as an alignment proc...

With the advent of new digital video systems that utilize compression to achieve a savings in transmission or storage bandwidth, the quality of the received output video can be dependent not only upon the inherent spatial and temporal information con...

We present results of subjective viewer assessment of video quality of MPEG-2 compressed video containing wide-band Gaussian noise. The video test sequences consisted of seven test clips (both classical and new materials) to which noise with a peak-s...

This paper discusses the analysis of an audiovisual desktop video-teleconferencing subjective experiment conducted at the Institute for Telecommunication Sciences. Objective models of the individual audio and video quality are presented. Also discuss...

In 1996, the American National Standards Institute (ANSI) adopted ANSI T1.801.03, which presents a number of new objective video quality metrics for quantifying the effects of digital compression and transmission impairments. The measurements in ANSI...

Significant research and development efforts by industry and government laboratories were brought to fruition in 1996 with the approval of American National Standard (ANSI) T1.801.03 entitled "American National Standard for Telecommunications - Digit...

Presents an in-depth analysis and discussion of the results from applying the ANSI T1.801.03-1996 objective video quality metrics to subjectively rated MPEG-1 and MPEG-2 video test scenes. The objective metrics presented in ANSI T1.801.03-1996 (Ameri...

Presents a computerized search method for determining the gain, level offset, active video shift, and video delay of a digital video transmission channel. The method uses digitized input and output NTSC video fields that have been time tagged with SM...

This report describes experiments conducted to explore the user-information transfer performance of the broadband integrated services digital network (B-ISDN), the emerging infrastructure for the global information age. These performance experiments ...

Recommends a process to be used by ANSI for duplication and traceability of video tapes for ANSI T1.801.02-1996 (American National Standard for Telecommunications - Digital Transport of Video Teleconferencing/Video Telephony Signals - Performance Ter...

Recommends a process to be used by ANSI for duplication and traceability of video tapes for ANSI T1.801.01-1996 (American National Standard for Telecommunications - Digital Transport of Video Teleconferencing/Video Telephony Signals - Video Test Scen...

This contribution recommends that calculations of visual channel delay be based on fields rather than frames for an interlaced video system such as NTSC. Using fields rather than frames will double the resolution of the measurement without significan...

Presents a summary of efforts to make the objective video quality metrics more efficient in terms of computation and transmission bandwidth. Objective to subjective correlation results are presented for the subjectively rated data set given in the 19...

Presents an objective analysis technique that can be used for detecting temporal edge noise in an output video sequence that has been digitally compressed. Temporal edge noise is defined in ANSI T1.801.02-1996 as "A form of edge busyness characterize...

Describes an algorithm for (1) quantifying the spatial gradients or edges in an image as a function of angle or orientation, (2) extracting low bandwidth features from these input and corresponding output spatial gradient images, and (3) deriving obj...

Provides an in-depth analysis of the behavior of the P6 video performance parameter, a parameter that measures the motion energy that is lost when a video scene is transmitted through a digital video compression system. Objective to subjective correl...

This contribution discusses the advantages and disadvantages of two methods for obtaining a composite rating of Hypothetical Reference Circuit (HRC) performance for an ensemble of source material. The term HRC refers to a specific realization of a vi...

Describes two objective measurements that can be used to quantify the spatial and temporal information content of a video test scene. Test scene coding difficulty (and hence video quality) often depend on the spatial and temporal information complexi...

Presents detailed intra-laboratory (within laboratory) and inter-laboratory (between laboratories) analysis of variance (ANOVA) results for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 ...

Presents results from the analysis of variance (ANOVA) of the subjective opinion scores for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 different video transmission systems that ranged...

This contribution presents objective to subjective correlation results for the traditional analog measurements of bandwidth (i.e., frequency response) and signal-to-noise ratio (SNR) for the T1A1 subjective video experiment that contained 625 mean op...

Presents in detail the ANOVA methods for analysis of the subjective T1A1 video test data, especially with regard to any systematic effect a laboratory may have on the Mean Opinion Score (MOS) of any given Hypothetical Reference Circuit (HRC). The ter...

Contribution T1A1.5/93-152 summarized the methods of measurement for objective video quality parameters based on the Sobel-filtered image and the motion difference image that were submitted prior to conducting the T1A1 subjective experiment (this exp...

Discusses the method of measurement used to obtain the signal to noise ratio (SNR) values for each of the twenty-five video systems in the T1A1 subjective video experiment. The 25 video systems ranged in bit rate from 64 kb/sec to 45 Mb/sec. The sign...

Discusses the method of measurement used to obtain the frequency response or bandwidth values for each of the twenty-five video systems in the T1A1 subjective video experiment. The 25 video systems ranged in bit rate from 64 kb/sec to 45 Mb/sec. The ...

Discusses the standardization of user-oriented, technology-independent measures of telecommunication service quality, including voice, video, and data performance standards. Standards committee work has progressed in three broad phases. In the first ...

Summarizes detailed methods of measurement for two objective video quality parameters based on the Fourier transform image. This contribution was submitted to ANSI T1A1 prior to conducting the T1A1 subjective experiment (this experiment collected 625...

Summarizes detailed methods of measurement for objective video quality parameters based on the Sobel-filtered image and the motion difference image. This contribution was submitted to ANSI T1A1 prior to conducting the T1A1 subjective experiment (this...

Discusses a real-time personal computer-based system for characterizing spatial (e.g., blurring) and temporal (e.g., jerky motion) distortions in a compressed digital video system. Comparisons with a high quality non real-time laboratory version of t...

Measurements that quantify perceptual video attributes in both the spatial and temporal domains are extracted from the video that has undergone digital coding and compression. These measurements are then used to compute a single score that quantifies...

The Institute for Telecommunication Sciences (ITS) has developed an objective video quality assessment system that emulates human perception. The system returns results that agree closely with quality judgements made by a large panel of viewers. Such...

Out-of-service measurements can be used to obtain a baseline video system delay. However, in many digital video systems, video delay is dynamic and dependent upon the scene content. This contribution describes an objective measure of video delay that...

The motion difference images of the original and degraded video sequences is used to dynamically measure the Transmitted Frame Rate (TFR) and the Average Frame Rate (AFR) of a digital video system codec (coder-decoder). The TFR measurement algorithm ...

Presents objective to subjective correlation results when the scores from multiple video test scenes are averaged. It is shown that the prediction errors of the objective video quality model are reduced, yielding objective measurement results that mo...

Discusses the approach used and the research conducted to develop an objective video quality assessment system that emulates human perception. The system returns results that agree closely with quality judgments made by a large panel of viewers for t...

ATIS Committee T1 contribution to standards project “Analog Interface Performance Specifications for Digital Video Teleconferencing/Video Telephony Service and DS3 Television”

Presents an overview of metrics that can be used to quantify the amount of spatial and temporal information in a video sequence. By applying the metrics to the input and output video, the amount of spatial and temporal information that is lost by the...

Discusses research efforts to derive objective measures of video quality that emulate human perception. The derivation of these metrics involves the following steps: (1) a set of test scenes in selected and distorted, (2) a set of candidate objective...

Presents preliminary results on the objective to subjective quality correlation for a set of 5 test scenes passed through three systems (NTSC, VHS, and DS1 MPEG coding). One objective metric that correlates well with subjective quality quantifies the...

The ability of the human eye to resolve detail in a video scene is related to how much motion is present at the point of focus and whether or not the eye can track the motion. Thus, stationary portions of the video scene can be resolved in great deta...

This report describes an automated method of video quality assessment based on extraction and classification of features from sampled input and output video. The first subsystem of the automated video quality measurement system is the feature extract...