Institute for Telecommunication Sciences / Research / Quality of Experience / Video Quality Research / Publications

Video Quality Documents

Robert Grosso and Margaret H. Pinson, “COCRID: A Challenging Optical Character Recognition Image Dataset,” Technical Memorandum TM-23-569, August 2023

This memorandum provides technical details for the image quality experiment COCRID: A Challenging Optical Character Recognition Dataset. The design goals of the COCRID dataset are (1) to train no-reference metrics that track the quality of recognized...

This article provides a comprehensive overview of no reference (NR) metrics for image quality analysis (IQA) and video quality analysis (VQA). We examine 26 independent evaluations of NR metrics (previously published) and analyze 32 NR metrics on six...

This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence ...

Robert Grosso and Margaret H. Pinson, “VMAF Compression Ratings that Disregard Camera Impairments (VCRDCI) Dataset,” Technical Memorandum TM-22-564, September 2022

This memorandum provides technical details of the VCRDCI dataset. This dataset was designed similarly to a video quality subjective experiment, but the VMAF metric was used to create simulated subjective data. The VCRDCI dataset contains 130 scenes t...

Joel Dumke and Margaret H. Pinson, “ITSnoise: An Image Quality Dataset With Sensor Noise,” Technical Memorandum TM-22-563, September 2022

This technical memorandum provides details for the image quality subjective experiment ITSnoise. This experiment includes 24 scenes, each digitally photographed using 12 different Image Capture Methods (ICMs) for a total of 288 images. The ICMs were ...

Pablo Pérez, Lucjan Janowski, Narciso García, and Margaret H. Pinson, “Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR) ,” Journal Article, July 2021

Recent studies have shown that it is possible to characterize subject bias and variance in subjective assessment tests. Apparent differences among subjects can, for the most part, be explained by random factors. Building on that theory, we propose a ...

This report describes a methodology that measures the precision of objective metrics that assess image quality, video quality, speech quality, or the overall audiovisual quality. We assess the confidence intervals of 60 subjective tests and use a con...

Jakub Nawala, Margaret H. Pinson, Mikolaj Leszczuk, and Lucjan Janowski, “Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content,” Journal Article, February 2020

We need data sets of images and subjective scores to develop robust no reference (or blind) visual quality metrics for consumer applications. These applications have many uncontrolled variables because the camera creates the original media and the im...

Margaret H. Pinson, Philip J. Corriveau, Mikolaj Leszczuk, and Michael Colligan, “Open Software Framework for Collaborative Development of No Reference Image and Video Quality Metrics,” Conference Paper, January 2020

This paper describes ongoing work within the video quality experts group (VQEG) to develop no-reference (NR) audiovisual video quality analysis (VQA) metrics. VQEG provides an open forum that encourages knowledge sharing and collaboration. The VQEG n...

Margaret H. Pinson, “Analysis of No-Reference Metrics for Image and Video Quality of Consumer Applications,” Technical Memorandum TM-20-547, January 2020

This paper analyzes the performance of eight no-reference (NR) metrics. Seven assess image quality (BRISQUE, CurveletQA, IL-NIQE, NIQE, OG-IQA, QAC and SSEQ) and one assesses video quality (VIIDEO). The challenge we address in this paper is moving fr...

Margaret H. Pinson and Samuel Elting, “ITS4S4: A Video Quality Study of Camera Pans,” Technical Memorandum TM-20-545, December 2019

This technical memorandum provides technical details for the video quality subjective experiment its4s4. This experiment includes 196 videos, each 4 seconds in duration, depicting camera pans at different speeds. Other impairments are avoided where p...

Lucjan Janowski, Ludovic Malfait, and Margaret H. Pinson, “Evaluating Experiment Design with Unrepeated Scenes for Video Quality Subjective Assessment,” Journal Article, June 2019

The conventional video subjective test design, in which subjects view and rate multiple versions of each source video sequence, was used for decades. New technology, like adaptive streaming, makes it almost impossible to use this design since much lo...

This technical memorandum provides technical details for the video quality subjective experiment its4s3. This experiment includes 594 videos, each 4 seconds in duration. The experiment consists of six sessions, each portraying a different public safe...

Margaret H. Pinson, “ITS4S2: An Image Quality Dataset With Unrepeated Images From Consumer Cameras,” Technical Memorandum TM-19-537, April 2019

This technical memorandum provides technical details for the image quality subjective experiment its4s2. This experiment contains 1,473 still images that were presented to subjects for 4 seconds each, as per still video. The goals are (1) to train no...

Video surveillance systems, designed to provide security, act like a guard in a watchtower-a security solution that has been well understood for thousands of years. Systems are designed to meet security needs of the customer who installs the system. ...

Margaret H. Pinson, “ITS4S: A Video Quality Dataset with Four-Second Unrepeated Scenes,” Technical Memorandum TM-18-532, February 2018

This report describes the video quality subjective test its4s, including the experiment design and footage attribution. Subjective experiment its4s includes 813 unique video sequences, each four seconds in duration. No video sequences are repeated. T...

Margaret H. Pinson, “Gaps in Public-Safety Cameras,” Journal Article, August 2017

A video is not always an accurate representation of what a first responder sees at an incident. New research suggests that first responders would benefit from a camera that removes airborne obscurants and shadows before the video is compressed. Curre...

Margaret H. Pinson, “Technology Gaps in First Responder Cameras,” Technical Memorandum TM-17-524, May 2017

This report identifies camera technology gaps that impact first responders. These technology gaps were identified by interviewing first responders about images, video, and camera systems in general. This is a working document that is intended to fost...

This paper presents a subjective image quality experiment design that specifically targets understanding consumer perception of visual quality. With this consumer-oriented goal in mind, images were captured from a large number of representative consu...

The new video coding standard, MPEG-H Part 2 High Efficiency Video Coding (HEVC) or H.265, was developed to be roughly twice as efficient as H.264/AVC—meaning H.265/HEVC could deliver the same quality as H.264/AVC using roughly half the bitrate. In t...

Lucjan Janowski and Margaret H. Pinson, “The Accuracy of Subjects in a Quality Experiment: A Theoretical Subject Model,” Journal Article, December 2015

How accurately are people able to use the absolute category rating (ACR) 5-level scale? Put another way, how repeatable are an individual subject’s scores? Several subjective experiments have asked subjects to rate the same sequences a couple of time...

Michele A. Saad et al., “Impact of Camera Pixel Count and Monitor Resolution Perceptual Image Quality,” Conference Paper, August 2015

Traditional 35mm film cameras are no longer the main devices today’s consumers use to capture images. Though the dominant technology has shifted to digital cameras and displays that differ widely in pixel count and resolution, our understanding of th...

Quan Huynh-Thu, Arthur A. Webster, Kjell Brunnström, and Margaret H. Pinson, “VQEG: Shaping Standards on Video Quality,” Conference Paper, June 2015

The Video Quality Experts Group (VQEG) is an international and independent organisation of technical experts in perceptual video quality assessment from industry, academia, and government organisations. The main goals of VQEG are to advance the field...

Consumers demand video delivery to a wide variety of multimedia capable playback devices in dissimilar contexts. Monitoring the quality of the network provides a preliminary indication of the video quality end-users can expect. However, more advanced...

Margaret H. Pinson, “A missing factor in objective video quality models: A study of color ,” Conference Paper, February 2015

This paper examines color from multiple perspectives. A theory is proposed that the current color spaces are poorly designed for video quality and video coding purposes. The implication is that objective video quality metrics could be improved by the...

Margaret H. Pinson, Lucjan Janowski, and Zdzislaw Papir, “Video Quality Assessment: Subjective Testing of Entertainment Scenes,” Journal Article, January 2015

This article describes how to perform a video quality subjective test. For companies, these tests can greatly facilitate video product development; for universities, removing perceived barriers to conducting such tests allows expanded research opport...

Margaret H. Pinson and Arthur A. Webster, “T1A1 Validation Test Database,” Journal Article, December 2014

During 1993–1994, the T1A1 subcommittee of the American National Standards Association (ANSI) accredited Alliance for Telecommunications Industry Solutions (ATIS) conducted an objective video quality metric validation test focused on video teleconfer...

Margaret H. Pinson and Lucjan Janowski, “A New Subjective Audiovisual & Video Quality Testing Recommendation,” Journal Article, December 2014

This paper introduces ITU-T Recommendation P.913, a new subjective video quality testing standard approved in January 2014. This Recommendation focuses on the evaluation of flat screens, laptops, and mobile devices. P.913 emphasizes flexibility of en...

Margaret H. Pinson, Lark Kwon Choi, and Alan Conrad Bovik, “Temporal Video Quality Model Accounting for Variable Frame Delay Distortions,” Journal Article, December 2014

We announce a new Video Quality Model (VQM) that accounts for the perceptual impact of variable frame delays (VFD) in videos with demonstrated top performance on the Laboratory for Image & Video Engineering (LIVE) Mobile Video Quality Assessment (VQA...

Lucjan Janowski and Margaret H. Pinson, “Subject Bias: Introducing a Theoretical User Model,” Conference Paper, September 2014

We propose a model for rating behavior based on subject bias and subject error. Evidence for subject bias can be found in freely available subjective experiments. When subject bias is removed from ratings, the sensitivity of statistical comparisons b...

Margaret H. Pinson and Lucjan Janowski, “AGH/NTIA: A Video Quality Subjective Test with Repeated Sequences,” Technical Memorandum TM-14-505, June 2014

This report provides full technical details for the video quality subjective test AGH/NTIA. Analyses of this dataset appear in separate publications. The purpose of this document is to provide design details that are beyond the scope of a conference ...

Margaret H. Pinson and Naeem Ram, “VQEG eLetter: Volume 1, Issue 1,” March 2014

The VQEG eletter publishes up-to-date technical advances on video quality related topics. The VQEG eLetter also makes available limited distribution publications that contain valuable technical information, such as out-of-print ATIS contributions. Th...

Margaret H. Pinson, Marc Sullivan, and Andrew A. Catellier, “A new method for immersive audiovisual subjective testing,” Conference Paper, January 2014

An immersive subjective test method is proposed in which subjects view each source stimulus only once. In order to encourage a subject’s engagement with test content, longer stimuli are used. Distractor questions are used in addition to the tradition...

Andrew A. Catellier and Luke Connors, “Web-Enabled Subjective Test (WEST) Research Tools Manual,” Handbook HB-14-501, January 2014

The Web-Enabled Subjective Test software allows researchers to conduct subjective tests on multiple devices with aggregated data collection and reporting. The system leverages modern web technologies and infrastructure to deliver test stimuli to typi...

Margaret H. Pinson, Nicolas Staelens, and Arthur A. Webster, “The history of video quality model validation,” Conference Paper, September 2013

This paper describes objective video quality validation efforts conducted in the past two decades. Validation efforts to be examined include a validation test performed by the T1A1 committee in the early 1990's; five rounds of validation testing perf...

Margaret H. Pinson, Marcus Barkowsky, and Patrick Le Callet, “Selecting Scenes for 2D and 3D Subjective Video Quality Tests,” Journal Article, August 2013

This paper presents recommended techniques for choosing video sequences for subjective experiments. Subjective video quality assessment is a well-understood field, yet scene selection is often driven by convenience or content availability. Three-dime...

In 2011, the Video Quality Experts Group (VQEG) ran subjects through the same audiovisual subjective test at six different international laboratories. That small dataset is now publically available for research and development purposes.

Margaret H. Pinson, “The Consumer Digital Video Library [Best of the Web],” Journal Article, July 2013

The difficulty of finding and getting rights to use the high quality video sequences needed for video quality research has hampered research in this area for many years. This article describes the Consumer Digital Video Library (CDVL) website (www.cd...

Margaret H. Pinson and Stephen Wolf, “Fast Low Bandwidth Model: A Reduced Reference Video Quality Metric,” Technical Memorandum TM-13-497, June 2013

This memorandum describes the Fast Low Bandwidth model. This summary is intended to help the reader understand the model from an algorithmic standpoint. Some knowledge of prior NTIA objective video quality models is necessary for the understanding of...

This paper explores the utility of visual acuity as a video quality metric for public safety applications. An experiment has been conducted to track the relationship between visual acuity and the ability to perform a forced-choice object recognition ...

Margaret H. Pinson, Karen Sue Boyd, Jessica Hooker, and Kristina Muntean, “How To Choose Video Sequences For Video Quality Assessment,” Conference Paper, January 2013

This paper presents recommended techniques for choosing video sequences for subjective experiments. Subjective video quality assessment is a well understood field, yet scene selection is often limited by content availability. The Consumer Digital Vid...

Traditionally, audio quality and video quality are evaluated separately in subjective tests. Best practices within the quality assessment community were developed before many modern mobile audiovisual devices and services came into use, such as inter...

Andrew A. Catellier, Margaret H. Pinson, William J. Ingram, and Arthur A. Webster, “Impact of Mobile Devices and Usage Location on Perceived Multimedia Quality,” Conference Paper, July 2012

We explore the quality impact when audiovisual content is delivered to different mobile devices. Subjects were shown the same sequences on five different mobile devices and a broadcast quality television. Factors influencing quality ratings include v...

Margaret H. Pinson, William J. Ingram, and Arthur A. Webster, “Audiovisual Quality Components: An Analysis,” Journal Article, November 2011

The perceived quality of an audiovisual sequence is heavily influenced by both the quality of the audio and the quality of the video. The question then arises as to the relative importance of each factor and whether a regression model predicting audi...

Margaret H. Pinson and Stephen Wolf, “Batch Video Quality Metric (BVQM) User’s Manual,” Handbook HB-11-441d, September 2011

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Windows 7® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as ou...

Stephen Wolf and Margaret H. Pinson, “Video Quality Model for Variable Frame Delay (VQM_VFD),” Technical Memorandum TM-11-482, September 2011

Time varying delays of the output (or processed) video frames with respect to the input (i.e., the original or reference) video frames present significant challenges for Full Reference (FR) video quality measurement systems. Time alignment errors bet...

Gregory W. Cermak, Margaret H. Pinson, and Stephen Wolf, “The Relationship Among Video Quality, Screen Resolution, and Bit Rate,” Journal Article, June 2011

How much bandwidth is required for good quality video for a given screen resolution? Data acquired during two Video Quality Experts Group (VQEG) projects allow at least a partial answer to this question. This international subjective testing produced...

Stephen Wolf, “Variable Frame Delay (VFD) Parameters for Video Quality Measurements,” Technical Memorandum TM-11-475, April 2011

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol—IP), and a video decoder can produce pauses in the video presentation, after which the video may continue with or without skippin...

Margaret H. Pinson, Arthur A. Webster, and William J. Ingram, “Preliminary Investigation into the Impact of Audiovisual Synchronization of Impaired Audiovisual Sequences,” Technical Memorandum TM-11-474, March 2011

The quality perception of an audiovisual sequence is heavily influenced by the quality of the audio, the quality of the video, and the audiovisual time synchronization. The questions then arise: what is the relative importance of each factor, and can...

Margaret H. Pinson, Stephen Wolf, and Gregory W. Cermak, “HDTV Subjective Quality of H.264 vs. MPEG-2, With and Without Packet Loss,” Journal Article, March 2010

The intent of H.264 (MPEG-4 Part 10) was to achieve equivalent quality to previous standards (e.g., MPEG-2) at no more than half the bit-rate. H.264 is commonly felt to have achieved this objective. This document presents results of an HDTV subjectiv...

Marcus Barkowsky, Margaret H. Pinson, Romuald Pépion, and Patrick Le Callet, “Analysis of Freely Available Subjective Dataset for HDTV including Coding and Transmission Distortions,” Conference Paper, January 2010

We present the design, preparation, and analysis of a subjective experiment on typical HDTV sequences and scenarios. This experiment follows the guidelines of ITU and VQEG in order to obtain reproducible results. The careful selection of content and ...

Margaret H. Pinson, Stephen Wolf, Neha Tripathi, and Chin Koh, “The Consumer Digital Video Library,” Conference Paper, January 2010

The shortage of high quality video sequences has hampered video quality research for 20 years. Every Video Quality Experts Group (VQEG) test has encountered difficulty in obtaining quality source content. For some validation tests, this caused a mult...

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol – IP), and a video decoder can produce pauses in the video presentation, after which the video may continue with or without skipp...

Kjell Brunnström, David Hands, Filippo Speranza, and Arthur A. Webster, “VQEG Validation and ITU Standardization of Objective Perceptual Video Quality Metrics [Standards in a Nutshell],” Journal Article, May 2009

For industry, the need to access accurate and reliable objective video metrics has become more pressing with the advent of new video applications and services such as mobile broadcasting, Internet video, and Internet Protocol television (IPTV). Indus...

Peak Signal to Noise Ratio (PSNR) has been used as a benchmark to evaluate new objective perceptual video quality metrics. For example, PSNR has been used as a benchmark for both the Multimedia (MM) and Reduced Reference Television (RRTV) test progra...

Stephen Wolf and Margaret H. Pinson, “Fast Low Bandwidth Video Quality Model (VQM) Description and Reference Code,” Technical Contribution, February 2009

Study Group 9 has been working on finalizing Draft New Recommendation J.redref which specifies Reduced Reference (RR) methods for performing video quality measurements of digital television systems. The U.S. National Telecommunications and Informatio...

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol – IP), and a video decoder can produce pauses in the video presentation that result from dropped or repeated video frames. For ex...

Margaret H. Pinson and Stephen Wolf, “Batch Video Quality Metric (BVQM) User’s Manual,” Handbook HB-09-441c, January 2009

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Vista® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as output...

Margaret H. Pinson and Stephen Wolf, “In-Service Video Quality Metric (IVQM) User’s Manual,” Handbook NTIA HB-09-434b, January 2009

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Red Hat Linux® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., a...

Margaret H. Pinson and Stephen Wolf, “Techniques for Evaluating Objective Video Quality Models Using Overlapping Subjective Data Sets,” Technical Report NTIA TR-09-457, November 2008

This report presents techniques for evaluating objective video quality models using overlapping subjective data sets. The techniques are demonstrated using data from the Video Quality Experts Group (VQEG) Multi-Media (MM) Phase I experiments. These r...

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol – IP), and a video decoder can produce pauses in the video presentation that result from dropped or repeated video frames. For ex...

Margaret H. Pinson and Stephen Wolf, “Batch Video Quality Metric (BVQM) User’s Manual,” Handbook HB-08-441b, November 2007

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Red Hat Linux® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., a...

Margaret H. Pinson and Stephen Wolf, “Reduced Reference Video Calibration Algorithms,” Technical Report TR-08-433b, November 2007

This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network ...

Margaret H. Pinson, Stephen Wolf, and Robert B. Stafford, “Video Performance Requirements for Tactical Video Applications,” Conference Paper, May 2007

The Public Safety Statement of Requirements (PS-SoR) for Communications and Interoperability focuses on the needs of first responders to communicate and share information as authorized, when it is needed, where it is needed, and in a mode or form tha...

Stephen Wolf and Margaret H. Pinson, “Application of the NTIA General Video Quality Metric (VQM) to HDTV Quality Monitoring,” Conference Paper, January 2007

This paper summarizes results from an experiment whose goal was to assess whether the NTIA General Video Quality Metric (VQM) is an acceptable objective metric for measuring High Definition TV (HDTV) video quality. The HDTV subjective test that was p...

Margaret H. Pinson and Stephen Wolf, “In-Service Video Quality Metric (IVQM) User’s Manual,” Handbook NTIA HB-06-434a, July 2006

The purpose of this handbook is to provide a user’s manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicatin...

Margaret H. Pinson and Stephen Wolf, “Reduced Reference Video Calibration Algorithms,” Technical Report NTIA TR-06-433a, July 2006

This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network ...

Margaret H. Pinson and Stephen Wolf, “In-Service Video Quality Metric (IVQM) User’s Manual,” Handbook NTIA HB-06-434, December 2005

The purpose of this handbook is to provide a user’s manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicatin...

Mark A. McFarland, Margaret H. Pinson, and Stephen Wolf, “Batch Video Quality Metric (BVQM) User’s Manual,” Handbook NTIA HB-06-441a, December 2005

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® operating system. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as output by a video...

Margaret H. Pinson and Stephen Wolf, “Reduced Reference Video Calibration Algorithms,” Technical Report NTIA TR-06-433, October 2005

This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network ...

Margaret H. Pinson and Stephen Wolf, “Video Quality Metric Software,” Software SD-05-435, June 2005

The Video Quality Metric (VQM) software was developed by ITS between 2002 and 2010. The VQM algorithms estimate how people perceive video quality. These VQM algorithms compare the processed video (output) with the original video (input). These models...

The purpose of this handbook is to provide a user’s manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicatin...

Margaret H. Pinson and Stephen Wolf, “Low Bandwidth Reduced Reference Video Quality Monitoring System,” Conference Paper, January 2005

This paper presents a new reduced reference (RR) video quality monitoring system that utilizes less than 10 kbits/s of reference information from the source video stream. This new video quality monitoring system utilizes feature extraction techniques...

Margaret H. Pinson and Stephen Wolf, “Video scaling estimation technique,” Technical Memorandum NTIA TM-05-417, January 2005

Digital video compression algorithms are being deployed that spatially stretch or shrink the video picture. Although small changes in spatial scaling are not usually noticeable to viewers, objective video quality measurement systems may be adversely ...

Margaret H. Pinson and Stephen Wolf, “A New Standardized Method for Objectively Measuring Video Quality,” Journal Article, September 2004

The National Telecommunications and Information Administration (NTIA) General Model for estimating video quality and its associated calibration techniques were independently evaluated by the Video Quality Experts Group (VQEG) in their Phase II Full R...

Margaret H. Pinson and Stephen Wolf, “The impact of monitor resolution and type on subjective video quality testing,” Technical Memorandum NTIA TM-04-412, March 2004

This memorandum compares subjective video quality test results from a professional cathode ray tube (CRT) television monitor with that of a consumer liquid crystal display (LCD) video phone monitor. The CRT monitor supported the full ITU-R Recommenda...

Video quality metrics (VQMs) have often been evaluated and compared using simple measures of correlation to subjective mean opinion scores from panels of observers. However, this approach does not fully take into account the variability implicit in t...

Stephen Wolf, “Color correction matrix for digital still and video imaging systems,” Technical Memorandum NTIA TM-04-406, December 2003

This document discusses a method for correcting inaccurate color output by digital still and video imaging systems. The method uses a known reference image together with a least–squares algorithm to estimate the optimal color channel mixing matrix th...

Margaret H. Pinson and Stephen Wolf, “Comparing Subjective Video Quality Testing Methodologies,” Conference Paper, July 2003

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. Some of these test methods are double stimulus where viewers rate the qu...

Margaret H. Pinson and Stephen Wolf, “An Objective Method for Combining Multiple Subjective Data Sets,” Conference Paper, July 2003

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. In addition to displaying the video sequences in different ways, subject...

Stephen D. Voran, “An Iterated Nested Least-Squares Algorithm for Fitting Multiple Data Sets,” Technical Memorandum NTIA TM-03-397, October 2002

A multiple data set fitting problem often arises in conjunction with the development of objective estimators of perceived audio or video quality. In such development work, we often seek the best linear relationship between a set of objective audio or...

Stephen Wolf and Margaret H. Pinson, “Video Quality Measurement Techniques,” Technical Report NTIA TR-02-392, June 2002

Objective metrics for measuring digital video performance are required by Government and industry for specification of system performance requirements, comparison of competing service offerings, service level agreements, network maintenance, and opti...

Margaret H. Pinson and Stephen Wolf, “Video Quality Measurement User’s Manual,” Handbook NTIA HB-02-01, February 2002

The purpose of this handbook is to provide a user’s manual for the video quality metric (VQM) tool. The VQM software tool performs automated batch processing of video files. Program VQM runs under the UNIX operating system and uses a control file to ...

Wael Ashmawi, Roch Guerin, Stephen Wolf, and Margaret H. Pinson, “On the Impact of Policing and Rate Guarantees in Diff-Serv Networks: A Video Streaming Application Perspective,” Conference Paper, August 2001

Over the past few years, there have been a number of proposals aimed at introducing different levels of service in the Internet. One of the more recent proposals is the Differentiated Services (Diff-Serv) architecture, and in this paper we explore ho...

This paper presents objective-to-subjective correlation results for a reduced-reference, in-service, video quality monitoring system. This reduced-reference system utilizes quality parameters that are computed by comparing features extracted from spa...

Stephen Wolf and Margaret H. Pinson, “Spatial-Temporal Distortion Metrics for In-Service Quality Monitoring of Any Digital Video System,” Conference Paper, September 1999

Many organizations have focused on developing digital video quality metrics which produce results that accurately emulate subjective responses. However, to be widely applicable a metric must also work over a wide range of quality, and be useful for i...

Margaret H. Pinson and Stephen Wolf, “Medium Bandwidth Techniques for Estimating Temporal Delays Between Input and Output Video Sequences,” Technical Contribution, May 1999

This contribution presents new techniques for estimating temporal delays between input and output video streams from video teleconferencing systems. The discussion in this contribution considers "variable alignment," defined as an alignment process t...

Margaret H. Pinson and Stephen Wolf, “Low Bandwidth Techniques for Estimating Temporal Delays Between Input and Output Video Sequences,” Technical Contribution, May 1999

This contribution presents a new technique for estimating temporal delays between input and output video streams from video teleconferencing systems. The discussion in this contribution is limited to "constant alignment," defined as an alignment proc...

Stephen Wolf and Margaret H. Pinson, “In-Service Performance Metrics for MPEG-2 Video Systems,” Conference Paper, November 1998

With the advent of new digital video systems that utilize compression to achieve a savings in transmission or storage bandwidth, the quality of the received output video can be dependent not only upon the inherent spatial and temporal information con...

Charles Fenimore, John Libert, and Stephen Wolf, “Perceptual Effects of Noise in Digital Video Compression,” Conference Paper, October 1998

We present results of subjective viewer assessment of video quality of MPEG-2 compressed video containing wide-band Gaussian noise. The video test sequences consisted of seven test clips (both classical and new materials) to which noise with a peak-s...

Coleen T. Jones and David J. Atkinson, “Development of Opinion-Based Audiovisual Quality Models for Desktop Video-Teleconferencing,” Conference Paper, May 1998

This paper discusses the analysis of an audiovisual desktop video-teleconferencing subjective experiment conducted at the Institute for Telecommunication Sciences. Objective models of the individual audio and video quality are presented. Also discuss...

Stephen Wolf et al., “Validating Objective Measures of MPEG Video Quality,” Journal Article, April 1998

In 1996, the American National Standards Institute (ANSI) adopted ANSI T1.801.03, which presents a number of new objective video quality metrics for quantifying the effects of digital compression and transmission impairments. The measurements in ANSI...

Stephen Wolf, “Measuring the End-to-End Performance of Digital Video Systems,” Journal Article, September 1997

Significant research and development efforts by industry and government laboratories were brought to fruition in 1996 with the approval of American National Standard (ANSI) T1.801.03 entitled "American National Standard for Telecommunications - Digit...

Stephen Wolf et al., “Objective and Subjective Measures of MPEG Video Quality,” Technical Contribution, October 1996

Presents an in-depth analysis and discussion of the results from applying the ANSI T1.801.03-1996 objective video quality metrics to subjectively rated MPEG-1 and MPEG-2 video test scenes. The objective metrics presented in ANSI T1.801.03-1996 (Ameri...

Presents a computerized search method for determining the gain, level offset, active video shift, and video delay of a digital video transmission channel. The method uses digitized input and output NTSC video fields that have been time tagged with SM...

David J. Atkinson, “Exploring B-ISDN performance: Selected experiments and results,” Technical Report TR-96-329, April 1996

This report describes experiments conducted to explore the user-information transfer performance of the broadband integrated services digital network (B-ISDN), the emerging infrastructure for the global information age. These performance experiments ...

Recommends a process to be used by ANSI for duplication and traceability of video tapes for ANSI T1.801.02-1996 (American National Standard for Telecommunications - Digital Transport of Video Teleconferencing/Video Telephony Signals - Performance Ter...

Recommends a process to be used by ANSI for duplication and traceability of video tapes for ANSI T1.801.01-1996 (American National Standard for Telecommunications - Digital Transport of Video Teleconferencing/Video Telephony Signals - Video Test Scen...

Stephen Wolf, “Field Vs. Frame Calculations for Visual Channel Delay,” Technical Contribution, December 1995

This contribution recommends that calculations of visual channel delay be based on fields rather than frames for an interlaced video system such as NTSC. Using fields rather than frames will double the resolution of the measurement without significan...

Stephen Wolf, “An Update on ITS Video Quality Parameters,” Technical Contribution, July 1995

Presents a summary of efforts to make the objective video quality metrics more efficient in terms of computation and transmission bandwidth. Objective to subjective correlation results are presented for the subjectively rated data set given in the 19...

Stephen Wolf, “An Analysis Technique for Detecting Temporal Edge Noise,” Technical Contribution, January 1995

Presents an objective analysis technique that can be used for detecting temporal edge noise in an output video sequence that has been digitally compressed. Temporal edge noise is defined in ANSI T1.801.02-1996 as "A form of edge busyness characterize...

Dwight Melcher and Stephen Wolf, “Objective Measures for Detecting Digital Tiling,” Technical Contribution, January 1995

Describes an algorithm for (1) quantifying the spatial gradients or edges in an image as a function of angle or orientation, (2) extracting low bandwidth features from these input and corresponding output spatial gradient images, and (3) deriving obj...

Stephen Wolf, “An In-depth Analysis of the P6 Lost Motion Energy Parameter,” Technical Contribution, January 1995

Provides an in-depth analysis of the behavior of the P6 video performance parameter, a parameter that measures the motion energy that is lost when a video scene is transmitted through a digital video compression system. Objective to subjective correl...

This contribution discusses the advantages and disadvantages of two methods for obtaining a composite rating of Hypothetical Reference Circuit (HRC) performance for an ensemble of source material. The term HRC refers to a specific realization of a vi...

Arthur A. Webster, “Two Criteria for Video Test Scene Selection,” Technical Contribution, December 1994

Describes two objective measurements that can be used to quantify the spatial and temporal information content of a video test scene. Test scene coding difficulty (and hence video quality) often depend on the spatial and temporal information complexi...

Coleen T. Jones, Ned Crow, Stephen Wolf, and Arthur A. Webster, “Analysis of T1A1.5 Subjective and Objective Test Data,” Technical Contribution, October 1994

Presents detailed intra-laboratory (within laboratory) and inter-laboratory (between laboratories) analysis of variance (ANOVA) results for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 ...

Coleen T. Jones and Ned Crow, “Status of NTIA's Analysis of Subjective Data,” Technical Contribution, July 1994

Presents results from the analysis of variance (ANOVA) of the subjective opinion scores for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 different video transmission systems that ranged...

This contribution presents objective to subjective correlation results for the traditional analog measurements of bandwidth (i.e., frequency response) and signal-to-noise ratio (SNR) for the T1A1 subjective video experiment that contained 625 mean op...

Presents in detail the ANOVA methods for analysis of the subjective T1A1 video test data, especially with regard to any systematic effect a laboratory may have on the Mean Opinion Score (MOS) of any given Hypothetical Reference Circuit (HRC). The ter...

Stephen Wolf and Margaret H. Pinson, “Corrections and Extensions to T1A1.5/93-152,” Technical Contribution, January 1994

Contribution T1A1.5/93-152 summarized the methods of measurement for objective video quality parameters based on the Sobel-filtered image and the motion difference image that were submitted prior to conducting the T1A1 subjective experiment (this exp...

Arthur A. Webster and Coleen T. Jones, “VTC Hypothetical Reference Circuit Signal to Noise Ratio Measurements,” Technical Contribution, January 1994

Discusses the method of measurement used to obtain the signal to noise ratio (SNR) values for each of the twenty-five video systems in the T1A1 subjective video experiment. The 25 video systems ranged in bit rate from 64 kb/sec to 45 Mb/sec. The sign...

Coleen T. Jones, “VTC Hypothetical Reference Circuit Bandwidth Measurements,” Technical Contribution, January 1994

Discusses the method of measurement used to obtain the frequency response or bandwidth values for each of the twenty-five video systems in the T1A1 subjective video experiment. The 25 video systems ranged in bit rate from 64 kb/sec to 45 Mb/sec. The ...

Neal B. Seitz, Stephen Wolf, Randall S. Bloomfield, and Stephen D. Voran, “User-Oriented Measures of Telecommunication Quality,” Journal Article, January 1994

Discusses the standardization of user-oriented, technology-independent measures of telecommunication service quality, including voice, video, and data performance standards. Standards committee work has progressed in three broad phases. In the first ...

Summarizes detailed methods of measurement for two objective video quality parameters based on the Fourier transform image. This contribution was submitted to ANSI T1A1 prior to conducting the T1A1 subjective experiment (this experiment collected 625...

Stephen Wolf, Margaret H. Pinson, Coleen T. Jones, and Arthur A. Webster, “A Summary of Methods of Measurement for Objective Video Quality Parameters Based on the Sobel Filtered Image and the Motion Difference Image,” Technical Contribution, November 1993

Summarizes detailed methods of measurement for objective video quality parameters based on the Sobel-filtered image and the motion difference image. This contribution was submitted to ANSI T1A1 prior to conducting the T1A1 subjective experiment (this...

Coleen T. Jones, “Real-Time Measurement System for ITS Video Quality Parameters,” Technical Contribution, August 1993

Discusses a real-time personal computer-based system for characterizing spatial (e.g., blurring) and temporal (e.g., jerky motion) distortions in a compressed digital video system. Comparisons with a high quality non real-time laboratory version of t...

Stephen D. Voran and Stephen Wolf, “An Objective Technique for Assessing Video Impairments,” Conference Paper, May 1993

Measurements that quantify perceptual video attributes in both the spatial and temporal domains are extracted from the video that has undergone digital coding and compression. These measurements are then used to compute a single score that quantifies...

Arthur A. Webster et al., “An objective video quality assessment system based on human perception,” Conference Paper, February 1993

The Institute for Telecommunication Sciences (ITS) has developed an objective video quality assessment system that emulates human perception. The system returns results that agree closely with quality judgements made by a large panel of viewers. Such...

Coleen T. Jones, Stephen Wolf, and Margaret H. Pinson, “Preliminary Results of One-Way Video Delay Measurement Algorithms,” Technical Contribution, July 1992

Out-of-service measurements can be used to obtain a baseline video system delay. However, in many digital video systems, video delay is dynamic and dependent upon the scene content. This contribution describes an objective measure of video delay that...

The motion difference images of the original and degraded video sequences is used to dynamically measure the Transmitted Frame Rate (TFR) and the Average Frame Rate (AFR) of a digital video system codec (coder-decoder). The TFR measurement algorithm ...

Stephen D. Voran, “The Effect of Multiple Scenes on Objective Video Quality Assessment,” Technical Contribution, July 1992

Presents objective to subjective correlation results when the scores from multiple video test scenes are averaged. It is shown that the prediction errors of the objective video quality model are reduced, yielding objective measurement results that mo...

Discusses the approach used and the research conducted to develop an objective video quality assessment system that emulates human perception. The system returns results that agree closely with quality judgments made by a large panel of viewers for t...

Stephen D. Voran and Stephen Wolf, “Objective Measures of Video Impairment: Analysis of 128 Scenes,” Technical Contribution, March 1992

ATIS Committee T1 contribution to standards project “Analog Interface Performance Specifications for Digital Video Teleconferencing/Video Telephony Service and DS3 Television”

Arthur A. Webster and Stephen Wolf, “Spatial and Temporal Information Measures for Video Quality,” Technical Contribution, January 1992

Presents an overview of metrics that can be used to quantify the amount of spatial and temporal information in a video sequence. By applying the metrics to the input and output video, the amount of spatial and temporal information that is lost by the...

Discusses research efforts to derive objective measures of video quality that emulate human perception. The derivation of these metrics involves the following steps: (1) a set of test scenes in selected and distorted, (2) a set of candidate objective...

Presents preliminary results on the objective to subjective quality correlation for a set of 5 test scenes passed through three systems (NTSC, VHS, and DS1 MPEG coding). One objective metric that correlates well with subjective quality quantifies the...

Stephen D. Voran and Stephen Wolf, “Motion-Still Segmentation Algorithm For VTC/VT Objective Quality Assessment,” Technical Contribution, January 1991

The ability of the human eye to resolve detail in a video scene is related to how much motion is present at the point of focus and whether or not the eye can track the motion. Thus, stationary portions of the video scene can be resolved in great deta...

Stephen Wolf, “Features for Automated Quality Assessment of Digitally Transmitted Video,” Technical Report TR-90-264, June 1990

This report describes an automated method of video quality assessment based on extraction and classification of features from sampled input and output video. The first subsystem of the automated video quality measurement system is the feature extract...