Institute for Telecommunication Sciences / Research / Quality of Experience / Video Quality Research / Subjective Testing / Subjective Testing
Improved Methods for Video Quality Subjective Testing
ITS contributes to efforts within the Video Quality Experts Group (VQEG) and ITU-T Study Group 12 to improve best practices for video quality subjective testing. ITS has conducted various studies to understand subjective test methods that ask questions like "what environmental factors matter in subjective testing" and "how do methods need to be modified for new video technologies."
Lab Facilities
This web page provides insights into audio and video subjective testing and describes ITS facilities for conducting subjective tests.
2025, History of ITU-T Rec. P.910 and P.913
This white paper explains the differences between ITU-T Rec. P.910 and P.913, including the reasons that P.913 was initially created and later discontinued.
Circa 2015, Subject Screening
This white paper provides an overview of subject screening techniques. MATLAB® code implementing the techniques is provided. Since this white paper was written, Zhi LI (Netflix) led a collaborative effort within VQEG to develop improved methods for subject screening. Those methods appear in ITU-T Rec. P.910.
Analyses of Subjective Test Methods
The following publications showcase ITS research into an improved understanding of video quality subjective test methods. Most of these studies were only possible due to international collaborations and insights from discussions at VQEG meetings.
- Margaret H. Pinson, “The Precision and Repeatability of Media Quality Comparisons: Measurements and New Statistical Methods,” Journal Article, February 2023
- Pablo Pérez, Lucjan Janowski, Narciso García, and Margaret H. Pinson, “Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR) ,” Journal Article, July 2021
- Margaret H. Pinson, “Confidence Intervals for Subjective Tests and Objective Metrics That Assess Image, Video, Speech, or Audiovisual Quality,” Technical Report NTIA TR-21-550, October 2020
- Lucjan Janowski, Ludovic Malfait, and Margaret H. Pinson, “Evaluating Experiment Design with Unrepeated Scenes for Video Quality Subjective Assessment,” Journal Article, June 2019
- Lucjan Janowski and Margaret H. Pinson, “The Accuracy of Subjects in a Quality Experiment: A Theoretical Subject Model,” Journal Article, December 2015
- Lucjan Janowski and Margaret H. Pinson, “Subject Bias: Introducing a Theoretical User Model,” Conference Paper, September 2014
- Margaret H. Pinson, Marc Sullivan, and Andrew A. Catellier, “A new method for immersive audiovisual subjective testing,” Conference Paper, January 2014
- Margaret H. Pinson, Marcus Barkowsky, and Patrick Le Callet, “Selecting Scenes for 2D and 3D Subjective Video Quality Tests,” Journal Article, August 2013
- Margaret H. Pinson et al., “Subjective and Objective Evaluation of an Audiovisual Subjective Dataset for Research and Development,” Conference Paper, July 2013
- Joel Dumke, “Visual acuity and task-based video quality in public safety applications,” Conference Paper, February 2013
- Margaret H. Pinson, Karen Sue Boyd, Jessica Hooker, and Kristina Muntean, “How To Choose Video Sequences For Video Quality Assessment,” Conference Paper, January 2013
- Margaret H. Pinson et al., “The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study,” Journal Article, October 2012
- Margaret H. Pinson and Stephen Wolf, “Techniques for Evaluating Objective Video Quality Models Using Overlapping Subjective Data Sets,” Technical Report NTIA TR-09-457, November 2008
- Margaret H. Pinson and Stephen Wolf, “Comparing Subjective Video Quality Testing Methodologies,” Conference Paper, July 2003
- Margaret H. Pinson and Stephen Wolf, “An Objective Method for Combining Multiple Subjective Data Sets,” Conference Paper, July 2003
- Stephen D. Voran, “An Iterated Nested Least-Squares Algorithm for Fitting Multiple Data Sets,” Technical Memorandum NTIA TM-03-397, October 2002
- Edwin L. Crow, “Methods for Analysis of Inter-laboratory Video Performance Standard Subjective Test Data,” Technical Contribution, March 1994
Analysis of Cameras, Monitors, and Codecs
The following publications showcase ITS research into an improved understanding of specific aspects of video quality subjective methods, including the impact of use case, device, and cameras capture. Most of these publications describe experimental methods used to create video quality datasets.
- Michele A. Saad et al., “Image Quality of Experience: A Subjective Test Targeting the Consumer’s Experience,” Conference Paper, February 2016
- Andrew A. Catellier and Margaret H. Pinson, “Characterization of the HEVC Coding Efficiency Advance Using 20 Scenes, ITU-T Rec. P.913 Compliant Subjective Methods, VQM, and PSNR ,” Conference Paper, December 2015
- Michele A. Saad et al., “Impact of Camera Pixel Count and Monitor Resolution Perceptual Image Quality,” Conference Paper, August 2015
- Andrew A. Catellier, Margaret H. Pinson, William J. Ingram, and Arthur A. Webster, “Impact of Mobile Devices and Usage Location on Perceived Multimedia Quality,” Conference Paper, July 2012
- Gregory W. Cermak, Margaret H. Pinson, and Stephen Wolf, “The Relationship Among Video Quality, Screen Resolution, and Bit Rate,” Journal Article, June 2011
- Margaret H. Pinson, Stephen Wolf, and Gregory W. Cermak, “HDTV Subjective Quality of H.264 vs. MPEG-2, With and Without Packet Loss,” Journal Article, March 2010
- Marcus Barkowsky, Margaret H. Pinson, Romuald Pépion, and Patrick Le Callet, “Analysis of Freely Available Subjective Dataset for HDTV including Coding and Transmission Distortions,” Conference Paper, January 2010
- Margaret H. Pinson and Stephen Wolf, “The impact of monitor resolution and type on subjective video quality testing,” Technical Memorandum NTIA TM-04-412, March 2004
- Stephen Wolf, “Color correction matrix for digital still and video imaging systems,” Technical Memorandum NTIA TM-04-406, December 2003
- Wael Ashmawi, Roch Guerin, Stephen Wolf, and Margaret H. Pinson, “On the Impact of Policing and Rate Guarantees in Diff-Serv Networks: A Video Streaming Application Perspective,” Conference Paper, August 2001