Institute for Telecommunication Sciences / Research / Quality of Experience / Video Quality Research / Software / VQM FAQ
VQM Frequently Asked Questions (FAQ)
TS began video quality research in 1989 to support the transition from analog to digital video transmission systems. ITS developed a new measurement paradigm that is based upon extraction and comparison of low bandwidth perception-based features (e.g., edges, motion) that can be easily communicated throughout the broadcast network.
This measurement paradigm has received three U.S. patents, was adopted as an ANSI standard in 1996 (ANSI T1.801.03-1996) and revised in 2003 (ANSI T1.801.03-2003), was included in two International Telecommunication Union (ITU) Recommendations in 2004 (ITU-T Recommendation J.144 and ITU-R Recommendation BT.1683), and is being used by organizations worldwide. This measurement paradigm is known internationally as "reduced-reference" video quality measurements (see International Telecommunications Union, Telecommunication Standardization Sector, ITU-T Recommendation J.143, "User requirements for objective perceptual video quality measurements in digital cable television"). This ITS-developed methodology has been extensively tested on a wide range of video systems and bit rates including video teleconferencing, MPEG (1, 2, and 4), DS3 (45 Mb/sec), as well as analog video systems.
Nicknamed "VQM," this metric is freely available. VQM has adequate performance for high definition television (HDTV). However, it cannot assess the quality impact of modern network systems where the overall delay can change (e.g., due to rebuffering). In 2011, ITS released a newer metric with variable frame delay (VFD) to address this need.
Although intended for in-service deployment, these metrics are being used as lab tools, out-of-service. Logistically, most companies are not able to connect the measurement point with the originating video supply. That connection is needed for in-service deployment of these reduced reference (RR) metrics. Also, other organizations have released full reference (FR) metrics that are suitable for out-of-service, lab bench testing. Consequently, ITS has discontinued research on RR metrics.
Frequently Asked Questions
Question: "What are the Video Quality Models (VQM)?"
Answer: The Video Quality Project has been developing objective Video Quality Models (VQM) for over two decades. Our VQM software offers an inexpensive alternative to subjective tests.
VQM is a standardized method of objectively measuring video quality. VQM's ratings closely predict the subjective quality ratings that would be obtained from a panel of human viewers. End-users and service providers can use VQM tools to:
- specify/verify system performance
- compare competing service offerings
- maintain and monitor the quality of their networks
- optimize the use of limited network resources such as bit-rate
Due to its excellent performance in the International Video Quality Expert's Group (VQEG) validation tests, the NTIA/ITS VQM methods were adopted as ANSI and ITU standards. VQM tools are available for the Windows® operating systems and customized for two different applications: batch processing and command line processing.
Question: "What usage restrictions does the VQM software have?"
Answer: The VQM software can be downloaded royalty free. The VQM software can be used for any commercial or non-commercial use. The GitHub repository provides the MATLAB® source, a 32-bit compiled version, and a 64-bit compiled version of VQM.
Question: "Is VQM available for Linux or OS?"
Answer: No. We do not have a Linux or OS compatible executable for the VQM software. However, you can download the VQM source code and run it on MATLAB®.
Question: "Where can I find free source video sequences for research and development?"
Answer: See the Consumer Digital Video Library
Question: "Which calibration option should I choose?"
Answer: If you do not know anything about your codec's calibration, here are our suggestions. Run BVQM twice: once with 'Reduced Reference Calibration Version 2' and once with 'Full Reference Calibration'. Both of these calibration options check everything. 'Full Reference Calibration' is more accurate but does not check for spatial scaling. 'Reduced Reference Calibration Version 2' will check whether or not your codec is spatially scaling the video.
Question: "Which model should I choose?"
Answer: The VQM software has a variety of algorithms to suit different needs:
- The NTIA General Model
- NTIA Fast Low Bandwidth Model
- Video Quality Model with Variable Frame Delay (VQM_VFD)
- The Peak Signal to Noise (PSNR) Model
The NTIA General Model and the Full Reference Calibration were validated and standardized. Click here for a description, or here for a full disclosure. The NTIA General Model is often called "VQM" in literature. It is more accurate than PSNR. Thiese algorithms weres trained mostly on standard definition video and a few CIF videos (352x288 pixels). The NTIA General Model is effectively a full reference model.
The NTIA Fast Low Bandwidth Model and the Reduced Reference Calibration were validated and standardized. The NTIA Fast Low Bandwidth Model is a reduced reference model with similar performance to the NTIA General Model. Currently, all implementations of this model are full reference.
The Video Quality Model with Variable Frame Delay (VQM_VFD) was released late in 2011. This is the most accurate ITS VQM to date. VQM_VFD includes a neural network that was trained on people's opinions of the quality of 11,255 test videos. These videos ranged in resolution from cell phone size to HDTV. This model has not been validated or standardized. Click here for more information.
The PSNR Model has wide industry acceptance yet limited accuracy. PSNR is very sensitivity to calibration errors. If you want to use PSNR on video that may contain a reduced frame rate or variable frame delay and you want this effect removed before PSNR is calculated, we recommend 'PSNR with Variable Frame Delay'. PSNR is less accurate than the NTIA General Model and the Fast Low Bandwidth Model for most applications.
Question: "How should I refer to these models in outside reporting?"
Answer: U.S. Department of Commerce policy prohibits NTIA/ITS from endorsing products.
Preferably, refer to the National or International Standards associated with the video quality model. This gives legitimacy and credence to your results.
- The NTIA General Model and Full Reference Calibration were standardized identically in all three Recommendations (ANSI T1.801.03-2003, ITU-T Rec. J.144, and ITU-R Rec. BT.1683).
- The Fast Low Bandwidth Model and Reduced Reference Calibration were standardized in separate Recommendations (ITU-T Rec J.249 and ITU-T Rec. J.244).
- An exhaustive search version of PSNR was standardized by ITU-T (ITU-T J.340). This model is very slow, yet compute ideal values for a constant spatial shift, constant temporal shift, and constant luminance gain and offset. This model can be computed by selected one of the PSNR search calibration options, followed by the PSNR model. Do NOT choose the PSNR_VFD model, which also includes advanced variable frame delay estimation.
Question: "I ran CVQM and nothing happened."
Answer: CVQM does not print anything to the command line. The results are written to log files, named after the processed video file.
Question: "I am confused by CVQM's log file. I ran the General Model, and looked in the log file. To check the result, I calculated the VQM based on 7 parameters by using this equation as your paper:
VQM = -0.2097*si_loss+0.5969*hv_loss+0.2483*hv_gain+0.0192*chroma_spread-2.3416*si_gain+0.0431*ct_ati_gain+0.0076*chroma_extreme
This was different than the value listed next to "general" in the CVQM log file. Why?"
Answer: The parameter values in the CVQM log file already have the weight applied. Sum these parameter values, and you will get the number listed next to "general" (i.e., the NTIA General Model). This model is called "VQM" in the above paper.
Question: "What references should I use for the NTIA models?"
Answer: NTIA General Model (also known as VQM), Developer's model and the Full Reference Calibration algorithms: An NTIA Technical Report contains a full disclosure of these algoirthms, while an IEEE journal article provides a more easily understood summary.
The NTIA Fast Lowbandwidth model is described in this ITU Contribution. A VPQM conference paper provides a more easily understood summary, but this paper is missing some details. The Reduced Reference calbiration algorithms are fully disclosed in an NTIA Technical Report.
The VQM_VFD model is described in an NTIA Technical Memorandum and a journal article available here.