February 2026 | Conference Paper
Unseen but not Unknown: Using Dataset Concealment to Robustly Evaluate Speech Quality Estimation Models
Cite This Publication
Jaden Pieper and Stephen D. Voran, “Unseen but not Unknown: Using Dataset Concealment to Robustly Evaluate Speech Quality Estimation Models,” in 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (Barcelona, Spain, May 2026).
Jaden Pieper and Stephen D. Voran 
Abstract:
We introduce Dataset Concealment (DSC), a rigorous new procedure for evaluating and interpreting objective speech quality estimation models. DSC quantifies and decomposes the performance gap between research results and real-world application requirements, while offering context and additional insights into model behavior and dataset characteristics. We also show the benefits of addressing the corpus effect by using the dataset Aligner from AlignNet when training models with multiple datasets. We demonstrate DSC and the improvements from the Aligner using nine training datasets and nine unseen datasets with three well-studied models: MOSNet, NISQA, and a Wav2Vec2.0-based model. DSC provides interpretable views of the generalization capabilities and limitations of models, while al-lowing all available data to be used at training. An additional result is that adding the 1000 parameter dataset Aligner to the 94 million parameter Wav2Vec model during training does significantly improve the resulting model’s ability to estimate speech quality for unseen data.
Keywords: speech quality; subjective test; corpus effect; no reference (NR) estimator; dataset alignment
For technical information concerning this report, contact:
Jaden Pieper
Institute for Telecommunication Sciences
(202) 236-7516
jpieper@ntia.gov
For funding information concerning this report, click this link.
Funding Information
Performing Agency
U.S. Department of Commerce
National Telecommunications and Information Administration
Institute for Telecommunication Sciences
325 Broadway
Boulder, CO 80305
https://ror.org/00mj5bc69
Funding Agency
U.S. Department of Commerce
National Telecommunications and Information Administration
Herbert C. Hoover Building
14th and Constitution Ave., N.W.
Washington, D.C. 20230
https://ror.org/032241511
Disclaimer:
Certain commercial equipment, components, and software may be identified in this report to specify adequately the technical aspects of the reported results. In no case does such identification imply recommendation or endorsement by the National Telecommunications and Information Administration, nor does it imply that the equipment or software identified is necessarily the best available for the particular application or uses.
For questions or information on this or any other NTIA scientific publication, contact the ITS Publications Office at ITSinfo@ntia.gov or 303-497-3572.