Institute for Telecommunication Sciences / Research / Quality of Experience / Audio Quality Research / Audio Quality Research Software / ABC-MRT History and Applications
ABC-MRT History and Applications
Modified Rhyme Tests (MRTs) measure the speech intelligibility of radio systems—something that is very important to public safety practitioners! ITS has carried out these tests since 2008 on behalf of multiple Public Safety agencies and organizations with concerns about the usability of emerging radio technologies, especially concerns about the intelligibility of digital speech encoding when serious background noise is an inevitable part of the work environment.
The MRT is a six-way forced choice word identification task. It is done with human subjects and this naturally entails significant costs and complexities. ITS has significant experience with audio tests that use human subjects, and by 2013 ITS had accumulated a fair-sized repository of recordings and associated MRT results. ITS also had done considerable work on the development and evaluation of signal processing algorithms that reduce the need for these tests.
So we were well-positioned to leverage the repository and invoke signal processing tools to create an automated alternative to the MRT. We knew that an efficient and effective alternative would be useful within ITS and would also provide significant benefit to other organizations concerned with measuring speech intelligibility. We were successful in our goal: in 2013, we developed an MRT alternative that does not require human subjects and is thus orders of magnitude faster and less expensive than the MRT. And, as we later discovered, we were correct that organizations outside of ITS would benefit from our work.
Because MRT involves the identification of words, our automated alternative has some superficial commonality with automatic speech recognition (ASR). ASR strives to work on large vocabularies and have maximal robustness to degradation of the input speech. In contrast, in the MRT task we have 50 independent vocabularies, each containing just 6 words. And maximal robustness in not the goal. Instead we seek a robustness that closely matches that of human subjects. That is how an automated MRT can produce results like those of the true MRT.
We found that correlating perception-based, time-frequency templates according to the articulation index bands provided an excellent simulation of the human word selection results. Thus our automated alternative is called the Articulation Band Correlation – MRT or ABC-MRT for short.
Our initial work focused on narrowband radio systems (nominally carrying frequencies from 300 to 3400 Hz). We published the narrowband ABC-MRT algorithm in the fall of 2013. This attracted interest from various parties who were intrigued with the opportunity to gather MRT-like results without the time, expense, and laboratory constraints normally required for full MRT work.
In the subsequent years ITS continued with MRT testing to support Public Safety and example reports can be found here and here. This work allowed us to augment our data repository with results for wideband radio systems (nominally carrying frequencies from 50 to 7000 Hz), as well as some superwideband, and fullband results. This in turn allowed us to extend ABC-MRT so that it automatically works for any of the speech bandwidths.
They key advance here was the notion of attention – in the MRT word selection task listeners attend (or tune in) to the most compelling evidence and can often disregard less compelling evidence. Modeling of attention with respect to frequency bands allows a single algorithm to work equally well regardless of the speech bandwidth in use. This extended algorithm is called ABC-MRT16 and we published the advance in the spring of 2017. In addition, we made the algorithms and the mandatory integral word templates and speech files available for all users at the ITS website.
The benefits of this work do indeed extend well beyond ITS. We supported engineers at Audio Precision as they developed and verified an ABC-MRT implementation for their latest measurement platform, the Audio Precision APx500 Flex Audio Analyzer. Since their founding in 1984 Audio Precision has become a demonstrated worldwide leader for audio analyzers and audio testing. With ITS support, in the spring of 2018 Audio Precision announced the availability of the ABC-MRT option, allowing their most advanced audio analyzer to provide automated MRT-like speech intelligibly measurements in addition to the conventional, signal-oriented measurements like frequency response, total harmonic distortion, and noise.
Spirent is a global leader in automated test and assurance for telecommunications networks. Spirent engineers sought ITS support with ABC-MRT16 and together were able to integrate and verify the ABC-MRT16 algorithm into the Spirent laboratory testing environment. This allowed Spirent to test the speech intelligibility of consumer-grade LTE handsets as well as handsets that were purpose-built for the Public Safety LTE network (FirstNet). In fall 2019 Spirent provided important ABC-MRT16 test results in a press-release and whitepaper. The importance of these ABC-MRT16 results is underscored by the additional reports in trade journals such as Urgent Communications, MissionCritical Communications, RCR Wireless News, and Firehouse.
We consider the ABC-MRT and ABC-MRT16 algorithms to be ITS success stories. ITS speech intelligibility tests to support Public Safety first responders illuminated the need (slow expensive tests with human subjects) and also produced the means (large data repository) for these successes. When the data was combined with ITS experience, insights, and persistence, the algorithms followed. We disseminated our results and stakeholders stepped forward to implement and apply our successes in commercial venues, thus multiplying the benefits across the industry and the Nation’s economy.