INTERNATIONAL ORGANISATION FOR STANDARDISATION
ORGANISATION INTERNATIONALE DE NORMALISATION
ISO/IEC JTC1/SC29/WG11
CODING OF MOVING PICTURES AND AUDIO

ISO/IEC JTC1/SC29/WG11
MPEG98/N2276
July 1998/Dublin

Source: Audio Subgroup
Title: Report on the MPEG-4 audio NADIB verification tests
Authors: Catherine Colomes (CCETT), Caroline Jacobson (TERACOM), Eric Scheirer (MIT), Audio Subgroup

Report on the MPEG-4 audio NADIB verification tests

This web page contains only excerpts of the document. The complete document is available as PDF file.

Summary

The MPEG-4 Audio coding tools covering 6kbit/s to 24kbit/s have undergone verification testing for an AM digital audio broadcasting application in collaboration with the NADIB(Narrow Band Digital Broadcasting) consortium. With the intent of identifying a suitable digital audio broadcast format to provide improvements over the existing AM modulation services, several codec configurations involving the MPEG-4 CELP, TwinVQ, and AAC tools have been compared to a reference AM system. It was found that higher quality can be achieved in the same bandwidth with digital techniques and that scalable coder configurations offered performance superior to a simulcast alternative.

Introduction

The MPEG-4 Audio coding tools cover a bitrate range from 2 kbit/s to 64 kbit/s with a corresponding subjective audio quality that needs to be evaluated. It was recognised that the verification tests should first address applications that are potentially of great interest for users. Three main applications for this first round of MPEG-4 audio verification tests have been identified [1]:

• Internet Audio applications
• digital audio broadcasting on AM modulated bands (16 to 24 kbit/s) and
• speech applications

The NADIB (Narrow Band Digital Broadcasting) consortium proposed to carry out the MPEG-4 verification tests for digital audio broadcasting based on their proposals [2] [3] Two different sites offered to run the listening tests : TERACOM (Stockholm - Sweden) and CCETT (Rennes - France). The final results analysis was performed by MIT (USA).

The whole process was completed by the beginning of June, and it is the purpose of this document to describe the procedures that have been followed and to present the outcome of the tests.

All the specifications listed below refer only to the NADIB tests, the two other verification tests being handled in separate documents.

Conclusions

The methodology and the statistical analysis of the results of formal listening tests that were conducted for the NADIB portion of the MPEG-4 audio verification testing have been presented. Several audio and speech coders have been tested. Please note that the speech coders were not designed for music which is present in several items used in this test.

The results of the analysis permitted the questions posed in the NADIB test plan to be answered and some general conclusions to be drawn for these tests.

• digital Audio in AM Bands using 24 kbit/sec audio coding has the potential to offer better quality compared to existing Analogue Modulation techniques,
• one layer encoding provides the best performance for a given net bit rate,
• if two levels of quality of service are targeted, one for normal reception conditions and one for bad reception conditions, a layered scaleable codec offers better quality than simulcast,
• in the scaleable mode, AAC+NBCELP is the best choice when programmes contain speech.
• with the encoders provided for this test, in the scaleable mode, when programmes contain music, TwinVQ performs better than NB-CELP but AAC+NBCELP performs somewhat better than AAC+TwinVQ.
• in the scaleable mode, AAC+NBCELP appears to be the best compromise for generic audio broadcasting amongst the coders in this test.
• on the material used in this test Twin VQ at 6 kbps performed worse than G.723.1 and NB CELP with the exception of some music items. TwinVQ is known to have lower quality for speech signals and also for speech+music, probably because speech signals are dominant when the bandwidth is limited below 4 kHz. Note that the TwinVQ under test directly quantizes input signals sampled at 24 kHz for the purpose of simple connection to AAC scaleable system. According to previous experience, it can be assumed that TwinVQ will achieve higher quality at the same bitrate if a lower sampling rate is used in the unscaled mode.
• the performance of the WB-CELP codec depends on the material to be encoded. For the speech items the results of WB-CELP and perfect AM are statistically equivalent (Note: this was not computed but concluded from the figures), for music items Perfect AM performs better.
• the NB-CELP test results are statistically equivalent to those of ITU-T G723.1 at CCETT or slightly worse at Teracom (although the confidence intervals of both coders overlap at Teracom, the coders are statistically different as a result of the Student t-test analysis). It has to be noted that the NB-CELP coder operates at a lower bitrate and has a shorter delay (see chapter 4).