INTERNATIONAL ORGANISATION FOR STANDARDISATION
ORGANISATION INTERNATIONALE DE NORMALISATION
ISO/IEC JTC1/SC29/WG11
CODING OF MOVING PICTURES AND AUDIO

ISO/IEC JTC1/SC29/WG11
MPEG97/N2006
Feb 1998/San Jose

Source: Audio Subgroup
Title: Report on the MPEG-2 AAC Stereo Verification Tests
Authors: David Meares (BBC), Kaoru Watanabe (NHK), Eric Scheirer (MIT Media Labs)

Report on the MPEG-2 AAC Stereo Verification Tests

Introduction

In November 1996, the details of the MPEG-2 Advanced Audio Coding (AAC) multichannel coding tests conducted at BBC and NHK were reported on in document WG11/N1419 [1]. Those tests showed a high quality performance for AAC at bit rates of approximately 320 kbps for 5-channel operation (64 kbps/ch). At that time, due to the differences in the use of a common bit reservoir and several joint processing strategies, the observation was made that although the 5-channel AAC had received a performance characterisation, it was still necessary to conduct separate tests of the stereo and mono performance of the AAC codecs.

As a consequence of that observation, the methodology and details of AAC formal stereo tests, both in the context of the established MPEG-2 (ISO/IEC 13818-7) standard and the forthcoming MPEG-4 (ISO/IEC 14496-3) standard were set forth in document WG11/N1845 [2].

Those tests have now been completed, and it is the purpose of this document to report the procedures, details and results of the tests.

Conclusions

The assessment of the stereo performance of AAC Main, Low Complexity and SSR Profiles have been carried out in comparison with one another and with MPEG-1 codecs at various representative bitrates. The conduct of these tests involved the mutual co-operation and support of a large number of MPEG members and their organisations. The overall conclusion is that , when auditioning using loudspeakers, AAC coding according to the ISO/IEC 13818-7 standard gives a level of stereo performance superior to that given by MPEG-1 Layer II and Layer III coders.

The test process was intended to answer the following set of questions which form the detailed conclusions of this study:

Is the performance of AAC codecs at the tested bitrate equal to or better than the performance of MPEG-1 Layer II and Layer III?

Section 10.6 presents the answer to this. Overall, all AAC profiles at 128 kbps give significantly better performance than do MPEG-1 Layer II at 192 kbps or Layer III at 128 kbps. Therefore the goal of high audio quality at 64 kbps per channel for MPEG-2 AAC has been achieved. Both AAC Main Profile and Low Complexity Profile provide quality at 96 kbps that is comparable to MPEG-1 Layer II at 192 kbps, and therefore give a 2 to 1 compression advantage. In addition, AAC Main Profile at 96 kbps gives better results than MPEG-1 Layer III at 128 kbps.

Are the listeners' results reliable, i.e. distinguishable from random votes?

The analysis conducted in Section 10.2 concludes that all listeners returned reliable results.

Does the test methodology allow meaningful conclusions to be drawn from these results?

The analysis confirms that meaningful conclusions can be drawn from these results. The effect SEAT was shown to be significant in its own right, particularly with reference to the rearmost seat position. However, as there were only a limited number of listeners who used this position and as the results from the other two seats could be combined, the further analysis only made use of the results from the front and centre seat positions.

How does the performance of the codecs vary with programme items?

This is shown in full in section 11. It is shown that all coders perform, to some extent, differently depending on the type of programme item with which they are being tested.

Is the performance of the coding of AAC codecs at the tested bitrate distinguishable from the original signal?

Sections 10.7 and 11 show that there is a statistical difference between the source and coded items, both overall and for some specific items. However, there were a large number of items for which no difference was recorded.

Is the performance of AAC codecs at the tested bitrate achieving ‘indistinguishable quality’ in the EBU definition of that phrase?

AAC Main Profile at 128 kbps and AAC Low Complexity Profile at 128 kbps both provided ""indistinguishable quality" and AAC SSR Profile at 128 kbps failed to achieve this by a margin of less than 1% relative to the decision criterion.

Is the following requirement of ITU-R Recommendation BS.1115 [5] fulfilled?
"For emission, the most critical material for the codecs must be such that the degradation may be 'perceptible but not annoying' (grade 4)"

AAC Main Profile at 128 kbps passes this criterion.

What is the relative ranking of the codecs tested?

The relative rankings are presented in section 10.10.

Are there any other features from the data that should be reported?

Comments relating to the significance of listener position are reported in Section 10.3.