INTERNATIONAL ORGANISATION FOR STANDARDISATION
ORGANISATION INTERNATIONALE DE NORMALISATION
ISO/IEC JTC1/SC29/WG11
CODING OF MOVING PICTURES AND AUDIO

ISO/IEC JTC1/SC29/WG11
N3641
October 2000, La Baule

Source: Audio Sub-group
Title: Call for Evidence Justifying the Testing of Audio Coding Technology

Call for Evidence Justifying the Testing of Audio Coding Technology

1. Introduction

In mid-1999 International Standard ISO/IEC 14496-3, MPEG-4 Audio Version 1 issued and in early 2000 ISO/IEC 14496-3 / AMD1, MPEG-4 Audio Version2 issued. Numerous tests have been conducted by MPEG (see references) to verify that the MPEG-4 standard contains state of the art technology. However, WG11 is always interested in new developments which may provide improvements over the existing MPEG-4 standard and which may lead to extensions of MPEG-4 or to new work items. For this reason MPEG seeks input on such new technology.

Therefore, WG11 issues with this document a call for evidence justifying the subjective testing of new audio coding technology in comparison with the MPEG-4 audio coding technology.

Interested parties are asked to provide at the 55th MPEG meeting, to be held January 15-19, 2001 in Eilat Israel, clear evidence that their technology outperforms MPEG-4 technology (see detailed timetable, below). In the spirit of MPEG-4, it is of greatest interest if the new technology demonstrates both compression and other functionality. WG11 shall judge the submitted material to assess if the proposed technology represents a significant enough improvement to warrant further quality assessment via a formal subjective test.

If there is such a need, these tests will be defined by WG11 and conducted under controlled conditions. Should a formal subjective test be conducted, proponents of the technology will be requested to underwrite the cost. Responding to this call does not imply any commitment on the part of the proposer; a decision to take part in the formal testing process can be made when the results of the Eilat meeting are available. Results of a formal subjective test will be made public, but WG11 cannot, prior to having the results of the test, commit to any course of action regarding the proposed technology.

In order to prepare for evaluations of proposed technologies at the January 2001 meeting, proposers are kindly requested to do the following:

Registration:

Register by 18th October 2000 an intention to compete with MPEG-4. Register by sending an email to Schuyler Quackenbush (Chairman of the MPEG Audio Subgroup, srq@research.att.com). Email should indicate contact names, company and the technology in MPEG-4 Audio that will be considered (e.g. audio coding, scalable coding or wideband speech coding).

NOTE: Registrants and procedures for access to both MPEG-4 and proponent encoded material, decoders and decoded material are show in ANNEX I

Documentation:

Submit by 1st December 2000, the following:

the bitstreams, decoders and decoded sound files (*.wav) associated with the proposed algorithms. At the same time the corresponding items for the MPEG-4 technology will be submitted and made available. Decoders shall be delivered as executables on any commonly available computing platform.

Submit by 20th December 2000, the following:

the documents that describe the performance of proposed algorithms in comparison to MPEG-4 technology.

The proposer's documents should be written in Microsoft Word97 and submitted via email to Schuyler Quackenbush. These documents will be uploaded to the MPEG document site as an input to the January MPEG meeting. The bitstreams, decoder executables and *.wav files should be uploaded to an FTP site indicated to a proposer in response to their registration. These files will be made accessible to members of WG11 as they become available.

Proposers should base their evidence on "reference quality" MPEG-4 encoders, similar to those that have been used in the MPEG verification tests. In most cases the publicly available MPEG-4 encoder software is not able to deliver "reference quality." Proposers should contact Schuyler Quackenbush for information on how to obtain both test material and "reference quality" encoded test material. There may be a reasonable fee associated with access to the reference quality encoder or encoded material. Every effort will be made to share these expenses amongst all the proposers that benefit.

It is required that the MPEG-4 coders used in the comparison be compliant MPEG-4 coders and that they adhere to test conditions and bitrates described below.

WG11 must emphasize that evidence presented as part of this call should not be interpreted as definitive subjective quality assessments. Such interpretations require an appropriately designed and conducted formal subjective test.

Participation:

Attend the MPEG meeting, 15-19 January 2001 in Eilat Israel. It is strongly urged that experts familiar with the proposed technology attend in order to allow discussions on details of the proposals. Proposers should bring to the meeting the decoded audio material associated the comparison between proposed technology and MPEG-4 technology on some appropriate media (e.g. DAT tape, audio CD or *.wav files) along with hardware to play that media for evaluation purposes.