INTERNATIONAL ORGANISATION FOR STANDARDISATION
ORGANISATION INTERNATIONALE DE NORMALISATION
ISO/IEC JTC1/SC29/WG11
CODING OF MOVING PICTURES AND AUDIO

ISO/IEC JTC1/SC29/WG11
N3483
July 2000, Beijing, China

Source: Audio Sub-group
Title: Call for Evidence Justifying the Testing of Audio Coding Technology

Call for Evidence Justifying the Testing of Audio Coding Technology

1. Introduction

In mid-1999 Iinternational Sstandard ISO/IEC 14496-3, MPEG-4 Audio Version 1 issued, and in early 2000 ISO/IEC 14496-3 / AMD1, MPEG-4 Audio Version2 issued. Numerous tests have been conducted by MPEG (see references) to verify that the MPEG-4 standard contains state of the art technology. However, WG11 is always interested in new developments which may provide improvements over the existing MPEG-4 standard and which may may lead to extensions of MPEG-4 or to new work items. For this purposereason, MPEG seeks input on such new technology.

Therefore, WG11 issues with this document a call for evidence justifying the subjective testing of new audio coding technology in comparison with the MPEG-4 audio coding technology. , which it intends to review at the 55^th MPEG meeting, January 15-19, 2001, in Eilat Israel.

Therefore, WG11 issues with this document a call for evidence justifying the subjective testing of audio coding technology in comparison with the MPEG-4 audio coding technology.

Interested parties are asked to provide at the 55^th MPEG meeting, to be held January 15-19, 2001 in Eilat Israel, January 2001 Eilat MPEG meeting clear evidence that their technology outperforms that of MPEG-4 technology (see detailed timetable, below). In the spirit of MPEG-4, it is of greatest interest that if the new technology demonstratetechnology demonstrates both compression and other functionality. It is required that proposers provide results compared to compliant MPEG-4 audio coders. WG11 shall then judge the submitted material to assess if the proposed technology represents a significant enough improvement to warrant further quality assessment via a to make an assessment on the need for formal subjective test tests.

If there is such a need, these tests will be defined by WG11 and conducted under controlled conditions. Should a formal subjective test be conducted, proponents of the technology will be required requested to underwrite the cost. /* Why only for expenses connected to subjective tests ? */ . Responding to this call does not imply any commitment on the part of the proposer; a decision to take part in these teststhe formal testing process can be made when the results of the Eilat meeting are available. Results of a formal subjective test will be made public, but WG11 cannot, prior to having the results of the test, commit to any course of action regarding the proposed technology.

In order to prepare for evaluations of proposed technologies at the January 2001 meeting, proposers are kindly requested to do the following:

Registration:

Register by 18^th October 2000their an intention to compete with MPEG-4 by 29^th December 2000. RegistratRegisterion is done by sending an email to Dr. Schuyler Quackenbush (Chairman of the MPEG Audio Subgroup, srq@research.att.com). Email should indicateing contact namesnames, company and the technology in MPEG-4 Audio that will be considered (e.g. audio coding, scalable coding or wideband speech coding).

Submit by 1^st December 2000, the following:

the bitstreams, decoders and decoded sound files (*.wav) associated with the proposed algorithms. At the same time the corresponding items for the MPEG-4 technology will be submitted and made available. Decoders shall be delivered as executables on any commonly available computing platform.

Submit by 5^th January20^th December 2001,2000, the following:

the documents that describe the performance of proposed algorithms in comparison to MPEG-4 technology.

The proposer's documents should be written in Microsoft Word97 and submitted via email to Schuyler Quackenbushvia email, a Microsoft Word97 document to Dr. Schuyler Quackenbush (srq@research.att.com). This document shall describe the performance of proposed algorithms in comparison to MPEG-4 technology.. These documents will be uploaded to the MPEG document site as an input to the January MPEG meeting. The bitstreams, decoder executables and *.wav files should be uploaded to an FTP site indicated to a proposer in response to their registration. These files will be made accessible to members of WG11 as they become available.

It is strictly required that the test conditions and bitrates described below are employed in the comparison. This document will be uploaded to the MPEG document site as an input document to the January MPEG meeting. Proposers should base their evidence on "reference quality" MPEG-4 encoders, similar to those that have been used in the MPEG verification tests. In most cases the publicly available MPEG-4 encoder software is not able to deliver "reference quality." Proposers should contact Schuyler Quackenbush for information on how to obtain both test material and "reference quality" encoded test material. There may be a reasonable fee associated with access to the reference quality encoder or encoded material. Every effort will be made to share these expenses amongst all the proposers that benefit.

It is required that the MPEG-4 coders used in the comparison be compliant MPEG-4 coders and that they adhere to test conditions and bitrates described below.

WG11 must emphasize that evidence presented as part of this call should not be interpreted as definitive subjective quality assessments. Such interpretations require an appropriately designed and conducted formal subjective test.

To bring to the MPEG meeting results of comparison between proposed and MPEG-4 technology on some appropriate media (e.g. DAT tape, audio CD or *.wav files) along with hardware to play that media for evaluation purposes.

Participation:

It is strongly urged that proposers to Aattend the MPEG meeting, January MPEG 15-19 January 2001 in Eilat Israel. It is strongly urged that experts familiar with the proposedmeeting in person technology attend in order to allow discussions on details of the proposals. Proposers should bring to the meeting the decoded audio material associated the comparison between proposed technology and MPEG-4 technology on some appropriate media (e.g. DAT tape, audio CD or *.wav files) along with hardware to play that media for evaluation purposes.

Proposers should base their evidence on "reference quality" MPEG-4 encoders, similar to those that have been used in the MPEG verification tests. In most cases the publicly available MPEG-4 encoder software is not able to deliver "reference quality." Proposers should contact Dr. Schuyler Quackenbush for information on how to obtain both test material and encoded test material.