ISO/IEC JTC1/SC29/WG11

MPEG 98/N2464

October 1998, Atlantic City, USA

More and more audiovisual information is available in digital form, in various places around the world. Along with the information, people appear that want to use it. However, before any information can be used, it will have to be located. At the same time, the increasing availability of potentially interesting material makes this search harder. Currently, solutions exist that allow searching for textual information. Many text-based search engines are available on the World Wide Web, and they are among the most visited sites - indicating a real demand. The same is, however, not possible for audiovisual content, as no generally recognized description of this material exists. In general, it is not possible to efficiently search the web for, say, a picture of ‘the Motorbike from Terminator II’, or to search a sequence where "King Lear congratulates his assistants on the night after the battle," or to search for "twenty minutes of video according to my preferences of today". In specific cases, solutions do exist. Multimedia databases on the market today allow searching for pictures using characteristics like color, texture and information about the shape of objects in the picture. One could envisage a similar example for audio, in which one can whistle a melody to find a song. While the capability to search for relevant information is a direct result of having standardised descriptions, several other applications can be envisaged. For example, one may wish to generate a paragraph of text describing an image or generate a different presentation of a video using its description.

MPEG started a new work item to provide a solution to the questions described above. The new member of the MPEG family, called "Multimedia Content Description Interface" (in short ‘MPEG-7’), will extend the limited capabilities of proprietary solutions in identifying content that exist today, notably by including more data types. In other words: MPEG-7 will specify a standard set of descriptors that can be used to describe various types of multimedia information. This description shall be associated with the content itself, to allow fast and efficient searching for material of a user’s interest. AV material that has MPEG-7 data associated with it, can be indexed and searched for. This ‘material’ may include: still pictures, graphics, 3D models, audio, speech, video, and information about how these elements are combined in a multimedia presentation (‘scenarios’, composition information). Special cases of these general data types may include facial expressions and personal characteristics.

The question of finding content is not restricted to database retrieval applications, which can be viewed as "pull" applications, but the question of finding content also exists in many other areas. For instance, there is an increasing amount of (digital) broadcast channels available, and this makes it harder to select the broadcast channel (radio or TV) that is potentially interesting. These applications of finding content within broadcast channels are referred to as "push" applications. MPEG-7 will address both "pull" and ‘push" applications. While searhing and filtering constitute a large class of applications, MPEG-7 does not limit itself to such applications alone. MPEG-7 descriptions will support other applications which can exploit a standardized description.

Providing novel solutions addressing text-only documents (i.e. standardising descriptions for text-only documents) is not among the goals of MPEG-7. The emphasis is on description of audiovisual content. The content may include text, or references to text, in addition to audiovisual information. MPEG-7 will consider existing solutions developed by other standardization groups for text only documents and support them as appropriate.

MPEG-7 will standardize the description of Audio-Visual Content, but will not standardize the algorithms for extracting the descriptions or searching / filtering using these descriptions. For details on the background and goals of MPEG-7 please refer to MPEG-7: Context and Objectives document [2].

MPEG-7 will address applications that can be stored (on-line or off-line) or streamed (e.g. broadcast, push models on the Internet), and can operate in both real-time and non real-time environments. A ‘real-time environment’ means that information is associated with the content while it is being captured.

MPEG-7 will standardize:

• A set of Descriptors (Ds)
• A set of Description schemes (DS)
• A language to specify description schemes, i.e. a Description Definition Language (DDL).
• Schemes for coding the descriptions

For a definition of the terms please refer to the MPEG-7 Requirements document [1].

It may be pointed out that while MPEG-7 aims to standardize a "Multimedia Content Description Interface, the emphasis of MPEG is on audio-visual content. That is, MPEG-7 does not aim to create description schemes or descriptors for text medium. However, MPEG-7 will consider existing solutions for describing text documents and support them as appropriate with suitable, necessary interfaces between audio-visual-content descriptions and the textual-content descriptions. Specifically MPEG-7 will:

• Embrace existing standard descriptions for text (e.g. HTML, SGML, RDF, etc.) if they are appropriate for the MPEG-7 descriptions for audio-visual data.
• Standardize the linking between descriptions of audio-visual data and text data.

MPEG-7 will study existing schemes for multimedia content description and act as appropriate based on the requirements of MPEG-7. However, the need to comply or inter-operate with existing database systems describing multimedia content will not be a constraint on MPEG-7.

For more details regarding the MPEG-7 background, goals, areas of interest, and work plan please refer to MPEG-7: Context and Objectives document [ 2] . A detailed list of functionality and requirements of MPEG-7 may be found in the MPEG-7 Requirements document [1]. For a list of potential applications which may be enabled by MPEG-7 please refer to the MPEG-7 Applications document [3].

MPEG calls for the following tools in order to develop the MPEG-7 standard:

• Descriptors
• Description Schemes
• Description Definition Language (DDL)
• Coding schemes for compact representation of Descriptions.
• Systems tools addressing the MPEG-7 Systems requirements specified in the MPEG-7 Requirements document [1]

While descriptors, description schemes, DDL and coding schemes proposals will be evaluated in February 1999 following the procedures defined in the MPEG-7 Evaluation document [4], the proposals addressing MPEG-7 systems tools will not be part of the MPEG-7 Evaluation process in February 1999. These tools will be considered at the Seoul MPEG meeting. The proposers are kindly invited to present and show a demo regarding these tools at the Seoul MPEG meeting, in March 1999 [4].

B. For the development of the standard (to be used in the eXperimentation Model (XM) and play an important role in the standardization process):

• Extraction methods
• Search Methods
• Evaluation and validation techniques

It should be noted that it is not necessary for a proposal to address all the elements called above. It is possible for a proposal, for example, to only propose descriptors for some set of features, or description schemes or parts of description schemes. The major purpose of the MPEG-7 evaluation process is to gather relevant technology for the collaborative development of the MPEG-7 standard.

Selected aspects of different proposals will be incorporated into a common model during the collaborative phase of the standard with the goal of building the best possible model. The commom model called the experimentation model (XM) may be implemented in more than one platform/architecture and may be implemented in different parts to satisfy different requirements and/or application areas.

The proposer should provide:

1. A proposal document according to the format included in Annex H of the MPEG-7 Evaluation document [4]. The proposal must highlight all information that will help MPEG in identifying the strengths of the proposal.
2. The proposer is strongly encouraged to give a presentation/demonstration of the proposal using relevant MPEG-7 test sets at the evaluation meeting in Lancaster. If the MPEG-7 test set is insufficient for the demonstration, another data set may be used. In this case the data set used must be made available to MPEG on the same conditions as the MPEG-7 Test and Evaluation Material (see annex I).

The MPEG-7 content set has been defined at the MPEG October 1998 meeting. The list of the content set items is included in doc. N2467 and in annex H of the MPEG-7 Evaluation document [4]. The conditons of use of the MPEG-7 content set are include doc. N2466 and in annex I of the MPEG-7 Evaluation document. The distribution procedure regarding the MPEG-7 content set is defined in doc. 2468.

The major processes in the development of the standard are as follows:

1. Evaluation of Proposals: On receiving the proposals along with the associated questionnaires and demonstrations (if available) they will be reviewed and evaluated by MPEG. The evaluations will be based on the detailed paper descriptions, questionnaires, performance results provided by proposer and tests conducted by MPEG, and demonstrations. This process will include both objective as well as subjective methodologies and will be carries out by a panel of experts. For details of the evaluation and testing procedures please refer to the MPEG-7 Evaluation Document [4].
2. eXperimentation Model: The initial eXperimental Model (XM) will be built from the screened proposals. This XM is a "common framework" on which relative evaluations and improvements of tools will be carried out. This XM may contain several tools for the same functionality and will have components for evaluating and improving the DDL, DS and Ds. If any part of a proposal is selected for inclusion in the XM, the proposer must provide the corresponding source code for integration into the XM, according to the conditions to be set by MPEG.
3. Core Experiments: The relative evaluation and improvements will start with a first set of Core Experiments which will be defined at the conclusion of the initial evalution of proposals. Improvements and additions to the eXperimentation Model (XM) will be based upon the results of core experiments. In general, core experiments require independent verification of results before the proposed change can be included in the XM. Proposers whose technique are accepted for inclusion in the XM must provide the corresponding source code for the XM, according to the conditions to be set by MPEG.

1. MPEG-7 Requirements Document , N2461, Atlantic City, October 1998.
2. MPEG-7 Context and Objectives Document, N2460, Atlantic City, October 1998.
3. MPEG-7 Applications Document, N2462, Atlantic City, October 1998.
4. MPEG-7 Evaluation Document, N2463, Atlantic City, October 1998.
5. MPEG-7 Evaluation Meeting Announcement, N2465, Atlantic City, October 1998.
6. Licensing Agreement for MPEG-7 Content Set, N2466, Atlantic City, October 1998.
7. MPEG-7 Set of Test Material, N2467, Atlantic City, October 1998.
8. MPEG-7 Content Distribution Process, N2468, Atlantic City, October 1998.