This ‘MPEG-7 Applications Document’ lists a number of applications that should be enabled by MPEG-7 tools. It does certainly not list all the applications enabled by MPEG-7, but rather gives an idea of what should be possible using MPEG-7 technology, including improving existing applications as well as presenting completely new ones.

The purpose of the document is:

• to provide a better understanding of what MPEG-7 should be, and what functionality it should deliver,
• to be a ‘Public Relations’ instrument that can help explain what MPEG-7 is, and
• to be of use when writing the concrete requirements for MPEG-7.

For each of the applications, four sections are given:

Nowadays, more and more audio-visual information is available from many sources around the world. Also, there are people who want to use this audio-visual information for various purposes. However, before the information can be used, it must be located. At the same time, the increasing availability of potentially interesting material makes this search more difficult. This challenging situation led to the need of a solution to the problem of quickly and efficiently searching for various types of multimedia material interesting to the user. Moreover, MPEG-7 is not only enables this type of search, but also enables filtering. Thus, MPEG-7 will support both push and pull applications. MPEG-7 wants to answer to this need, providing this solution.

MPEG-7, formally called ‘Multimedia Content Description Interface’, will standardise:

• A set of description schemes and descriptors
• A language to specify description schemes, i.e. a Description Definition Language (DDL).
• A scheme for coding the description

For more details regarding the MPEG-7 background, goals, areas of interest, and work plan please refer to document N2460, "MPEG-7: Context and Objectives" [1]. MPEG-7’s initial requirements are indicated in document N2461, "MPEG-7 Requirements" [2].

The increased volume of audio-visual data available in our everyday lives requires effective multimedia systems that make it possible to access, interact and display complex and inhomogeneous information. Such needs are related to important social and economic issues, and are imperative in various cases of professional and consumers applications such as:

• Education,
• Journalism (e.g. searching speeches of a certain politician using his name, his voice or his face),
• Tourist information,
• Cultural services (history museums, art galleries, etc.),
• Entertainment (e.g. searching a game, karaoke),
• Investigation services (human characteristics recognition, forensics),
• Geographical information systems,
• Remote sensing (cartography, ecology, natural resources management, etc.),
• Surveillance (traffic control, surface transportation, non-destructive testing in hostile environments, etc.),
• Bio-medical applications,
• Shopping (e.g. searching for clothes that you like),
• Architecture, real estate, and interior design,
• Social (e.g. dating services), and
• Film, Video and Radio archives.

A preliminary note on the division of this document:

MPEG-7 began its life as a scheme for making audio-visual material "as searchable as text is today." Although the proposed multimedia content descriptions are now acknowledged to serve much more than search applications, they remain for many the primary applications for MPEG-7. These retrieval, or "pull" applications, involve databases, audio-visual archives, and the web-based internet paradigm (a client requests material from a server.)

• interoperability between video archive operators,
• perennial relevance of the meta-data, and
• a wider diffusion of the data to the professional and the general public.

MPEG-7, in short, must accommodate visual and other search of such existing multimedia databases.

In addition, a vast amount of the older, analogue audio-visual material will be digitised in years to come, which creates a tremendous opportunity to include content-based indexing features (which can be extracted during the digitisation/compression process) into those existing data-bases.

In the case of new audio-visual material, the ability to associate descriptive information within video streams at various stages of video production can dramatically improve the quality and productivity of manual, controlled-vocabulary annotation of video data in a video archive. For example, pre-production and post-production scripts, information captured or annotated during shooting, and post-production edit lists would be very useful in the retrieval and re-use of archival material.

Essential associated activities to this one are cost-efficient video sequence indexing and shot-level indexing for stock footage libraries [4].

A sample architecture is outlined in Annex B.

Specific requirements for those applications are:

• Support of full-text descriptions as well as structured fields (database descriptions);
• Multi-language support;
• We desire the ability to interoperate between different content description semantics (e.g. different database schemas, different thesauri, etc.) or to translate from each content description semantic into MPEG-7 semantics;
• The ability to reference audio-visual objects or object instances and time references, even in analogue format;
• The ability to include descriptions with incomplete or missing time references (a shot description that has not been timecoded);
• The ability to handle multiple versions of the same document at several stages in the production process, and descriptions that apply to multiple copies of the same material.

Requirements specific to MPEG-7 are:

• There must exist a class of descriptors that support unstructured (free-) text in multiple languages.

• There must exist a class of descriptors that support structured text.
• There must exist a mechanism by which different MPEG-7 DS’s can interoperate.
• There must be a robust linking mechanism that allows for temporal references to material with incomplete timecode.
• There must be a mechanism by which document versions may be identified.

In this application, due to the users’ expertise, one formulates relevant and possibly detailed "pull" queries, which specify the desired features of some video segments. With present video databases, these queries are mainly based on objective characteristics at segment level. However, they can also take advantage of subjective characteristics of these segments, as perceived by one or several users.

The ability to formulate a single query on the client side, and send it to many distributed databases is very important to many production studios. The returned items should include visual abstracts, copyright and pricing information, as well as a measure of the technical quality of the source video material.

In this application, one should separate news programs, which must be made widely and instantly available for a short period of time, from other production programs, which can be retrieved on a permanent basis, usually from secondary or tertiary storage. On-line news services providing instant access to the day’s news footage are being built by many broadcasters and archives (including INA, BBC, etc), using proprietary formats, and would benefit from standardisation if they were to be consolidated into common services such as the Eurovision News Exchange (which currently uses broadcast channels, not databases).

Still pictures have similar applications and requirements as pertaining to design. The web designer must not only make new designs but also collect the already available graphics on the net for use in the designed web sites. Other design fields have similar uses for visual search.

Requirements are similar to the previous application. They are mainly characterised by:

• Support of feature-based and concept-based queries at segment level, and
• Support for similarity queries.
• Support for different data formats.
• Support for art & design specific parameters.
• Media summaries for fast browsing.

MPEG-7 Specific Requirements

Requirements specific to MPEG-7 are:

• There must exist a mechanism that embodies conceptual knowledge.
• There must exist a mechanism that allows for links to media summaries.

see also 4.1 ‘Application relevant work and references’, and Annex A.

A similar application dealing with Karaoke, but also relating to music sales, below, is enabling solo Karaoke-ists to expand their repertoire in the privacy of their own home. Much of the industry is currently driven by people wishing to practice in their own homes. One can easily imagine a complete on-line database, in which someone selects a song she knows from the radio, sings a few bars, and the entire arrangement is downloaded to their computer, with appropriate payment extracted.

The consumer music industry is currently struggling with how to reach consumers with increasingly fragmented tastes. Music, as with all broadcast media artefacts, is undergoing the same internet-flavoured transformation as cable television: tastes are changing to prefer narrowcast over broadcast. An ideal way of presenting consumers with available music is to allow them effortless search.

The mechanics are similar to the above Karaoke example. Querents may hum approximate renditions of the song they seek from a kiosk or from the comfort of their own home. Alternately, they may seek out music with similar features (musicians, style, tempo, or year of creation) to those that they already know. From there, they may listen to an appropriate sample (and perhaps view associated information such as lyrics or a video), and choose to buy the music on the spot.

• Robust representations of melody and other musical features which allow for reasonable errors on the part of the indexer in order to accommodate query-by-humming,
• Associated information, and
• Cross-modal search.

Requirements specific to MPEG-7 are:

• There must exist mechanism that supports melody and other musical features.
• There must exist a mechanism that supports descriptors based on information associated with the data (e.g., textual data).
• Support description schemes that contain descriptors of visual, audio, and/or other features, and support links between the different media.

The area of query-by-humming may be the most-researched field within auditory query by content. A few example papers include:

A sound designer may specify a sound effect type, for example, naming the source of the sound, and select from variations on that sound. A designer may provide a prototypical sound, and detail features such as, "bigger, more distant, but keeping the same brightness." One may even vocalise the type of abstract sound one seeks, in an onomatopoetic variation of query-by-humming. Essential to the application is the ability to navigate a space of similar sound effects.

• Compact representation of sound effects,
• Sound source name and characteristics, and
• Ability to specify classes of audio-visual objects, with features to accommodate selection.

Requirements specific to MPEG-7 are:

• Support for hierarchical descriptors and description schemes.
• Support for text based descriptors.
• Efficient coding of descriptors.

• Representation of textual content of an auditory event.

Requirements specific to MPEG-7 are:

• Support for text based descriptors.

[to come]

In a similar vein, auditory events in soundtracks may be just as accessible as spoken lines in certain circumstances. A key example is the screeching violins in the "Psycho" soundtrack at the point of the infamous shower scene. Those repeated harsh notes ("Scree-ee-ee-ee!") are iconic to a movie-going public, and a key feature of an important movie.

• Search by example audio.

Requirements specific to MPEG-7 are:

• Support for audio descriptors.

[to come]

• Word-in mark

• Device mark

• Composite mark

If a mark is registered, then no person or enterprise other than its owner may use it for goods or services identical with or similar to those for which the mark is registered. Any unauthorized use of a sign similar to the protected mark is also prohibited, if such use may lead to confusion in the minds of the public. The protection of a mark is generally not limited in time, provided its registration is periodically renewed (typically, every 10 years) and its use continues. Therefore, this number is expected to keep growing rapidly, and it is estimated that the number of registrations and renewals of marks effected worldwide in 1995 was in the order of millions.

In order to register a mark, one has to make sure no identical ones are registered before. For the types of "Word-in mark" and "Composite-mark," text annotation may be adequate for the retrieval from the database. "Device-mark" type, however, is characterized only by the shape of the object. In addition, this type may not have distinct orientation or scale. When the operator enters a new mark to database for registration, he/she wants to make sure that no identical one is already in the system in disregard of its orientation angle or scale. Furthermore, he/she may want to see how similar shaped ones are already in the system even if there is no identical one. The search process should be robust to noise in image or minor variations in its shape. Any relevant information such as annotation or textual description of the mark should also be accessible if requested.

A mark designer may want the same thing. In addition, to avoid possible inadvertent infringement of the copyright, the designer may wish to see whether some possible variations of the mark under design are already registered.

In this respect, it is desirable for the system capable of returning the retrieved results in terms of the similarity, and displaying the results simultaneously for comparison. So far, the current practice to retrieve similar or the same device-type mark is performed manually by human operator resulting in many duplicated registrations.

Therefore, there is an enormous potential need for an automatic retrieval of marks by the contents-based similarity not only in the international community but also in domicile. The submission of the mark to the system can be done interactively on-line to refine the search-process on a web-based Internet paradigm [5].

Specific requirements for those applications are:

• Efficient interactive response time.
• Support for a mechanism by which a mark image may be submitted for similarity based retrieval.
• Support for visual based descriptors by which modifications can be made to any of the retrieved results for fine-tuning the search-process (relevance feedback).

Requirements specific to MPEG-7 are:

• Support for shape-based and content-based queries;
• Support for precise, shape-oriented similarity queries;
• Support for scale and orientation independence of marks;
• Support for media summaries for fast browsing;
• Support for linking relevant information
• Support for D’s and DS’s invariant under transformations irrelevant to the intended features

In contrast with the above "pull" applications, the following "push" applications follow a paradigm more akin to broadcasting, and the emerging webcasting. The paradigm moves from indexing and retrieval, as above, to selection and filtering. Such applications have very distinct requirements, generally dealing with streamed descriptions rather than static descriptions stored on databases.

Broadcast media are unlikely to disappear any time soon. In fact, there is a movement to make the World Wide Web, primarily a pull medium, more broadcast-like. If we can enable users to select information more appropriate to their uses and desires from a broadcast stream of 500 channels, using the same meta-information as that used in search, then this is an application for MPEG-7.

This application gives rise to several sub-types, primarily divided among types of users. A consumer-oriented selection gives rise to personalised audio-visual programmes, for example. This can go much farther than typical video-on-demand in collecting personally relevant news programmes, for example. A content-producer oriented selection made on the segment or shot level is a way of collecting raw material from archives.

• Efficient interactive response times, and
• The capability to characterise a media object by a set of concepts that may be dependent on locality or language.

Requirements specific to MPEG-7 are:

• Support for descriptors and description schemes that allow multiple languages.
• There must exist a mechanism by which concepts may be represented

In the broadcast area, the MPEG-7 description can provide the user with assistance in selection of broadcast data, be it for immediate or later viewing, or for recording. In a personalized broadcast scenario, the data offered to the user can be filtered from broadcast streams according to his own profile, the generation of which may be done automatically (e.g. based on location, age, gender or on the previous selection behavior) or semi-automatically (e.g. based on pre-set interests). The broadcast of MPEG-7 description streams will enable providers of Electronic Programme Guides (EPGs) with a variety of capabilities, wherein presentation of MPEG-7 data (also along with the original AV data) will also be an important aspect. In combination with NVOD (Near-Video on Demand) services and recording, new functionalities like stepping forward/backward based on keyframe selection and changes in the sequel of scenes for speed-up in presentation are possible. Extended interactivity functionalities, related to specific events in the programmes, are of importance for future broadcast services as well. This can include "offline" interactivity based on a recorded broadcast stream, which can require identification of the associated event during callback. It can be expected that MPEG-7 data will be transmitted along with the AV data streams, or (e.g. for an EPG channel) also as separate streams [6].

• Description of broadcast media objects in terms of, for example, content type, author, cast, parental rating, textual description, temporal relationship, locality- and language-dependent features, service provider, and IP protection.
• Description of specific events in broadcast media objects
• Specification of interactivity capability related to specific events (e.g. polling) and definition of associated interaction channels (e.g. telephone numbers or hyperlinks)
• Presentation of content-related data, also along with the associated media objects, and manipulation of the presentation based on the content description (e.g. by event status)
• Support for APIs that define receiver-side filter function, interaction capability or definition of extended content description
• Support for unique presentation of the media objects and the associated content description, controllable by user interaction and description parameters
• The broadcast metadata to be put into MPEG-7 streams must be flexible, the broadcasters must be able to "spice" the content based on available resources. As little information as possible should be mandatory, such that broadcasters can start the service without a huge investment. Upwards compatibility with existing standards like DVB-SI, ATSC PSIP or metadata definitions originating from the EBU/SMPTE task forces also falls under this aspect.
• Capability to set up content- or service-related links between different broadcast media objects (not necessarily transmitted simultaneously)

Requirements specific to MPEG-7 are:

• Support for descriptors and description schemes that allow definition of broadcast media objects and events as listed above in their temporal (absolute and relative time bases, duration of validity) and physical (channel, stream) context
• Support for streaming of MPEG-7 data
• Support for interactivity, including identification of the related event during a callback procedure

• The ability to provide contextual and domain knowledge, and
• The ability to represent events and the temporal relationships between events.

Requirements specific to MPEG-7 are:

• There must exist a mechanism by which contextual information may be encoded.
• There must exist a mechanism by which temporal relationships may be represented

Thus, it is important to develop technical aids to facilitate communication for people with special needs. For example, a search agent that does not exclude images as information resource for the blind but rather makes available the MPEG-7 meta-data. Aided by that meta-data, sonification (auditory display), or haptic display is made possible. Similarity of meta-data helps to provide a set of information in different modalities, in case the particular information is not accessible for the user.

Such applications provide full participation in society by removing communication and information access barriers that restrict interactions between people with and without disabilities, and they will lead to improved global commerce opportunities.

[no new ones apparent]

Requirements specific to MPEG-7 are:

• Support description schemes that contain descriptors of visual, audio, and/or other features .

The following potential MPEG-7 applications do not limit themselves to traditional, media-oriented, multimedia content‘, but are functional within the meta-content representation to be developed under MPEG-7. They reach into such diverse, but data-intensive, domains as medicine and remote sensing. Such applications can only serve to increase the usefulness and reach of this proposed international standard.

Catalogues of items for which stock maintenance is difficult or expensive but for which the search process is essentially visual (e.g. garden design, architecture, interior decorating, oriental carpets) are especially aided by this application. For such items, detailed digital image-databases could be supported and updated centrally and accessed from distributed selling points.

• Support for interactive queries with few predicted constraints,
• Support for precise, product-oriented similarity queries, and
• MPEG-7 should operate "as fast as possible," allowing efficient interactive response times.

Requirements specific to MPEG-7 are:

• Support for visual based descriptors.

Biochemical interactions crucially depend on the three-dimensional structure of the participating modules (e.g. the shape-complementarity between signal-molecules and cell-receptors, or the key-in-lock concepts for immunological recognition). Thanks to the sustained effort of a large number of biomedical laboratories, the list of molecules for which the chemical composition and spatial structure are documented us growing continuously. Given the fact that it is still extremely difficult to predict the structure of a biomolecule on the basis of its primary structure (i.e. the string of constituent atoms), searching these databases will only be helpful if it can be done on the basis of shape. It is not difficult to make the leap from the ability to search on 3-D models, as proposed by MPEG-7. Such applications would be extremely helpful in drug-design, as one could e.g. search for biomolecules with shapes similar to a candidate-drug to get an idea of possible side effects.

• The ability to link libraries of correlated and relevant information (e.g. images, patient history, clinical findings, medication regime),
• The ability to perform on-line annotations and mark regions-of-interest with any shape or form of connectivity (from small and compact to large and diffuse),
• The ability to search images that contain similar regions-of-interest, ignoring the visual characteristics of the rest of the image,
• The ability to handle both n -dimensional data (and their time-evolution).

Requirements specific to MPEG-7 are:

• Support for n-dimensional data descriptors.
• Support for descriptors containing temporal information.
• There must exist a mechanism by which a specific segmentation or hierarchical level may be chosen, to the exclusion of all other segments.

One development that might be relevant is the Picture Archiving and Communication System (PACS) which allows physicians to annotate images with text, references and questions. This greatly facilitates interaction and consultation with colleagues and specialists.

Furthermore, there is the STARE-project (STructured Analysis of the REtina) at the University of California, San Diego, which is an information system for the storage and content-based retrieval of ocular fundus images.

<http://oni.ucsd.edu/stare>

A challenging aspect is to provide capabilities of exploiting such complex databases from on-line systems supporting the following functionalities:

• textual query,
• image query based on either whole or part of a reference image (one or several spectral bands),
• content-based retrieval,
• browsing, and
• confidentiality and data protection.

MPEG-7 should be an appropriate framework for solving such requests.

• support various description schemes,
• support for different data (e.g. multispectral, hyperspectral, and SAR) associated with various sensors (ground resolution, wavelength),
• ability to include multiple descriptions of the same documents,
• ability to link correlated information, similar region of interest, and
• ability to take into account time evolution for 2D and 3D data.

Requirements specific to MPEG-7 are:

• Support for multiple descriptors and description schemes for the same data.
• Support for descriptors for unique data types.
• Support for descriptors embodying temporal change.
• There must be a mechanism by which descriptors and description schemes within a description may be linked to other D’s and DS’s within the same description.

[to come]

This could be a new paradigm for multimedia, adding a ‘method’ layer on top of MPEG-7’s ‘representation’ layer. By making multimedia ‘aware,’ to an extent, one opens access to beginning users and increases productivity for experts. Such hidden intelligence on the part of the data itself shifts multimedia editing from direct manipulation to loose management of data.

Semi-automated multimedia editing is a broad category of applications. It can facilitate video editing for home users as well as experts in studios through varying amounts of guidance or assistance through the process. In its simpler version, assisted editing can consist of an MPEG-7-enabled browser for the selection of video shots, using a suitable shot description language. In an intermediate version, assisted editing can include planning, i.e. proposing shot selections and edit points, satisfying a scenario expressed in a sequence description language.

• Pointers as ‘handles’ that refer to the data directly, to allow manipulation of the multimedia.

Requirements specific to MPEG-7 are:

• Ability to link descriptors to the data that they describe.

Sack, W., & Don, A. (1993) Splicer: An Intelligent Video Editor (Unpublished Working Paper, MIT).

By providing direct access to short video sequences within a large database, MPEG-7 can promote the use of audio, video and film archive material in higher education in many areas:

• History: Radio, television and film provide detailed accounts of many contemporary events, useful for class-room presentations, provided that a sufficiently precise (MPEG-7) description can be queried based on dates, places, personalities, etc. (see also 4.5-Historical Speech Databases)
• Performing arts (music, theatre): Fine-grained, standardised descriptions can be used to bring a selection of relevant documents into the classroom for special classes, using on line video archives as opposed to costly local tape libraries. For instance, several productions of a theatrical scene, or musical work, can thus be consulted for comparison and illustration. Because classic and contemporary theatre are widely available in translation, this application can target worldwide audiences.
• Film Music: A tool can be developed for improving the knowledge and skills of users in the domain of film theory/practice and film music (music for film genres). Depending on the user’s background the system should provide enough material to not only improve the user’s ability in understanding the complexity of each single medium but also to handle the complex relationships between the two media film and music. To achieve this, the system should offer an environment in which the student can perform guided/supported experiments, e.g. on editing film, mixing sound, or combining both, which requires that the system can analyse and criticise the results achieved by the user.

• Linking mechanisms to synchronise between MPEG-7 descriptors and other sources of information (e.g.HTML, SGML, World Wide Web services, etc.)
• Mechanisms for allowing specialised vocabularies.
• The ability to allow real time operation in conjunction with a database.

Requirements specific to MPEG-7 are:

• Support for interoperation with description schemes.
• Support for descriptors and description schemes that allow specialised languages or vocabularies.
• There must exist a mechanism by which descriptions may link to external information.

As time elapses and the database is sufficiently large, the system, at both sides, should have the ability to support operations on the database, such as:

1. Search on the audio/video database for a specific event (synthetic or current data). Event is a sequence of audio/video data.
2. Find similar events in the past.
3. Make decisions on the current data related to the accumulated database, and/or to a-priori known data.

A related application is in security and forensics, in the matching of faces or fingerprints.

• Real time operation in conjunction with a database, and
• Domain-specific features.

Requirements specific to MPEG-7 are:

• Support for descriptors and description schemes that allow specialised languages or vocabularies.
• Support for descriptors for unique data types.

One aspect in the description of control information between (video) objects is that objects are not necessarily associated via temporal spatial relationships. Accumulation of the control and video information allows visually based functions such as redo, undo, search-by-task, or object relationship changes [7].

• Description of relationships between arbitrary object nodes in addition to spatio-temporal relationships, as in e.g. BIFS
• Allow searches based on the arbitrary (control) associations.
• MPEG-7 should operate "as fast as possible," allowing efficient interactive response times.

Requirements specific to MPEG-7 are:

• Support for descriptors and description schemes containing spatio-temporal relationships.
• Support for descriptors containing relationships between arbitrary objects.

We are still seeking references regarding ANSI guidelines for Multi-lingual Thesaurus and International radio and television typology.

A search engine could freely access any complete or partial description associated with any AV object in any set of data, perform a ranking and retrieve the data for display by using the link information. An example architecture is illustrated in fig.1.

Fig. 1. Example of a client-server architecture in a MPEG-7 based data search.