1. INTRODUCTION *

1.1. General Context *

1.2. Intended Use of this Document *

1.3 Detailed Time Schedule and Contacts *

1.4 What is Called For *

2. EVALUATION OF DESCRIPTORS *

2.1 Evaluation Criteria *

2.2 Evaluation Procedures *

2.3 Form for Proposing Descriptor(s) *

2.4 Summary of Proposal's Content for Descriptors *

3. EVALUATION OF DESCRIPTION SCHEMES *

3.1 Evaluation Criteria *

3.2 Evaluation Procedures *

3.3 Form for Proposing Description Scheme(s) *

3.4 Summary of Proposal's Content for Description Schemes *

4. EVALUATION OF DDL *

4.1 Evaluation Criteria *

4.2 Evaluation Procedures *

4.3 Form for Proposing DDL *

4.4 Summary of Proposal's Content for DDL *

5. EVALUATION OF CODING SCHEMES FOR DESCRIPTIONS *

5.1 Evaluation Criteria *

5.2 Evaluation Procedures *

5.3 Content Data *

5.4 Form for Proposing Coding Schemes for Descriptions *

5.5 Summary of Proposal's Content for Coding Schemes for Descriptions *

6. DEVELOPING THE MPEG-7 STANDARD AFTER THE CFP *

6.1 The Evaluation of the Proposals *

6.1.1. The February meeting *

6.1.2. The March meeting *

6.2 The eXperimentation Model *

6.3 The Core Experiment (CE) Process *

7. MPEG-7 Content set *

Annex A: Distribution of MPEG-7 content set *

Annex B: MPEG-7 evaluation pre-registration form *

Annex C: MPEG-7 proposal format *

Annex D: Cover Page for each proposal *

Annex E: Form to fill out for Descriptors *

Annex F: Form to fill out for Description Schemes *

Annex G: Form to fill out for Coding Schemes *

Annex H: MPEG-7 content set *

1 MPEG-7 Audio Content Set *

2 MPEG-7 Still images Content Set *

3 MPEG-7 Video Content Set *

Detailed list of items: *

Annex I: Licensing Agreement for the MPEG-7 Content Set *

MPEG-7 Audio Content Set *

MPEG-7 Still images Content Set *

MPEG-7 Video Content Set *

Annex J1: Descriptor evaluation output sheet: Feature evaluation *

Annex J2: Descriptor evaluation output sheet: Proposal evaluation *

Annex J3: Descriptor evaluation output sheet: Evaluation conclusion *

Annex K1: Description Schemes proposal evaluation sheet *

Annex K2: Description Schemes evaluation conclusions *

Annex L: DDL proposals evaluation output sheet *

General Information *

PART 1: Evaluation of the DDL proposal against the requirements *

PART 2: Ability to define other Ds, DSs, Descriptions *

PART 3: Summary of the evaluation *

Annex M1: Coding Schemes proposal evaluation sheet *

ANNEX M2: CODING SCHEMES EVALUATION CONCLUSION *

ANNEX N: UML a graphical notation for DS & D proposals. *

The scope of the future MPEG-7 standard is described in the documents "MPEG-7 Context and Objectives", "MPEG-7 Requirements", and "MPEG-7 Applications." All documents are publicly available from the MPEG web site: http://drogo.cselt.stet.it/mpeg/.

As with all MPEG standards, the MPEG-7 Standard will be drafted in a collaborative process, following a Call For Proposals (CFP), merging and improving the relevant proposals. The proposal evaluation and development of the standard which will follow is conducted by MPEG experts.

Evaluation starts with assessing the scope and technical merits of each proposal. Then, an initial Experimentation Model (XM) is built from a first selection of parts of all proposals. This starts the collaborative phase of the standardization. During the collaborative phase, the XM is updated and improved in an iterative fashion till MPEG-7 reaches the Working Draft (WD) stage. Improvements to the XM are made through Core Experiments (CEs), which are prompted by the contribution of new elements for the standard. CEs will be defined to test the contributed tools within the framework of the XM according to well-defined test conditions and criteria. The goal is to develop the best possible XM.

Finally, those parts of the XM (or of the Working Draft) that correspond to the normative elements of MPEG-7 will be standardized.

Evaluation takes place at all stages of the MPEG-7 process, in order to select appropriate items for inclusion in the standard. This document however, focuses only on specifying the test and evaluation process for responses to the CFP. , The test and evaluation process will take place at the evaluation meeting in February 1999, before the collaborative phase starts. This document also presents the time schedule for the whole process, the content of the proposals, and other guidelines important to proposers.

This document is closely related to the Proposal Package Description (PPD) document, which introduces the general context of the call. The two documents together (PPD and Evaluation document) constitute what we call the Call For Proposal (CFP). People intending to answer the MPEG-7 Call for Proposal should find all the information they need regarding the mechanics of proposal evaluation in these two documents.

In addition, proposers should also read the "MPEG-7 Context and Objectives", "MPEG-7 Requirements," "MPEG-7 Applications" documents, and other documents listed in the Call for Proposals in order to gain a better understanding of the expected use and content of MPEG-7.

In case of any remaining questions about the current document, you can directly contact its editor (see 1.3).

As stated in the introduction, this document aims to specify the evaluation process that will be applied to proposals. The evaluation itself will be mainly conducted in February 1999. However, the entirety of evaluation-related issues will be conducted in a broader time frame. For sake of clarity, the time frame of the whole process is detailed as follows:

• October 12-16, 1998: MPEG Atlantic City meeting:
• Issue formal Call for Proposals (CFP). This includes
• the Proposal Package Description (PPD), which guides each proposer by detailing all the elements he/she has to provide in order to submit a proposal, and
• the Evaluation Process document, which describes how the evaluation of the proposals will be conducted.
• Finalize selection, organization, and method of distribution for Test Materials.
• Issue Lancaster AHG notice.

• November 1, 1998: Deadline for content set request:
• Everybody willing to buy the MPEG-7 content set should send e-mail to the contact for content set distribution: Seungyup Paek (syp@ctr.columbia.edu).
• December 1, 1998: Pre-registration deadline:
• Everybody planning to submit a proposal for evaluation at the February 1999 Ad Hoc group meeting must pre-register. The deadline for pre-registration is December 1^st.
• Everybody planning to attend the Lancaster meeting should register by December 1^st.
• December 7-11, 1998: MPEG Eilat meeting:
• Finalize test and evaluation logistics for use at the February, 1999 Ad Hoc group meeting.
• February 1, 1999: Deadline for submission of MPEG-7 proposals.
• The registration is the cover page of the submitted proposal.
• February 15-19, 1999: MPEG-7 Evaluation Ad Hoc group Lancaster meeting:
• Perform tests and evaluate proposals according to the current document’s methodologies.
• Identify technologies selected to enter the collaboration phase.
• March 15-19, 1999: MPEG Seoul meeting:
• Develop the first eXperimental Model (XM 1.0).
• Define the first set of Core Experiments.

These achievements conclude the initial evaluation phase covered in this document.

All the relevant contacts for proposers are listed in the following:

(1) For any questions related to this document, you may contact the editor of this document:

(2) Regarding the MPEG-7 content set, the contact person is:

Note that the content set is described in Annex H, its distribution process is detailed in Annex A as well as in W2468, and its conditions of use are detailed in Annex I as well as in W2466.

(3) For any question about evaluation pre-registration, please contact Michael F. Vetter (See below). Note that the pre-registration form, presented in Annex B, must be filled out and sent by December 1^st to:

(4) For the registration to the Lancaster meeting, the contact persons are:

Information about the Lancaster meeting and about how to register can be found in N2465. Note that there are facilities fees included in this registration.

(5) For specific matters about the procedure for proposal submission, please contact Michael F. Vetter or Rob Koenen (See above). Each proposal must be submitted by February 1^st 1999 in electronic format, preferably Microsoft Word 6.0. Word 97 or Framemaker may be used in case Word 6.0. is inadequate for the document. An ftp site will be set up for these matters. All the needed related information will be provided after pre-registration.

In order to develop the MPEG-7 standard, MPEG calls for the following:

A. For the normative part:

• Descriptors (Ds)
• Description Schemes (DSs)
• Description Definition Language (DDL)
• Coding methods for compact representation of Descriptions
• Systems tools addressing the MPEG-7 Systems requirements specified in the MPEG-7 Requirements document.

While D, DS, DDL, and coding scheme proposals will be evaluated in February 1999 following the procedures defined in the current document, the proposals addressing MPEG-7 systems tools will not be part of the MPEG-7 Evaluation process in February 1999. These tools will be considered at the Seoul MPEG meeting. The proposers are kindly invited to present and show a demo regarding these tools at the Seoul MPEG meeting, in March 1999.

B. For the development of the standard, to be used in the XM and play an important role in the standardization process:

• Extraction methods
• Search Methods
• Evaluation and validation techniques

The proposals from group B are needed for the design and improvement of the XM. They will be discussed during the Lancaster meeting, and in the MPEG-7 Evaluation Ad Hoc Group(s) between Lancaster and Seoul. The conclusion of these discussions will be submitted as input to Seoul for the design of XM1.0. Whether and how these techniques will be formally evaluated will be decided after the XM1.0 is built.

It should be noted that it is not necessary for a proposal to address all of the elements listed above. It is possible for a proposal, for example, to only propose descriptors for some set of features, or description schemes or parts of description schemes. Selected aspects of different proposals will be incorporated into a common model (the XM) during the collaborative phase of the standard with the goal of building the best possible model.

The proposer should provide:

1. A proposal document according to the format in Annex D. It should highlight all information that will help MPEG in identifying the strengths of the proposal.
2. The proposer is strongly encouraged to give a presentation/demonstration of the proposal using relevant MPEG-7 test sets at the evaluation meeting in Lancaster. If the MPEG-7 test set is insufficient for the demonstration, another data set may be used. In this case the data set used must be made available to MPEG on the same conditions as the MPEG-7 Test and Evaluation Material (see annex I).

The feature captures important characteristic(s) of the AV material.

• Effectiveness

Gives better retrieval accuracy (e.g. precision, recall rate) with respect to other descriptors for the same feature.

• Application domain

The Descriptor is applicable to a wide range of application domains.

• Expression efficiency

The Descriptor expresses the given feature(s) precisely, and completely.

• Processing efficiency
• An efficient Descriptor value calculation method exists.
• An efficient matching method (allowing rank ordering) associated with this Descriptor exists.
• Scalability
• For a given application, the performance does not degrade with larger amount of data
• Multi-level representation

The Descriptor represents the features at multiple levels of abstraction.

The Descriptors proposals will be evaluated by groups of experts drawn from participants at the evaluation meeting, following the evaluation procedure described as follows:

Proposals will be categorized according to the feature they represent.

Groups of MPEG experts will be formed to evaluate proposals by categories.

The evaluation will be based on the following steps:

1. Evaluate feature relevance

Goal: Understand the feature, and assess how it captures important characteristic(s) of the AV material.

How: paper evaluation, from all paper descriptions of all proposals addressing this feature.

Who: experts

Output: feature evaluation sheet (see Annex J1), which includes represented features and their importance, with associated explanations.

 

Then, for each proposal independently within a given category:

 

2. Evaluate the paper document.

Goal: The goal of this step is to have an initial assessment of the proposal based on the documentation included (questionnaire, summary and detailed structured description).

During this step, experts should prepare eventual questions to ask to the proposers at the next step, to clarify some points if needed.

How: Experts will review/analyze this information against requirements and evaluation criteria, first without being influenced by the participation of the proposers. A limited duration should be given to this step.

Who: Experts

Output: the proposal evaluation sheet (Annex J2, proposal summary and criteria table: all relevant criteria), with a short summary of the proposal, and its first evaluation along the analyzed criteria.

3. Hear presentation / see demonstration.

Goal: To enhance the understanding of the proposal by the experts.

The presentation shall demonstrate the appropriateness of the solution, and disclose the appropriate range of use. The demonstration will provide evidence of (some of) the functionalities claimed.

Who: experts and proposer(s) whose submission is evaluated.

How: Experts will interact with the proposer(s) through a presentation and possibly a demonstration.

Both demonstration and presentation will each have a 10 minutes time limit.

Output: Updated (modified/completed) proposal evaluation sheet (criteria table and eventually summary of the proposal).

Remarks:

• This step could present some new elements regarding feature relevance.
• This step is not mandatory, and a proposal without any demonstration will not be penalized during evaluation. However, it is clear that this step could only benefit the proposal, as it gives an opportunity for the proposers to clarify, and provide evidence of features that are not easy to demonstrate on paper.

1. Evaluate results of similarity based retrieval

Goal: To refine the evaluation of the effectiveness and expression efficiency of the Descriptor.

Who: Experts

How: Proposers will provide

• Descriptor values for selected and relevant elements (e.g. image, shot) in the content set, following the items labeling provided by MPEG.
• A system/program for similarity based retrieval, which should be executable on a machine brought to the meeting by the proposer.

The quality of similarity based retrieval using the proposed descriptor will be evaluated using the MPEG-7 test sets sample inputs provided on-site, or samples provided by the proposer. The test will be run as follows:

1. An input item will be selected from the database elements (the input item will be selected by experts so that it is appropriate for the described feature).
2. The retrieval program will be run to identify and rank the n database items closest to the input. (Note that the speed of the retrieval program implementation will not be used as criterion at this stage of the evaluation).
3. The ranking provided by the retrieval program will be judged by experts following their own perception of similarity and dissimilarity based on the feature.

This process will be repeated on several items of the test set, until the evaluators reach a consensus.

• If the proposer is unable to attend the Lancaster meeting, he/she should arrange for a representative to act on his/her behalf.
• It is the responsibility of the proposer to choose the elements of the content set on which he/she provides Descriptors values. These elements should be relevant to the evaluated Descriptor (e.g. color images for a color Descriptor), and in sufficient number to allow evaluators to draw conclusions from this step. Proposers should justify their selection among all the content set elements. The evaluators will judge the rational of this selection, and draw conclusions from this step taking it into account.

Output: Updated proposal evaluation sheet (criteria table, effectiveness and expression efficiency criteria only - similarity measure speed should not be taken into account).

Goal: To summarize the results of the previous steps to allow the selection of technologies for inclusion in the XM1.0, or in Core Experiments.

Who: Experts.

How: Experts arrive at a consensus.

Output: Evaluation summary of the proposal evaluation sheet (Annex J2) and evaluation conclusions sheet (see Annex J3), the final decision on this being taken as the Seoul meeting.

When proposing a Descriptor, each proposer has to fill the form enclosed in Annex E. Besides, proposers have to fill the proposal cover page (Annex D), and then refer to the proposal format guidelines given in Annex C.

The paper proposal, to be sent by February 1^st, 1999, must include:

• Cover page: Annex D
• Paper description, following the format described in Annex C
• Specific Descriptor form: Annex E

At the Lancaster meeting, proposers provide:

• Presentation of their proposal (not mandatory but strongly encouraged, see 2.2 step 3)
• A demonstration of their proposal (not mandatory but strongly encouraged, see 2.2 step 3)
• Elements for similarity based retrieval (not mandatory, if applicable, see 2.2 step 4)

• Effectiveness of the DS in accomplishing its stated purpose.
• Application domain

The DS is applicable for a wide range of applications. "Applicable" means directly usable or usable as a component of a larger DS.

• Comprehensiveness

The DS provides an off the shelf solution for a given application domain. For this application domain, it takes into account relevant Descriptors and relevant relations between the Descriptors.

• Abstraction at Multiple Hierarchical Levels

The DS can provide abstractions at multiple levels. An example is a hierarchical scheme where the base layer gives a coarse description and successive layers give more refined descriptions. The type of hierarchy used is appropriate for the purpose of the DS. Descriptors within the DS are amenable to being prioritized.

• Flexibility

Part of the DS can be used effectively:

• Ability to instantiate a part of a DS.
• Ability to efficiently access a part of a DS.
• Ability to accept additional Descriptors; existing Descriptors can be replaced with new Descriptors.
• Extensibility

The DS is easily extensible to other DSs (in a way similar to inheritance in Object-Oriented Programming).

• Scalability
• For a given application, the performance does not degrade with larger amount of data.
• Scalability across different applications (down or up).
• Simplicity

A minimal number of Descriptors and possible relationships are used to meet the needs of a particular application domain.

The evaluation procedure for Description Schemes is as follows:

The proposals are categorized according to the media addressed, applications addressed, functionality etc. Groups of MPEG experts will be formed to evaluate proposals by categories.

This phase relies on:

• Evaluation of proposals’ documents;
• Presentation and demonstration of evidence.

Both are essential to the complete evaluation of DS proposals. They are handled as described in the following.

The evaluation will be based on the following steps:

1) Evaluate the paper document.

Goal: The goal of this step is to have an initial assessment of the proposal based on the documentation included (questionnaire, summary and detailed structured description).

During this step, experts should prepare eventual questions to ask to the proposers at the next step, to clarify some points if needed.

How: Experts will review/analyze this information against requirements and evaluation criteria, first without being influenced by the participation of the proposers. A limited duration should be given to this step.

Who: Experts

Output: the proposal evaluation sheet (Annex K1, proposal summary and criteria table: all relevant criteria), with a short summary of the proposal, and its first evaluation along the analyzed criteria.

Goal: To enhance the understanding of the proposal by the experts.

The presentation shall demonstrate the appropriateness of the solution, and disclose the appropriate range of use. The demonstration will provide evidence of (some of) the functionalities claimed, and of how the DS satisfies the evaluation criteria.

Who: experts and proposer(s) whose submission is evaluated.

How: Experts will interact with the proposer(s) through a presentation and possibly a demonstration.

Both demonstration and presentation will each have a 10 minutes time limit.

Output: Updated (modified/completed) proposal evaluation sheet (criteria table and eventually summary of the proposal).

Remark: This step is not mandatory, but strongly encouraged. It gives an opportunity for the proposers to clarify, and provide additional evidence of facts that are not easy to demonstrate on paper.

Goal: To summarize the results of the previous steps. This should allow:

• To identify the strong points of the DS,
• To identify how the DS might be adapted or combined with other DSs to enter the XM1.0, and/or be tested through Core Experiments.

Who: Experts.

How: Experts arrive at a consensus.

Output: Evaluation summary of the proposal evaluation sheet (Annex K1) and evaluation conclusions sheet (see Annex K2), the final decision on this being taken as the Seoul meeting.

When proposing a Description Scheme, each proposer has to fill the form enclosed in Annex F. Besides, proposers have to fill the proposal cover page (Annex D), and then refer to the proposal format guidelines given in Annex C.

The paper proposal, to be sent by February 1^st, 1999, must include:

• Cover page: Annex D
• Paper description, following the format described in Annex C
• Specific Descriptor form: Annex F

At the Lancaster meeting, proposers provide:

• Presentation of their proposal (not mandatory but strongly encouraged, see 3.2 step 2)
• A demonstration of their proposal (not mandatory but strongly encouraged, see 3.2 step 2)

1. Compositional capabilities: The DDL shall supply the ability to compose a DS from multiple DSs.
2. Platform independence: The DDL shall be platform and application independent. This is required to make the representation of content as reusable as possible even on grounds of changing technology.
3. Grammar: The DDL shall follow a grammar that is unambiguous, and allow easy parsing (interpretation) by computers.
4. Primitive data types: provide a set of primitive data types, e.g. text, integer, real, date, time/time index, version, etc.
5. Composite datatypes: The DDL must be able to succinctly describe composite datatypes that may arise from the processing of digital signals (e.g., histograms, graphs, and rgb-values).
6. Multiple media types: The DDL must provide a mechanism to relate Ds to data of multiple media types of inherent structure, particularly audio, video, audio-visual presentations, the interface to textual description, and any combinations of these.
7. Partial instantiation: The DDL shall provide the capability to allow a DS to be partially instantiated by descriptors.
8. Mandatory instantiation: The DDL shall provide the capability to allow the mandatory instantiation of descriptors in a DS.
9. Unique identification: The DDL shall provide mechanisms to uniquely identify DSs and Ds so that they can be referred to unambiguously.
10. Distinct name spaces: The DDL shall provide support for distinct name-spaces. Note: Different domains use the same descriptor for different features or different purposes.
11. Transformational capabilities: The DDL shall allow the reuse, extension and inheritance of existing Ds and DSs.
12. Relationships within a DS and between DSs: The DDL provides the capability to express the following relationships between DSs and among elements of a DS and express the semantics of these relations

a) Spatial relations

b) Temporal relations

c) Structural relations

d) Conceptual relations

13. Relationship between description and data: The DDL shall supply a rich model for links and/or references between one or several descriptions and the described data.
14. Intellectual Property Management: The DDL shall provide a mechanism for the expression of Intellectual Property Management and Protection (IPMP) for description schemes and descriptors.
15. Real time support: The DDL shall desirably provide features to support real time applications (database output like electronic program guides)

The evaluation procedure for each Description Definition Language is as follows:

1) Evaluate the paper document.

Goal: The goal of this step is to have an assessment of the ability of the proposal to meet the requirements for the DDL described in the MPEG-7 requirements document (summarized above in section 4.1). During this step, experts should prepare eventual questions to ask to the proposers at the next step, to clarify some points if needed.

How: Experts will review/analyze this information against requirements and evaluation criteria, first without being influenced by the participation of the proposers. A limited duration should be given to this step.

Who: Experts.

Output: A copy of the form in Annex L part 1 will be completed, assessing how the DDL meets the requirements. .

2) Define a test set of Ds and DSs

Goal: The goal of this step is to select a set of DSs and Ds to be used in the following step of the evaluation.

How: A set of DSs and Ds will be identified from the DDL proposals by reviewing the proposal packages. The experts will ensure that the chosen DSs and Ds are of manageable complexity.

Who: Experts.

Output: A set of DSs and Ds to be used in the next step.

Goal: To enhance the understanding of the proposal by the experts.

The presentation shall demonstrate the appropriateness of the solution, and the ability to use their DDL beyond the provided examples.

How: Experts will interact with the proposer(s) through a presentation, which will have 10 minutes time limit.

The evaluators will ask the proposer to show that the DDL can be used to generate DSs and Ds and related descriptions contained in the test set, when different from the examples of the proposal.

Who: Experts and proposers

Output: The output will be the completion of the Annex L Part 2, which is designed to provide an assessment of the ability of each DDL to generate DSs, Ds and descriptions.

Goal: To summarize the results of the previous steps to allow the selection of DDLs or parts of DDLs for inclusion in the XM1.0, or in Core Experiments.

How: Experts arrive at a consensus.

Who: Experts.

Output: Evaluation summary of the proposal evaluation sheet (Annex L Parts 1 and 2) and evaluation conclusions sheet (Annex L part 3).

Remark: The evaluation procedure for DDL will formally end at the 48th Vancouver meeting. The Lancaster meeting will provide a summary of each DDL’s capability, and an assessment of its suitability for the XM DDL and for participating in the Core Experiments. It will be further discussed on the reflector between Lancaster and Seoul. The conclusion of these discussions will be submitted as input to Seoul. Then, it is recommended that a Core Experiment be conducted until the Vancouver meeting, by utilizing the recommended DDL’s to construct the XM1.0 DSs and Ds. The result of this Core Experiment will conclude the DDL evaluation phase: the collaborative phase starts.

No specific form has to be filled when submitting a DDL. Proposers only have to fill the proposal cover page (Annex D), and then refer to the proposal format guidelines given in Annex C.

The paper proposal, to be sent by February 1^st, 1999, must include:

• Cover page: Annex D
• Paper description, following the format described in Annex C

At the Lancaster meeting, proposers provide:

• Presentation of their proposal (not mandatory but strongly encouraged, see 4.2 step 3)

• Compression efficiency
• Complexity of the description encoding and decoding processes.
• Lossless compression

Ability to losslessly compress descriptions instantiations.

• Streaming capability

It is possible to multiplex and stream the coded description.

• Error resilience

The coded description shall be robust against transmission errors.

Ability to be applied to a wide range of descriptions

The main evaluation of proposals will be done based on the documentation included with the proposal (questionnaire, summary and detailed structured description). MPEG experts will review/analyze this information against requirements and evaluation criteria to identify promising techniques for the XM development. If the proposer has performed tests for bitstream verification of his/her coding scheme, either in combination with a D/DS/DDL or separate, a demo is highly encouraged. Furthermore, a short presentation should be given to help the MPEG experts better understand the proposed coding scheme and its possible extensibility and applicability to a wider range of descriptions.

If bitstream verification is performed as part of the proposal, it should be based on encoding descriptions extracted from MPEG-7 content set in the addressed categories (see section 7). If other content is used, it should be justified why it was not possible to use the official material.

When proposing a Coding Scheme, each proposer has to fill the form enclosed in Annex G. Besides, proposers have to fill the proposal cover page (Annex D), and then refer to the proposal format guidelines given in Annex C.

The paper proposal, to be sent by February 1^st, 1999, must include:

• Cover page: Annex D
• Paper description, following the format described in Annex C
• Specific Descriptor form: Annex G

At the Lancaster meeting, proposers provide:

• Presentation of their proposal (not mandatory but strongly encouraged, see 5.2)
• A demonstration of their proposal (not mandatory, see 5.2)

The main objective of the Call For Proposals it to lead to the best possible starting of the collaborative phase, which is recognized as one of the strengths of MPEG.

It is reasonable to expect that the competitive process, which culminates at the evaluation meeting, will produce several top performing, possibly competitor, elements. Each of these will have strengths compared to each other, and the goal of the ensuing collaborative process is to produce convergence to a single standard combining the best features of all proposals. During this process, opportunities for further improvements will arise, and these should be verified and incorporated. The method to accomplish this work is called the Core Experiment process and the basis on which it works is the eXperimentation Model (XM).

For each proposal to be evaluated, all information shall be provided following the guidelines of the CFP, by February 1st 1999, so that MPEG members can study them to prepare the evaluation meeting in advance. The evaluation meeting will be held in Lancaster University (UK), February 15-19. During this meeting, MPEG members will:

• Categorize submissions to perform the evaluation on related proposals,
• Evaluate the proposals with help of paper submissions, optional demonstrations, applicable tests on MPEG-7 test sets.
• Select some technologies to enter the collaborative phase.

Proposers will be given the opportunity to present and clarify their proposal.

The quality and the clarity of the proposal are the responsibility of the proposers.

Finally, proposers are encouraged to be present at the February meeting to help with the evaluation.

The results of the February meeting, together with the work done on the eXperimentation Model structure and started in October 1998, form the basis to build the first version of the eXperimentation Model. Its first version will be finalized in March 1999, at the MPEG Seoul meeting. This meeting will also specify the first set of Core Experiments to be performed.

The XM is a "common experimentation framework" which is used for further development of the standard. It will have components for evaluating and improving the DDL, DSs and Ds. Moreover, its associated experimentation database(s) will be more complete than the test sets used for evaluation. An XM may have different implementations within the MPEG group.

As a result of both the collaborative work on XM structure started at the Dublin meeting in October 1998 and the evaluation phase, an XM will be built and its first version will be finalized at the MPEG Seoul meeting, March 15-19, 1999. XM will consequently include proposals and/or parts of proposals that were either made before CFP within the Ad Hoc group on XM definition, or given as answers to the CFP.

After the XM is established, new elements can be brought to MPEG-7 following a Core Experiment procedure. They will be then evaluated inside the XM so that in the final standard, if two elements accomplish similar thing(s) in similar conditions, only the best will be chosen.

The Core Experiments (CEs) will provide a more complete evaluation of Ds, DSs, DDLs, combinations of Ds, DSs handling combinations of Ds, etc within XM.

The heart of this process is for multiple, independent, directly comparable experiments to be performed to determine whether or not a proposed algorithmic technique or syntax element has merit. A Core Experiment has to be completely and uniquely defined, so the results are unambiguous. In addition to the specification of the element to be evaluated (syntax, related algorithmic techniques…), a Core Experiment also specifies the parameters to be used so that the results can be compared. Therefore, a Core Experiment may include one or more of the following elements: benchmark databases, associated 'truth', benchmark queries, objective and subjective evaluation.

A Core Experiment is proposed by one or more MPEG experts. It is accepted by consensus, providing that two or more independent experts agree to perform the experiment. Conventionally this is interpreted to mean that the independent experts work for different companies (different divisions or subsidiaries of a company are not considered independent).

Normally, a Core Experiment will be conducted between two successive MPEG meetings. If necessary, and again by consensus, a Core Experiment may be continued until a further MPEG meeting.

The outcome of a Core Experiment is determined by consensus. If the experiment can be evaluated subjectively, then an informal "test" is performed by presenting the results in the MPEG meeting. First, the independent results should evidently be equivalent. Second, they should indicate whether or not the technique has merit. Since this will at times be difficult to determine because there is some merit but it is marginal, the consensus may be that an evaluation of implementation complexity will be used to make the final decision.

In case the technology presented in a Core Experiment is selected for entering the XM, this can only become effective if some MPEG member agrees to donate the source code that fully implements this technology.

MPEG-7 is providing a content set, made of still images, audio, and audio-visual data, to be used for evaluation. This content set is described in Annex H. Its distribution process is detailed in Annex A as well as in W2468, and its conditions of use are detailed in Annex I as well as in W2466.

Each proposer planning either to show a demonstration or to participate to the similarity based retrieval evaluation shall use appropriate subsets of this content set. It is the responsibility of the proposer to choose the items of the content set that he/she will use for demonstration or similarity-based retrieval. These elements should be relevant to the evaluated proposal (e.g. color images for a color Descriptor), and in sufficient number to allow evaluators to draw conclusions from what is shown. Moreover, note that the choice of subset must be justified in the proposal/presentation.

In case the MPEG-7 content set is insufficient for demonstrating his/her proposal, the proposer may use his/her own test set and provide justifications for the same. In this case the content set used must be made available to MPEG under the conditions mentioned in Annex I.

The evaluators will judge the rationale of the selection of the content set, and take it into account when drawing conclusions from any evaluation step using it.

The following is the procedure to obtain the MPEG-7 content set:

1. Each group (company, university, individual etc.) requesting MPEG-7 content must send e-mail to the primary contact (See below) for content set distribution by November 1^st, 1998. The content set consists of all image, video and audio content that were reviewed for MPEG-7 evaluation. It will not be possible to obtain a subset of the content. For example, it will not be possible to only obtain the images, without the video or audio material. The MPEG-7 content set will require approximately 20 CDs. Each group must provide the following information. No requests will be honored if any of the following information is missing:

• Name of a contact person and his/her e-mail address and phone number.
• Complete mailing address.
• Agreement to pay the full amount required for the test material, as described below.

1. Once all the requests have been received by the primary contact by November 1^st, 1998, the primary contact will calculate the charge required by group n as follows:

(C/N) + (Shipping charge for group n)

C is the total cost required for the duplication of the content set.

N is the total number of groups that have requested the content set by the November 1^st deadline.

Once the charges for each group have been calculated, the primary contact will send an e-mail to all the groups that have requested the content set, and notify each group of their respective payment amounts. This will be done by November 3rd, 1998. At this time, An approximation of the cost for each group is as follows:

Approximate storage requirement of the content set: 20 CDs;

Approximate cost to produce a minimum of 250 copies of the content set: $15,640;

Approximate cost for each group, assuming that 100 groups request the content set: $160.

These approximations do not include shipping charges or the tax. The cost is subject to change.

 

2. Once each group receives the payment amount, they must do all of the following:

• Prepare a check made out to Conversion Media, with the correct amount. Conversion Media is the company that will be duplicating and shipping all the content sets. It is important to indicate ‘MPEG-7 Content set’ on the check.
• Send the check to Conversion Media by November 8^th, 1998. The address of Conversion Media is as follows:

• Send e-mail to the primary contact to confirm that payment has been sent.

1. The material will be shipped by November 25^th, 1998.

 

2. Groups that did not request the content set by the November 1^st, 1998 deadline will be able to obtain the content set by contacting members of MPEG-7 that receive the content set. However, the content set can only be made available subject to the terms and conditions specified in the MPEG-7 content distribution agreement (See N2466 or Annex I of N2463).

The primary contact for the content set distribution process described in this document is:

Seungyup Paek

Columbia University New Media Technology Center - USA

syp@ctr.columbia.edu.

All organizations intending to submit proposals to the MPEG-7 evaluation and development process are requested to complete this pre-registration form and submit it to

by the 1^st December 1998.

Pre-registration of proposals is mandatory. For more information, please see the MPEG-7 Evaluation Document.

A pre-registration form must be submitted for each descriptor, description scheme, Description Definition Language (DDL), or coding scheme for descriptors submitted for evaluation. The same applies for the systems tools as well as for the non-normative tools which are useful for the development of the standard, such as extraction and search methods as well as evaluation and validation techniques.

E-Mail:____________________________________________________________________________________

If this pre-registration refers to a normative tool, please circle which item it refers to:

1. For each proposal, proposers should provide a detailed paper describing the proposed technology, with emphasis on the components of the proposal that should be evaluated. The paper should:

• Explain which MPEG-7 requirements in the Requirements document the proposal addresses and how it satisfies them.
• Explain which of the MPEG-7 evaluation criteria given in the current document the proposal meets how it meets them.

2. For Descriptors, please provide the data needed to perform the similarity evaluation, if applicable, as stated in the current document. If the MPEG-7 test set is augmented with other material or a different test set is used, please state why the MPEG-7 test sets were not suitable. Also the data set used must be made available to MPEG on the same conditions as the MPEG-7 Test and Evaluation Material. (See Annex I).

Please give also:

2.1. Typical search / filtering tasks the proposal applies to.

2.2. Explanation of the similarity measure used, if any.

3. For Description Schemes the proposal must include a detailed explanation using English (and pseudo-code if required) and a graphical representation. It is strongly suggested that, to the extent possible, the graphical representation of the proposed description schemes follow the UML notations provided in Annex N. It is strongly encouraged that a presentation and demonstration be given to support the material contained in the written proposal. Provide a written description of any demonstration planned to accompany this proposal and detail the evaluation criteria that will be shown. If your DS demonstration depends on use of Descriptors, please provide a brief summary of these Descriptors (Descriptors proposed separately may simply be referenced by proposal ID). All demonstrations of DSs shall use the MPEG-7 Test Material (or subset consisting of all relevant material) unless the proposer can justify that other test material is necessary to demonstrate the DS.

4. For DDL, the submission must also include:

• A description of the application domain and media types used in the proposal;
• Examples of how the proposal fulfils the requirements for the chosen application domain and media type: Description Schemes, Descriptors built with the DDL. (It is desirable that a set of Ds and DSs for another application domain and for another media is provided).
• A definition of the proposed DDL grammar, its syntax and semantics together with a BNF (Bakaus Nauer Form) representation.

Note that for DDL, the description of how the proposal meets the MPEG-7 requirements should be made by completing a copy of annex L part 1.

Then, fill out the cover page (given in annex D) and the relevant form(s), given in annexes E, F, G and H, for each element (Descriptor, Description Scheme, Description Definition Language, Coding Scheme) in the proposal that MPEG should evaluate.

Note that a separate form must be completed for each Description Scheme and for each of the associated Descriptors if it is desired to have them separately considered/evaluated by MPEG.

Please note that brevity and clarity helps the evaluation.

Proposal Id: _____________

(Number obtained from pre-registration)

Company/Institution:_____ ______________________________________

2. Do you have other proposal(s) related to this one? Please list the proposal Ids.

9. Will you run a (not more than 10 minutes) demonstration of your proposed solution at the evaluation Ad Hoc meeting in February? (yes or no)
10. If the demonstration will be done using one or more MPEG-7 Test Sets, please Indicate them.
11. Please indicate the MPEG-7 test sets for which you will be providing the descriptor value and system/program for similarity retrieval.
12. Does the descriptor apply to global as well as local features? (For example, does it apply to a whole image as well as objects in the image?)

2. What is the main functionality of your DS?

1. Which part of the description does your coding scheme apply to?

(a) DDL	(b) Description scheme(s)	(c) Descriptor(s)
(d) Descriptor data	(e) Others (please specify) ______________

 

2. Is your coding scheme media specific?

If yes, which media has your coding scheme been developed for and/or applied to?

2. Is this applicable to other domains? Do you have evidence of this?

3. State up to three MPEG-7 applications (for examples you may refer MPEG-7 Applications Document) to which your proposal applies.

 

4. Was your coding scheme developed for a specific D, DS, DDL? (yes or no)

The MPEG-7 Content set is composed of three main categories: Audio (about 12 hours), Still images (about 7000 images) and Video (about 13 hours). The items in each category are described in the following sections.

For supplemental content, please refer to the Video Content Set, especially the music and entertainment category. It contains much Audio-visual material, Also of potential interest are items with monologues and dialogues in diverse languages.

3. MPEG-7 Video Content Set

All items have been assigned to one of the three possible types: "Program", "Sequence" or "Shot". A "Program" is a piece of material of rather long duration which most of the time includes both the beginning and the end of the program. A "sequence" is a clip of a few minutes that has been extracted from a program. Finally, the "shot" type corresponds to either "program" or "sequences" where the list of editing effects (shot cut, fade, etc.) will be available. Finally, note that all video material is encoded in MPEG-1 format, except item V30.

This document establishes the conditions to abide in order to use the MPEG-7 content set.

The files here included are part of the MPEG-7 content set. When using or accessing these files, users agree with the following conditions:

1. The use of MPEG-7 content set is authorized only for the purpose of the development of the MPEG-7 standard, notably for the evaluation, collaboration and verification phases.
2. MPEG-7 content cannot be used for any commercial purposes, notably any type of broadcasting. Moreover the users are not allowed to make this content generally and freely available by any means, notably through the Internet, since this may contradict clause 1.
3. All intellectual property rights for the MPEG-7 content set remain the property of the respective owners.
4. The content owners are under no obligation or liability in respect of the fitness of the content for a particular purpose. The owners do not warrant that the sequences will meet user’s specific requirements.
5. Copies of this content can only be made by MPEG members for the purposes mentioned in 1. and must always contain the usage conditions here defined by including a "Usage conditions" file present in the same directory of the content files.
6. MPEG is allowed to make public the results (not the content itself) of tests and experiments made using the MPEG-7 content set.
7. MPEG members may include only limited parts of the MPEG-7 content set in scientific communications (e.g. conferences, workshops) and publications, provided that no commercial purposes exist. Whenever this is done, the corresponding content owner must be explicitly acknowledged.
8. Prior to any usage of MPEG-7 content, any person that disagrees with the present conditions, must not make any use of the MPEG-7 content and return the received support (CD-ROM, tape or disk) to its supplier. Moreover any copies that may have been made must be destroyed. In the event such a person has obtained the MPEG-7 content set or any of its parts through Internet access, such material shall be destroyed, and the provider of such material informed
9. The MPEG-7 content has been made available for MPEG usage under the conditions above specified by the following organizations/individuals:

Feature description: (few lines explanations)

Feature’s relevance to MPEG-7 Requirements: (few lines summary)

Global appreciation on relevance:

Summary of the proposal: (a few lines)

Evaluation facts should mention:

• Not applicable/ applicable (for instance multi-level representation may not be addressed by the current proposal)
• What supported these facts: paper/demo/test…
• The summary of the facts themselves, for instance: very good in such and such ways, weaknesses in this area… showed with evidence or just estimated but would need to be confirmed… not understood at all…

Conclusion should mention:

• If possible an estimate of how easy/difficult this would be to improve/add (e.g.: multi-level representation is not possible, or processing efficiency should be easy to reduce…)
• How sure the experts are (evidence shown, no evidence shown but very likely, very hard to tell…)
• global evaluation (Not Applicable/ --/ - / + / ++)

• Main strong points, qualitatively: (2-3 lines summary)

• Main weak points, qualitatively: (2-3 lines summary)

• Overall evaluation: (0/1/2/3/4/5)

Eventual remarks: (points of importance, not covered above)

Number of proposals

Feature: name, short description and relevance (0-5)

List of proposals with overall assessment (0-5), from 5 to 0.

Summary of the proposal: (a few lines)

Evaluation facts should mention:

• Not applicable/Applicable (for instance multi-level abstraction may not be addressed by the current proposal)
• What supported these facts: paper/presentation/demo/ …
• The summary of the facts themselves, for instance: very good in some ways, weaknesses in other areas… showed with evidence or just estimated but would need to be confirmed… not understood at all…

Conclusions should mention:

• If possible an estimate of how easy/difficult this would be to improve/add (e.g.: multi-level abstraction is not possible, or processing efficiency should be easy to reduce…)
• How sure the experts are (evidence shown, no evidence shown but very likely, very hard to tell…)
• Global evaluation (Not Applicable/ --/ - / + / ++)

• Main strong points, qualitatively: (2-3 lines summary)

• Main weak points, qualitatively: (2-3 lines summary)
• Overall evaluation: (0/1/2/3/4/5)

Eventual remarks: (points of importance, not covered above)

Number of proposals

List of proposals with global appreciation’s (0-5), from 5 to 0.

This proposal will be used in the construction of the DDL for the Experimental Model XM:

YES, FULL PROPOSAL / YES, PARTS OF THE PROPOSAL / NONE

This proposal will be chosen to participate in Core Experiments CE: YES/NO

This proposal is not recommended for either the CE or XM: YES/NO

Summary of the proposal: (a few lines)

Evaluation facts should mention:

• Not applicable/Applicable (for instance lossless compression may not be addressed by the current proposal)
• What supported these facts: paper/presentation/demo/ …
• The summary of the facts themselves, for instance: very good in some ways, weaknesses in other areas… showed with evidence or just estimated but would need to be confirmed… not understood at all…

Conclusions should mention:

• If possible an estimate of how easy/difficult this would be to improve/add (e.g.: multi-level abstraction is not possible, or processing efficiency should be easy to reduce…)
• How sure the experts are (evidence shown, no evidence shown but very likely, very hard to tell…)
• Global evaluation (Not Applicable/ --/ - / + / ++)

• Main strong points, qualitatively: (2-3 lines summary)

• Main weak points, qualitatively: (2-3 lines summary)

• Overall evaluation: (0/1/2/3/4/5)

Eventual remarks: (points of importance, not covered above)

Number of proposals

List of proposals with global appreciation’s (0-5), from 5 to 0.

Universal Modelling Language (UML) [Rational1] was developed by Rational Systems Inc. to address the diversity of approaches to modelling object orientated systems. It combines features of several modelling notations that are described in more detail in the following this reference [UML1]. UML was originally intended to become a de-facto standard but has since been adopted by OMG. It has been used in recent CORBA specifications [OMG1] This document is intended to provide a brief overview of UML concepts, provide sufficient graphical notation to allow UML to be used as the graphical notation for responses to the MPEG-7 call for proposals. It’s use is to be restricted to providing a graphical notation for description schemes (DSs) and descriptors (Ds). This document is not intended to be a complete introduction to UML nor is it intended to prevent the use of UML symbols not introduced in this document. A complete introduction to UML can be found either in [UML 1] or [UML 2]. A more extensive introduction to UML can be found in [UML 3]

UML is a software development tool which builds on the substantial body of work in the software-engineering field addressing the design, development and management of large software projects [SOM 1] and as such provides several components, these are

UML provides several views on a system a brief description of each is given below.

UML provides several diagrams each with its own set diagrammatic conventions to allow a particular view of a system to be described these are:

[UML 2] UML Notation Guide version 1.1 September 1997 http://www.rational.com/uml