The Multimedia Evaluation Benchmark, MediaEval, offers challenges in artificial intelligence related to data that includes multiple modalities (e.g., audio, visual, textual, and/or contextual). The goal of MediaEval is to develop and evaluate new algorithms and technologies for analyzing, exploring and accessing information in multimedia data. MediaEval pursues a “Quest for Insight”: we push beyond improving evaluation scores to achieving deeper understanding about the challenges, including data and the strengths and weaknesses of particular types of approaches. Our larger aim is to promote reproducible research that makes multimedia a positive force for society. MediaEval is now calling for proposals for tasks to run in the 2025 benchmarking season.

  • Call for Task proposals (first deadline): Wed. 11 December
  • Call for Task proposals (final deadline): Wed. 22 January

The proposal should describe the motivation of the task, including a description of the use scenario in which the results of the tasks would be used (e.g., application that serves users). It should provide a definition of the specific problem that task participants are required to solve. Also, it should include information on the data (including source and licensing), and on how the solutions developed by task participants will be evaluated (the metric and description of how the metric is related to the use scenario). We ask you to think carefully about specific research questions that are related to the challenge, and mention these in the proposal. These research questions will guide participants in pursuing the “Quest for Insight”, i.e., going beyond thinking only about evaluation scores. Finally, the proposal must also include a statement of how the task is related to MediaEval (i.e., the human or social component), and how it extends the state of the art.

Indication of Intent

If you plan to submit a task proposal, we strongly suggest that you submit, by email, an “Indication of Intent” in the form of a short task summary (a blurb of 50-100 words) as soon as possible. The description should include a clear statement of what participants are expected to do, which data is used, and how participant submissions are evaluated. The summary should finish with a statement of the motivation for the task.

Full Task Proposal

A task proposal contains the following elements. Note that there is no specified length for the proposal, but in general proposals do not exceed three pages.

Part I: Task Description

This is a version of your task description that will be posted to the MediaEval website. Its goal is to inform and attract the interest of potential participants. It consists of the following parts:

  • Task title: Give your task an informative title.
  • Task description: State the goal of the task and what is expected of task participants in a simple easy-to-understand manner. The task description should make clear what the task requires of participants.
  • Motivation and background: Describe the motivating use scenario, i.e., how would the results of the task be used in an application that serves users. Also, state how the task extends the state of the art.
  • Target group: Describe the type of researchers who would be interested in participating in the task.
  • Data: Describe the data, including how the data will be collected and licensed.
  • Evaluation methodology: Describe the evaluation methodology, including how the ground truth will be created and the evaluation metrics.
  • “Quest for Insight”: List several research questions related to the challenge, which the participants can strive to answer in order to go beyond just looking at evaluation metrics.
  • References and recommended reading: list 3-4 references related to the task that teams should have read before attempting the task.
  • List of task organizers. (Designate a lead task organizer whose contact details will appear on the website for the task.)
Part II: Big Picture of the Task

Please address each of the following points with 2-3 sentences each:

Innovation: MediaEval strives to offer innovative tasks. New tasks open up new terrain for multimedia researchers, continuing tasks introduce novel aspects every year that drive forward the state of the art. Describe the novel contribution of your task. Focus: MediaEval focuses on tasks that have a human or social aspect. This means that they serve groups of users, work with multimedia content produced by users, and/or address issues of affect and subjectivity. MediaEval strives to promote reproducible research that makes multimedia a positive force for society. Please comment on the human or social aspect of your task. Risk management: What are the main risks that you foresee for the task, and how you plan to address them (i.e., what challenges will you face in organizing the task and how do you expect to overcome them)?

Part III: Task Organization Team

Write a very brief paragraph outlining the relevant interests and experience of your organizing team. Your team should be large enough to handle the organization and management of the task. This includes evaluating participant runs, and carrying out failure analysis on the results. Ideally teams should consist of members from multiple research sites and multiple projects. A mix of experienced and early-career researchers is preferred. MediaEval has a strong tradition of encouraging and supporting early-stage researchers in gaining experience in organization of benchmark tasks. Note that your task team can add members after the proposal has been accepted.

Submission

Please submit your proposal (as a text file, .doc, .docx or link to an editable Google doc file) by emailing it to Martha Larson m (dot) larson (at) cs (dot) ru (dot) nl with Steven Hicks steven (at) simula (dot) no and Mihai Gabriel (Gabi) Constantin mihai.constantin84 (at) upb (dot) ro on cc.

MediaEval 2025 Schedule
  • Registration for task participation opens: April 2025
  • Development data release: May 2025
  • Test data release: June 2025
  • Runs due: Wed. 24 Sept 2025
  • Working notes papers due: Wed. 8 Oct 2025
  • MediaEval 2025 Workshop, Sat.-Sun. 25-26 October 2025, Dublin, Ireland and Online.

The MediaEval 2025 workshop will be held during a weekend so that it can occur exactly between ACM Multimedia 2025 and CBMI 2025. The scheduling helps to reduce traveling for those who are traveling. The workshop will be officially co-located with CBMI 2025.