What is MediaEval?

MediaEval is a benchmarking initiative dedicated to evaluating new algorithms for multimedia analysis, retrieval and exploration. It emphasizes the ‘multi’ in multimedia (by involving multiple modalities such as visual, textual, audio, and sensor data) and focuses on human and social aspects of multimedia tasks. Our larger aim is to promote reproducible research that makes multimedia a positive force for society. MediaEval attracts participants who are interested in multimodal approaches to multimedia involving, e.g., speech recognition, multimedia content analysis, music and audio analysis, user-contributed information (tags, tweets), viewer affective response, environmental sensing, social networks, temporal and geo-coordinates.

For more information about past years see:

SIGMM Records Volume 18, Issue 1, Jan 2026
SIGMM Records Volume 17, Issue 1, Jan 2025
SIGMM Records Volume 16, Issue 4, Oct 2024
SIGMM Records Volume 15, Issue 1, January 2023
SIGMM Records Volume 12, Issue 2, June 2020
MediaEval 2017 Overview of the Year
IEEE Multimedia Vol. 24, No. 1, 93-96, 2016 The Benchmarking Initiative for Multimedia Evaluation: MediaEval 2016
ERCIM News 97, April 2014
IEEE Speech & Language Processing Technical Committee Newsletter Feb. 2014
SIGMM Records Volume 5, Issue 2, June 2013
ERCIM News 94, July 2013
ERCIM News 88, January 2012
SIGMM Records Volume 3, Issue 2, June 2011
SIGMM Records Volume 2, Issue 2, June 2010

Who runs MediaEval?

MediaEval is a community-driven benchmark that is run by the MediaEval organizing committee consisting of the task organizers of all the individual task in a given year. MediaEval tasks are largely autonomous, and each team of task organizers is responsible for running their own tasks.

The overall operations of MediaEval are run by the MediaEval Coordination Committee:

  • Mihai Gabriel Constantin, University Politehnica of Bucharest, Romania
  • Steven Hicks, SimulaMet, Norway
  • Martha Larson, Radboud University, Netherlands

How can I get involved?

MediaEval is an open initiative, meaning that any interested research group is free to signup and participate. Groups sign up for one or more tasks, they then receive task definitions, data sets and supporting resources, which they use to develop their algorithms. At the very end of the season, groups submit their results and then they attend the MediaEval workshop. See also Why Participate? Prior to 2021, MediaEval published videos on the MediaEval YouTube channel.

MediaEval welcomes new task proposals. At the beginning of the year, a call for new task proposals appears. Groups of researchers can propose to organize a new task, or to continue a task organized in past years. Proposing a task requires creating a task organization team, creating a task design (task definition that fits the user scenario, evaluation methodology) and laying the ground work for task logistics (source of data, source of ground truth, evaluation metric). If you have an idea for a task that is not fully developed, you can propose a MediaEval Task Forces, which is a group of people informally working towards a task to be proposed in a future year of MediaEval.

What is a MediaEval task?

A MediaEval task consists of four parts:

  • Data provided to the benchmark participants,
  • A task definition that describes the problem to be solved,
  • Ground truth against which participants’ algorithms are evaluated,
  • An evaluation metric.

MediaEval tasks are oriented towards user needs in specific application settings and, to the extent possible, are based on scenarios of use derived from real-world problems.

What is the MediaEval Workshop?

The culmination of the yearly MediaEval benchmarking cycle is the MediaEval Workshop. The workshop brings together task participants to present their findings, discuss their approaches, learn from each other, and make plans for future work.

Since 2017, MediaEval has been offering opportunities for remote participation and in 2020-2021 the workshop took place fully online. Since 2022, the workshop has been hybrid, welcoming both in person and online participants.

  • MediaEval 2010 Workshop: Santa Croce in Fossabanda, Pisa, Italy, 24 October 2010, ACM Multimedia 2010.
  • MediaEval 2011 Workshop: Santa Croce in Fossabanda, Pisa, Italy, 1-2 September 2011, Interspeech 2011.
  • MediaEval 2012 Workshop: Santa Croce in Fossabanda, Pisa, Italy, 4-5 October 2012, ECCV 2012.
  • MediaEval 2013 Workshop: Reial Acadèmia de Bones Lletres de Barcelona, 18-19 October 2013, ACM Multimedia 2013.
  • MediaEval 2014 Workshop: Reial Acadèmia de Bones Lletres de Barcelona, 16-17 October 2014.
  • MediaEval 2015 Workshop: Schloss Wurzen, Germany, 14-15 September 2015, Interspeech 2015.
  • MediaEval 2016 Workshop: Netherlands Institute for Sound and Vision, Netherlands, 20-21 October 2016, ACM Multimedia 2016.
  • MediaEval 2017 Workshop: Trinity College Dublin, Ireland, 13-15 September 2017, CLEF 2017.
  • MediaEval 2018 Workshop: EURECOM Sophia Antipolis, France, 29-31 October 2018.
  • MediaEval 2019 Workshop: EURECOM Sophia Antipolis, France, 27-29 October 2019, ACM Multimedia 2019.
  • MediaEval 2020 Workshop: Online, 11,14-15 December 2020.
  • MediaEval 2021 Workshop: Online, 13-15 December 2021.
  • MediaEval 2022 Workshop: Media City Bergen, Norway and Online, 12-13 January 2023.
  • MediaEval 2023 Workshop: University of Amsterdam, Netherlands and Online Amsterdam, 1-2 Feb 2024, MMM 2024.
  • MediaEval 2025: Dublin, Ireland, 25-26 Oct 2025, Between CMBI 2025 and ACM MM 2025.
  • MediaEval 2026: KIT (formerly Royal Tropical Institute), Amsterdam, Netherlands, 15-16 June 2026, ACM ICMR 2026.

Each year, the workshop publishes a working notes proceedings containing papers written by the task organizers and task participants. The aims of the MediaEval Working Notes Proceedings are described in more detail in this paper.

The MediaEval workshop also welcomes attendees who did not participate in specific tasks, but who are interested in multimedia research, or getting involved in MediaEval in the future.

How did MediaEval come into being?

Martha Larson and Gareth Jones founded MediaEval in 2008 as VideoCLEF, a track in the CLEF Campaign. Martha Larson serves as the overall coordinator.

MediaEval became an independent benchmarking initiative in 2010 under the auspices of the PetaMedia Network of Excellence. In 2011, it also received support from ICT Labs of EIT. Since 2012, MediaEval has run as a fully bottom-up benchmark, in that it is not associated with a single “parent project”.

For support over the years we would particularly like to thank the ELIAS (Evaluating Information Access Systems), an ESF Research Networking Programme, the ACM SIGIR Special Interest Group on Information Retrieval, and the ACM SIGMM Special Interest Group on Multimedia. We are grateful for the support, specifically, because it enables us to offer travel grants to students and other researchers in need of support. The Multimedia Computing Group at TU Delft has made an important contribution to MediaEval, and we would especially like to thank Saskia Peters. Please refer to the pages of the individual years for complete lists of supporters.

Why participate ?

MediaEval is innovative: Expand your research horizons by trying out a new task. We make every effort to keep the threshold low for entering new tasks.

MediaEval is flexible: Choose the tasks that interest you. Innovate your own combination of visual, text, speech, audio, sensor and social features.

MediaEval is both competitive and supportive: Develop or refine your techniques by comparing your results to those of others. Both newbies as well as seasoned researchers participate.

MediaEval brings researchers together: Meet other people working on similar topics.

MediaEval strengthens projects: Join MediaEval as a task organizer. Propose a new task based on your project.

MediaEval is cost effective: We bundle resources to keep costs low. Participation in MediaEval is free of charge and we are making every effort to make the yearly workshop reasonably priced.

MediaEval lets you meet new people and see the world: MediaEval welcomes new participats and also in the past has held events in some pretty interesting places!