MediaEval 2025

The MediaEval Multimedia Evaluation benchmark offers challenges in artificial intelligence for multimedia data. Participants address these challenges by creating algorithms for analyzing, exploring and accessing information in the data. Solutions are systematically compared using a common evaluation procedure, making it possible to establish the state of the art and track progress. Our larger aim is to promote reproducible research that makes multimedia a positive force for society.

MediaEval goes beyond other benchmarks and data science challenges in that it also pursues a “Quest for Insight” (Q4I). With Q4I we push beyond only striving to improve evaluation scores to also working to achieve deeper understanding about the challenges. For example, characteristics of the data, strengths and weaknesses of particular types of approaches, and observations about the evaluation procedure.

MediaEval 2025 Schedule:

Registration for task participation opens: April 2025
Development data release: May 2025
Test data release: June 2025
Runs due: Wed. 24 Sept 2025
Working notes papers due: Wed. 8 Oct 2025
MediaEval 2025 Workshop, Sat.-Sun. 25-26 October 2025, Dublin, Ireland and Online.

The MediaEval Coordination Committee (2025):

Mihai Gabriel Constantin, University Politehnica of Bucharest, Romania
Steven Hicks, SimulaMet, Norway
Martha Larson, Radboud University, Netherlands

Task List

Medico: VQA (with multimodal explanations) for gastrointestinal imaging

The goal is to use Visual Question Answering (VQA) to interpret and answer questions based on gastrointestinal images, aiming to enhance decision support and improve AI-driven medical decision-making. We provide a gastrointestinal dataset containing images and videos with VQA labels and additional metadata.

Memorability: Predicting movie and commercial memorability

The goal of this task is to study the long-term memory performance when recognizing small movie excerpts or commercial videos. We provide the videos, precomputed features or EEG features for the challenges proposed in the task such as How memorable a video, if a person familiar with a video or if you can predict the brand memorability?

MultiSumm: Multimodal summarization of multiple topically related websites

Participants are provided with multimodal web content from several cities listing food sharing initiatives (FSIs) in each city. For each city, participants are tasked with creating a multimodal summary of the FSI activities in the city that satisfy specified criteria. Evaluation will explore the use of emerging LLMs-based methods in automated assessment of multimodal multi-document summarization.

NewsImages: Retrieval and generative AI for news thumbnails

Participants receive a large set of articles (including the headline and article lead) in the English-language from international publishers. We offer two subtasks: retrieving an image for each article from a collection of images that can serve as a thumbnail, or generating an article thumbnail.

Synthetic Images: Advancing detection of generative AI used in real-world online images

The goal of this challenge is to develop AI models capable of detecting synthetic images and identifying the specific regions in the images that have been manipulated or synthesized. Approaches will be tested on images synthesized with state-of-the-art approaches and collected from real-world settings online.