MediaEval 2025

The MediaEval Multimedia Evaluation benchmark offers challenges in artificial intelligence for multimedia data. Participants address these challenges by creating algorithms for analyzing, exploring and accessing information in the data. Solutions are systematically compared using a common evaluation procedure, making it possible to establish the state of the art and track progress. Our larger aim is to promote reproducible research that makes multimedia a positive force for society.

MediaEval goes beyond other benchmarks and data science challenges in that it also pursues a “Quest for Insight” (Q4I). With Q4I we push beyond only striving to improve evaluation scores to also working to achieve deeper understanding about the challenges. For example, characteristics of the data, strengths and weaknesses of particular types of approaches, and observations about the evaluation procedure.

We are happy to announce the tasks that will be offered in 2025. Please watch this website for more detailed task information that will be posted soon.

Synthetic Images: Advancing detection of generative AI used in real-world online images

The goal of this challenge is to develop AI models capable of detecting synthetic images and identifying the specific regions in the images that have been manipulated or synthesized. Approaches will be tested on images synthesized with state-of-the-art approaches and collected from real-world settings online.

Participants are provided with multimodal web content from several cities listing food sharing initiatives (FSIs) in each city. For each city, participants are tasked with creating a multimodal summary of the FSI activities in the city which satisfy specified criteria. Evaluation will explore the use of emerging LLMs-based methods in automated assessment of multimodal multi-document summarization.

NewsImages: Retrieval and generative AI for news thumbnails

Participants receive a large set of articles (including the headline and article lead) in the English-language from international publishers. We offer two subtasks: retrieving an image for each article from a collection of images that can serve as a thumbnail, or generating an article thumbnail.

Memorability: Predicting the memorability of movie clips and commercial videos

The goal of this task is to predict the memorability of media items. For the memorability task, we provide movie excerpts, tasking teams with inferring how memorable videos are based on visual or EEG features, and commercial videos with the purpose of inferring the memorability of videos and the brands present in the videos.

Medico: VQA for gastrointestinal imaging

The goal is to use Visual Question Answering (VQA) to interpret and answer questions based on gastrointestinal images, aiming to enhance decision support and improve AI-driven medical decision-making. We provide a gastrointestinal dataset containing images and videos with VQA labels and additional metadata.

MediaEval 2025 Schedule:
The MediaEval Coordination Committee (2025):