MediaEval 2023

The MediaEval Multimedia Evaluation benchmark offers challenges in artificial intelligence for multimedia data. Participants address these challenges by creating algorithms for retrieval, analysis, and exploration. Solutions are systematically compared using a common evaluation procedure, making it possible to establish the state of the art and track progress. Our larger aim is to promote reproducible research that makes multimedia a positive force for society.

MediaEval goes beyond other benchmarks and data science challenges in that it also pursues a “Quest for Insight” (Q4I). With Q4I we push beyond only striving to improve evaluation scores to also working to achieve deeper understanding about the challenges. For example, properties of the data, strengths and weaknesses of particular types of approaches, and observations about the evaluation procedure.

The MediaEval 2023 Workshop will be held 1-2 February, collocated with MMM 2024 in Amsterdam, Netherlands and also online.

Detailed workshop schedule is in the Workshop Program

Task schedule

July-September 2023: Validation data release
October 2023: Test data release
15-30 November: Runs due (see the individual tasks pages below for the exact deadlines)
15 December 2023: Working notes papers due
1-2 February 2024: MediaEval 2023 Workshop, Collocated with MMM 2024 in Amsterdam, Netherlands and Online.

The MediaEval Coordination Committee (2023)

Mihai Gabriel Constantin, University Politehnica of Bucharest, Romania
Steven Hicks, SimulaMet, Norway
Martha Larson, Radboud University, Netherlands (Overall coordinator and main contact person)

MediaEval is grateful for the support of ACM Special Interest Group on Multimedia

Task List

Medical Multimedia Task - Transparent Tracking of Spermatozoa

Participants detect and track sperm in microscope videos, using provided microscope videos and metadata. They must analyze both global and individual sperm attributes such as motility, speed, and distance traveled. The annotations follow WHO sperm quality standards and are verified by experts.

Musti: Multimodal Understanding of Smells in Texts and Images

Participants develop classifiers to predict whether a text passage and an image evoke the same smell source or not and identify common smell sources text passages and images. Optionally, the challenge can be addressed in a cross-language setting.

NewsImages: Connecting Text and Images

Participants are supplied with a large set of articles (including text body, and headlines) and the accompanying images from international publishers. The task requires participants to predict which image was used to accompany each article.

Predicting Video Memorability

Participants automatically predict memorability scores for videos that reflect the probability that a video will be remembered. They will be provided with an extensive data set of videos with memorability annotations, related information, pre-extracted state-of-the-art visual features, and Electroencephalography (EEG) recordings.

SportsVideo: Fine Grained Action Classification and Position Detection in Table Tennis and Swimming Videos

Participants address video analysis challenges in two sports: table tennis and swimming videos. Subtasks include position detection and action classification and involve leveraging different modalities in the video data: visual, sound, and text.