See the MediaEval 2020 webpage for information on how to register and participate.
The “Medico automatic polyp segmentation challenge” aims to develop computer-aided diagnosis systems for automatic polyp segmentation to detect all types of polyps (for example, irregular polyp, smaller or flat polyps) with high efficiency and accuracy. The main goal of the challenge is to benchmark semantic segmentation algorithms on a publicly available dataset, emphasizing robustness, speed, and generalization.
Participants will get access to a dataset consisting of 1,000 segmented polyp images from the gastrointestinal tract and a separate testing dataset. The challenge consists of two mandatory tasks, each focused on a different requirement for efficient polyp detection. We hope that this task encourages multimedia researchers to apply their vast knowledge to the medical field and make an impact that may affect real lives.
The participants are invited to submit the results on the following tasks:
1) Polyp segmentation task (required) - The polyp segmentation task asks participants to develop algorithms for segmenting polyps on a comprehensive dataset.
2) Algorithm efficiency task (required) - The algorithm efficiency task is similar to task one but puts a stronger emphasis on the algorithm’s speed in terms of frames-per-second. To ensure a fair evaluation, this task requires participants to submit a Docker image so that all algorithms are evaluated on the same hardware.
Colonoscopy is currently the gold-standard medical procedure for examining the colon for lesions and other abnormalities such as cancer. Colorectal cancer (CRC) is the third most prevailing strain in terms of cancer incidence and second in terms of mortality globally. As early detection is critical for patient survival, regular screening through colonoscopy is a prerequisite for cancer detection and colorectal cancer prevention. Regardless of the success of colonoscopy, it is still estimated to miss around 20% of polyps. This is mostly due to the heavy reliance on the skill of the clinician operating the endoscope and his/her ability to detect polyps. An automated computer-aided diagnosis (CAD) system could be one of the potential solutions for the overlooked polyps. Such detection or surveillance systems can give doctors a so-called “third-eye”, thereby alerting them of missed polyps or other common lesions.
The task is of interest to the researchers working with multimedia segmentation, deep learning (semantic segmentation), and computer vision. We especially encourage young researchers to contribute to the field of endoscopy by developing an automated computer-aided diagnosis system that could be potentially used in clinical settings.
The dataset contains 1,000 polyp images and their corresponding ground truth mask. The datasets were collected from real routine clinical examinations at Vestre Viken Health Trust (VV) in Norway by expert gastroenterologists. The VV is the collaboration of the four hospitals that provide healthcare service to 470,000 peoples. The resolution of images varies from 332✕487 to 1920✕1072 pixels. Some of the images contain green thumbnail in the lower-left corner of the images showing the position marking from the ScopeGuide (Olympus). The training dataset can be downloaded from https://datasets.simula.no/kvasir-seg/.
The test dataset is now released. It can be downloaded from https://drive.google.com/file/d/1uP2W2g0iCCS3T6Cf7TPmNdSX4gayOrv2/view?usp=sharing.
The task will use mean Intersection over Union (mIoU) or Jaccard index as an evaluation metric, which is a standard metric for all medical segmentation task. If the teams have the same mIoU values, then the teams will be further evaluated on the basis of the higher value of the dice coefficient. We strongly recommend calculating other standard evaluation metrics such as dice coefficient, precision, recall, F2, and frame per second (FPS) for a comprehensive evaluation.
In the challenge overview paper, the organizers will be calculating the metrics such as the Dice coefficient, mIoU, Recall, Precision,Overlap, F2, MAE, FPS, s-score, and Clinical relevance of the methods submitted by each team.
The submissions will be verified by inspecting the corresponding Docker image. If you have a problem with submitting the Docker image, we are flexible to accept a zip file that contains the predicted mask for task 1. For task 2, we require the Docker submission so that we can evaluate on the same hardware. For the more instruction about the Docker submission, please refer to our GitHub repository.
By registering this challenge, each individual or team agrees to use only the provided dataset. After the challenge, the dataset test dataset will be made available and the authors can only use it for publication or any non-commercial use. A participating team will be allowed to make a maximum of 5 submissions. The test data can not be used while training the model. The results will be evaluated by the organizers and presented at MediaEval 2020.
To facilitate communication within the participants, we have set up a Discord channel. You can use this channel for discussion of the challenge or ask questions. Please email debesh@simula.no for the invitation at the Discord.
Debesh Jha, Pia H. Smedsrud, Michael A. Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D. Johansen. 2020. Kvasir-seg: A segmented polyp dataset, In Proceeding of International Conference on Multimedia Modeling (MMM), 451-462.
Debesh Jha, Pia H. Smedsrud, Michael A. Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D. Johansen. 2019. Resunet++: An advanced architecture for medical image segmentation, In International Symposium on Multimedia (ISM), 225-2255.
Tobias Ross et al. 2020. Robust Medical Instrument Segmentation Challenge 2019, arXiv preprint.
Sharib Ali et al. 2020. An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy, Scientific Reports.
Workshop will be held online. Exact dates to be announced.