Submission results

Leaderboard - PR-AUC-macro

Team Run PR-AUC-macro ROC-AUC-macro External data
1 SAIL-MiM-USC ensemble_all_data 0.160961 0.781229 MSD, Music4All
2 SAIL-MiM-USC best_single_model 0.156114 0.778264 MSD, Music4All
3 baseline best2019 0.154609 0.772913 -
4 SAIL-MiM-USC ensemble_jamendo_only 0.142199 0.762584 -
5 HCMUS run4_ensemble_wavenet_eff_b7 0.141446 0.766347 NSynth
6 HCMUS run1_wavenet_eff_ensemble 0.141343 0.768048 NSynth
7 HCMUS run2_mel_eff 0.139806 0.762775 NSynth
8 AugsBurger fusion_with_attention_CNN 0.131346 0.75339 -
9 UAI-CNRL run 0.127513 0.736007 -
10 AugsBurger fusion_of_CBAMs 0.122753 0.740579 -
11 AugsBurger CBAM_GRU_256 0.120311 0.739457 -
12 AUGment run4_AReLU_AttConv_vggish 0.117882 0.735327 -
13 AUGment run3_AReLU_AttConv 0.113609 0.732323 -
14 AUGment run2_AttConv 0.108263 0.716909 -
15 baseline vggish 0.107734 0.725821 -
16 AUGment run1_AReLU 0.107274 0.728143 -
17 AugsBurger CBAM_GRU_128x2 0.107014 0.715848 -
18 HCMUS run3_wavenet_eff_b7 0.105443 0.718574 NSynth
19 UIBK-DBIS run5_ae_crnn 0.096506 0.704386 -
20 UIBK-DBIS run4_ae_ecrnn_f1 0.090345 0.684949 -
21 UIBK-DBIS run1_ae_ecrnn_manually 0.090083 0.688518 -
22 UIBK-DBIS run2_ecrnn_manually 0.088748 0.695354 -
23 UIBK-DBIS run3_ae_ecrnn_manually_3 0.086223 0.682994 -
24 baseline popular 0.031924 0.5 -

Leaderboard - F-score-macro

Team Run F-score-macro External data
1 SAIL-MiM-USC ensemble_all_data 0.220355 MSD, Music4All
2 SAIL-MiM-USC best_single_model 0.214483 MSD, Music4All
3 baseline best2019 0.212419 -
4 SAIL-MiM-USC ensemble_jamendo_only 0.197606 -
5 AugsBurger fusion_with_attention_CNN 0.190123 -
6 UAI-CNRL run 0.188362 -
7 AugsBurger fusion_of_CBAMs 0.182119 -
8 AugsBurger CBAM_GRU_256 0.176505 -
9 AUGment run4_AReLU_AttConv_vggish 0.173821 -
10 AUGment run3_AReLU_AttConv 0.170373 -
11 AugsBurger CBAM_GRU_128x2 0.167762 -
12 HCMUS run1_wavenet_eff_ensemble 0.166857 NSynth
13 baseline vggish 0.165694 -
14 AUGment run2_AttConv 0.164292 -
15 AUGment run1_AReLU 0.163079 -
16 HCMUS run3_wavenet_eff_b7 0.162191 NSynth
17 UIBK-DBIS run1_ae_ecrnn_manually 0.110748 -
18 UIBK-DBIS run2_ecrnn_manually 0.105222 -
19 UIBK-DBIS run5_ae_crnn 0.104062 -
20 UIBK-DBIS run3_ae_ecrnn_manually_3 0.102853 -
21 UIBK-DBIS run4_ae_ecrnn_f1 0.099583 -
22 HCMUS run2_mel_eff 0.061512 NSynth
23 HCMUS run4_ensemble_wavenet_eff_b7 0.059469 NSynth
24 baseline popular 0.002642 -

Precision vs recall (macro)

All submissions

AUGment

Source code: https://github.com/SrividyaTR/MediaEval2020-EmotionThemeInMusic-UsingAttentionMethods

Paper: https://www.eigen.no/MediaEval_20_paper_27.pdf

PR-AUC-macro ROC-AUC-macro F-score-macro precision-macro recall-macro PR-AUC-micro ROC-AUC-micro F-score-micro precision-micro recall-micro
run1_AReLU 0.107274 0.728143 0.163079 0.132147 0.347565 0.118236 0.777542 0.160793 0.099977 0.410497
run2_AttConv 0.108263 0.716909 0.164292 0.130748 0.328551 0.132401 0.774603 0.159789 0.100628 0.387758
run3_AReLU_AttConv 0.113609 0.732323 0.170373 0.139861 0.347886 0.130649 0.782434 0.168341 0.105416 0.417636
run4_AReLU_AttConv_vggish 0.117882 0.735327 0.173821 0.14781 0.338847 0.137826 0.785888 0.170372 0.107862 0.405209

AugsBurger

Paper: https://www.eigen.no/MediaEval_20_paper_60.pdf

PR-AUC-macro ROC-AUC-macro F-score-macro precision-macro recall-macro PR-AUC-micro ROC-AUC-micro F-score-micro precision-micro recall-micro
CBAM_GRU_128x2 0.107014 0.715848 0.167762 0.13221 0.362919 0.120988 0.769566 0.15628 0.096794 0.405473
CBAM_GRU_256 0.120311 0.739457 0.176505 0.149108 0.347888 0.147697 0.789937 0.16753 0.105811 0.402036
fusion_of_CBAMs 0.122753 0.740579 0.182119 0.156775 0.379102 0.150032 0.792665 0.164222 0.101451 0.430724
fusion_with_attention_CNN 0.131346 0.75339 0.190123 0.166658 0.341665 0.15481 0.80143 0.190805 0.124373 0.409572

HCMUS

Paper: https://www.eigen.no/MediaEval_20_paper_49.pdf

PR-AUC-macro ROC-AUC-macro F-score-macro precision-macro recall-macro PR-AUC-micro ROC-AUC-micro F-score-micro precision-micro recall-micro
run1_wavenet_eff_ensemble 0.141343 0.768048 0.166857 0.164233 0.292303 0.166512 0.810437 0.160265 0.106191 0.326547
run2_mel_eff 0.139806 0.762775 0.061512 0.217086 0.042421 0.158799 0.805604 0.08373 0.417062 0.046536
run3_wavenet_eff_b7 0.105443 0.718574 0.162191 0.128442 0.330638 0.127659 0.770255 0.166088 0.104459 0.405077
run4_ensemble_wavenet_eff_b7 0.141446 0.766347 0.059469 0.187627 0.041887 0.167412 0.808931 0.07813 0.324012 0.044421

SAIL-MiM-USC

Source code: https://github.com/usc-sail/media-eval-2020

Paper: https://www.eigen.no/MediaEval_20_paper_67.pdf

PR-AUC-macro ROC-AUC-macro F-score-macro precision-macro recall-macro PR-AUC-micro ROC-AUC-micro F-score-micro precision-micro recall-micro
best_single_model 0.156114 0.778264 0.214483 0.182961 0.419388 0.181689 0.81769 0.183258 0.114197 0.463644
ensemble_all_data 0.160961 0.781229 0.220355 0.199995 0.391787 0.186651 0.81978 0.193639 0.122862 0.456769
ensemble_jamendo_only 0.142199 0.762584 0.197606 0.168918 0.387686 0.173985 0.804861 0.169183 0.104116 0.451084

UAI-CNRL

Source code: https://github.com/alishdipani/Multimediaeval2020-emotions-and-themes-in-music

Paper: https://www.eigen.no/MediaEval_20_paper_71.pdf

PR-AUC-macro ROC-AUC-macro F-score-macro precision-macro recall-macro PR-AUC-micro ROC-AUC-micro F-score-micro precision-micro recall-micro
run 0.127513 0.736007 0.188362 0.163955 0.348706 0.136924 0.786479 0.173498 0.110527 0.403226

UIBK-DBIS

Source code: https://github.com/dbis-uibk/MediaEval2020

Paper: https://www.eigen.no/MediaEval_20_paper_9.pdf

PR-AUC-macro ROC-AUC-macro F-score-macro precision-macro recall-macro PR-AUC-micro ROC-AUC-micro F-score-micro precision-micro recall-micro
run1_ae_ecrnn_manually 0.090083 0.688518 0.110748 0.06556 0.602292 0.084826 0.731992 0.105514 0.057564 0.631809
run2_ecrnn_manually 0.088748 0.695354 0.105222 0.060612 0.692202 0.098483 0.745664 0.098055 0.052761 0.692755
run3_ae_ecrnn_manually_3 0.086223 0.682994 0.102853 0.060328 0.628552 0.068761 0.700225 0.100887 0.054715 0.646087
run4_ae_ecrnn_f1 0.090345 0.684949 0.099583 0.057188 0.645761 0.07474 0.71062 0.099978 0.054058 0.664067
run5_ae_crnn 0.096506 0.704386 0.104062 0.060084 0.672646 0.091134 0.720173 0.100305 0.054147 0.679799

baseline

Source code: https://github.com/MTG/mtg-jamendo-dataset

PR-AUC-macro ROC-AUC-macro F-score-macro precision-macro recall-macro PR-AUC-micro ROC-AUC-micro F-score-micro precision-micro recall-micro
best_2019 0.154609 0.772913 0.212419 0.190088 0.400924 0.17785 0.815949 0.189553 0.11919 0.462718
popular 0.031924 0.5 0.002642 0.001427 0.017857 0.034067 0.513856 0.057312 0.079887 0.044685
vggish 0.107734 0.725821 0.165694 0.138216 0.30865 0.140913 0.775029 0.177133 0.116097 0.37348