Submission results
Leaderboard - PR-AUC-macro
Team | Run | PR-AUC-macro | ROC-AUC-macro | External data | |
---|---|---|---|---|---|
1 | SAIL-MiM-USC | ensemble_all_data | 0.160961 | 0.781229 | MSD, Music4All |
2 | SAIL-MiM-USC | best_single_model | 0.156114 | 0.778264 | MSD, Music4All |
3 | baseline | best2019 | 0.154609 | 0.772913 | - |
4 | SAIL-MiM-USC | ensemble_jamendo_only | 0.142199 | 0.762584 | - |
5 | HCMUS | run4_ensemble_wavenet_eff_b7 | 0.141446 | 0.766347 | NSynth |
6 | HCMUS | run1_wavenet_eff_ensemble | 0.141343 | 0.768048 | NSynth |
7 | HCMUS | run2_mel_eff | 0.139806 | 0.762775 | NSynth |
8 | AugsBurger | fusion_with_attention_CNN | 0.131346 | 0.75339 | - |
9 | UAI-CNRL | run | 0.127513 | 0.736007 | - |
10 | AugsBurger | fusion_of_CBAMs | 0.122753 | 0.740579 | - |
11 | AugsBurger | CBAM_GRU_256 | 0.120311 | 0.739457 | - |
12 | AUGment | run4_AReLU_AttConv_vggish | 0.117882 | 0.735327 | - |
13 | AUGment | run3_AReLU_AttConv | 0.113609 | 0.732323 | - |
14 | AUGment | run2_AttConv | 0.108263 | 0.716909 | - |
15 | baseline | vggish | 0.107734 | 0.725821 | - |
16 | AUGment | run1_AReLU | 0.107274 | 0.728143 | - |
17 | AugsBurger | CBAM_GRU_128x2 | 0.107014 | 0.715848 | - |
18 | HCMUS | run3_wavenet_eff_b7 | 0.105443 | 0.718574 | NSynth |
19 | UIBK-DBIS | run5_ae_crnn | 0.096506 | 0.704386 | - |
20 | UIBK-DBIS | run4_ae_ecrnn_f1 | 0.090345 | 0.684949 | - |
21 | UIBK-DBIS | run1_ae_ecrnn_manually | 0.090083 | 0.688518 | - |
22 | UIBK-DBIS | run2_ecrnn_manually | 0.088748 | 0.695354 | - |
23 | UIBK-DBIS | run3_ae_ecrnn_manually_3 | 0.086223 | 0.682994 | - |
24 | baseline | popular | 0.031924 | 0.5 | - |
Leaderboard - F-score-macro
Team | Run | F-score-macro | External data | |
---|---|---|---|---|
1 | SAIL-MiM-USC | ensemble_all_data | 0.220355 | MSD, Music4All |
2 | SAIL-MiM-USC | best_single_model | 0.214483 | MSD, Music4All |
3 | baseline | best2019 | 0.212419 | - |
4 | SAIL-MiM-USC | ensemble_jamendo_only | 0.197606 | - |
5 | AugsBurger | fusion_with_attention_CNN | 0.190123 | - |
6 | UAI-CNRL | run | 0.188362 | - |
7 | AugsBurger | fusion_of_CBAMs | 0.182119 | - |
8 | AugsBurger | CBAM_GRU_256 | 0.176505 | - |
9 | AUGment | run4_AReLU_AttConv_vggish | 0.173821 | - |
10 | AUGment | run3_AReLU_AttConv | 0.170373 | - |
11 | AugsBurger | CBAM_GRU_128x2 | 0.167762 | - |
12 | HCMUS | run1_wavenet_eff_ensemble | 0.166857 | NSynth |
13 | baseline | vggish | 0.165694 | - |
14 | AUGment | run2_AttConv | 0.164292 | - |
15 | AUGment | run1_AReLU | 0.163079 | - |
16 | HCMUS | run3_wavenet_eff_b7 | 0.162191 | NSynth |
17 | UIBK-DBIS | run1_ae_ecrnn_manually | 0.110748 | - |
18 | UIBK-DBIS | run2_ecrnn_manually | 0.105222 | - |
19 | UIBK-DBIS | run5_ae_crnn | 0.104062 | - |
20 | UIBK-DBIS | run3_ae_ecrnn_manually_3 | 0.102853 | - |
21 | UIBK-DBIS | run4_ae_ecrnn_f1 | 0.099583 | - |
22 | HCMUS | run2_mel_eff | 0.061512 | NSynth |
23 | HCMUS | run4_ensemble_wavenet_eff_b7 | 0.059469 | NSynth |
24 | baseline | popular | 0.002642 | - |
Precision vs recall (macro)
All submissions
AUGment
Source code: https://github.com/SrividyaTR/MediaEval2020-EmotionThemeInMusic-UsingAttentionMethods
Paper: https://www.eigen.no/MediaEval_20_paper_27.pdf
PR-AUC-macro | ROC-AUC-macro | F-score-macro | precision-macro | recall-macro | PR-AUC-micro | ROC-AUC-micro | F-score-micro | precision-micro | recall-micro | |
---|---|---|---|---|---|---|---|---|---|---|
run1_AReLU | 0.107274 | 0.728143 | 0.163079 | 0.132147 | 0.347565 | 0.118236 | 0.777542 | 0.160793 | 0.099977 | 0.410497 |
run2_AttConv | 0.108263 | 0.716909 | 0.164292 | 0.130748 | 0.328551 | 0.132401 | 0.774603 | 0.159789 | 0.100628 | 0.387758 |
run3_AReLU_AttConv | 0.113609 | 0.732323 | 0.170373 | 0.139861 | 0.347886 | 0.130649 | 0.782434 | 0.168341 | 0.105416 | 0.417636 |
run4_AReLU_AttConv_vggish | 0.117882 | 0.735327 | 0.173821 | 0.14781 | 0.338847 | 0.137826 | 0.785888 | 0.170372 | 0.107862 | 0.405209 |
AugsBurger
Paper: https://www.eigen.no/MediaEval_20_paper_60.pdf
PR-AUC-macro | ROC-AUC-macro | F-score-macro | precision-macro | recall-macro | PR-AUC-micro | ROC-AUC-micro | F-score-micro | precision-micro | recall-micro | |
---|---|---|---|---|---|---|---|---|---|---|
CBAM_GRU_128x2 | 0.107014 | 0.715848 | 0.167762 | 0.13221 | 0.362919 | 0.120988 | 0.769566 | 0.15628 | 0.096794 | 0.405473 |
CBAM_GRU_256 | 0.120311 | 0.739457 | 0.176505 | 0.149108 | 0.347888 | 0.147697 | 0.789937 | 0.16753 | 0.105811 | 0.402036 |
fusion_of_CBAMs | 0.122753 | 0.740579 | 0.182119 | 0.156775 | 0.379102 | 0.150032 | 0.792665 | 0.164222 | 0.101451 | 0.430724 |
fusion_with_attention_CNN | 0.131346 | 0.75339 | 0.190123 | 0.166658 | 0.341665 | 0.15481 | 0.80143 | 0.190805 | 0.124373 | 0.409572 |
HCMUS
Paper: https://www.eigen.no/MediaEval_20_paper_49.pdf
PR-AUC-macro | ROC-AUC-macro | F-score-macro | precision-macro | recall-macro | PR-AUC-micro | ROC-AUC-micro | F-score-micro | precision-micro | recall-micro | |
---|---|---|---|---|---|---|---|---|---|---|
run1_wavenet_eff_ensemble | 0.141343 | 0.768048 | 0.166857 | 0.164233 | 0.292303 | 0.166512 | 0.810437 | 0.160265 | 0.106191 | 0.326547 |
run2_mel_eff | 0.139806 | 0.762775 | 0.061512 | 0.217086 | 0.042421 | 0.158799 | 0.805604 | 0.08373 | 0.417062 | 0.046536 |
run3_wavenet_eff_b7 | 0.105443 | 0.718574 | 0.162191 | 0.128442 | 0.330638 | 0.127659 | 0.770255 | 0.166088 | 0.104459 | 0.405077 |
run4_ensemble_wavenet_eff_b7 | 0.141446 | 0.766347 | 0.059469 | 0.187627 | 0.041887 | 0.167412 | 0.808931 | 0.07813 | 0.324012 | 0.044421 |
SAIL-MiM-USC
Source code: https://github.com/usc-sail/media-eval-2020
Paper: https://www.eigen.no/MediaEval_20_paper_67.pdf
PR-AUC-macro | ROC-AUC-macro | F-score-macro | precision-macro | recall-macro | PR-AUC-micro | ROC-AUC-micro | F-score-micro | precision-micro | recall-micro | |
---|---|---|---|---|---|---|---|---|---|---|
best_single_model | 0.156114 | 0.778264 | 0.214483 | 0.182961 | 0.419388 | 0.181689 | 0.81769 | 0.183258 | 0.114197 | 0.463644 |
ensemble_all_data | 0.160961 | 0.781229 | 0.220355 | 0.199995 | 0.391787 | 0.186651 | 0.81978 | 0.193639 | 0.122862 | 0.456769 |
ensemble_jamendo_only | 0.142199 | 0.762584 | 0.197606 | 0.168918 | 0.387686 | 0.173985 | 0.804861 | 0.169183 | 0.104116 | 0.451084 |
UAI-CNRL
Source code: https://github.com/alishdipani/Multimediaeval2020-emotions-and-themes-in-music
Paper: https://www.eigen.no/MediaEval_20_paper_71.pdf
PR-AUC-macro | ROC-AUC-macro | F-score-macro | precision-macro | recall-macro | PR-AUC-micro | ROC-AUC-micro | F-score-micro | precision-micro | recall-micro | |
---|---|---|---|---|---|---|---|---|---|---|
run | 0.127513 | 0.736007 | 0.188362 | 0.163955 | 0.348706 | 0.136924 | 0.786479 | 0.173498 | 0.110527 | 0.403226 |
UIBK-DBIS
Source code: https://github.com/dbis-uibk/MediaEval2020
Paper: https://www.eigen.no/MediaEval_20_paper_9.pdf
PR-AUC-macro | ROC-AUC-macro | F-score-macro | precision-macro | recall-macro | PR-AUC-micro | ROC-AUC-micro | F-score-micro | precision-micro | recall-micro | |
---|---|---|---|---|---|---|---|---|---|---|
run1_ae_ecrnn_manually | 0.090083 | 0.688518 | 0.110748 | 0.06556 | 0.602292 | 0.084826 | 0.731992 | 0.105514 | 0.057564 | 0.631809 |
run2_ecrnn_manually | 0.088748 | 0.695354 | 0.105222 | 0.060612 | 0.692202 | 0.098483 | 0.745664 | 0.098055 | 0.052761 | 0.692755 |
run3_ae_ecrnn_manually_3 | 0.086223 | 0.682994 | 0.102853 | 0.060328 | 0.628552 | 0.068761 | 0.700225 | 0.100887 | 0.054715 | 0.646087 |
run4_ae_ecrnn_f1 | 0.090345 | 0.684949 | 0.099583 | 0.057188 | 0.645761 | 0.07474 | 0.71062 | 0.099978 | 0.054058 | 0.664067 |
run5_ae_crnn | 0.096506 | 0.704386 | 0.104062 | 0.060084 | 0.672646 | 0.091134 | 0.720173 | 0.100305 | 0.054147 | 0.679799 |
baseline
Source code: https://github.com/MTG/mtg-jamendo-dataset
PR-AUC-macro | ROC-AUC-macro | F-score-macro | precision-macro | recall-macro | PR-AUC-micro | ROC-AUC-micro | F-score-micro | precision-micro | recall-micro | |
---|---|---|---|---|---|---|---|---|---|---|
best_2019 | 0.154609 | 0.772913 | 0.212419 | 0.190088 | 0.400924 | 0.17785 | 0.815949 | 0.189553 | 0.11919 | 0.462718 |
popular | 0.031924 | 0.5 | 0.002642 | 0.001427 | 0.017857 | 0.034067 | 0.513856 | 0.057312 | 0.079887 | 0.044685 |
vggish | 0.107734 | 0.725821 | 0.165694 | 0.138216 | 0.30865 | 0.140913 | 0.775029 | 0.177133 | 0.116097 | 0.37348 |