-
Notifications
You must be signed in to change notification settings - Fork 205
1604Argus
- R_rg_2avgBV_EP100_mask_L1e-5
- R_rg_2danBV_EP100_mask_L1e-5_W13
- R_rg_2rnnBV_EP100_mask_L1e-4_i13d13 (8)
- R_rg_2cnnBV_EP100_mask_L1e-4_i13d13 (8)
- R_rg_2rnncnnBV_EP100_mask_L1e-4_i13d13 (8)
- R_rg_2a51BV_EP100_mask_L1e-5_fasgmn_crelu
- R_urg11299592rnnBV_EP100_mask_rmsprop_mlp
| Model | trn QAcc | val QAcc | val QF1 | tst QAcc | tst QF1 | settings |
|---|---|---|---|---|---|---|
| avg | 0.871582 | 0.816243 | 0.715536 | 0.743671 | 0.671109 | (defaults) |
| ±0.008774 | ±0.007793 | ±0.013344 | ±0.019701 | ±0.031045 | ||
| DAN | 0.883804 | 0.821856 | 0.745652 | 0.754351 | 0.691760 |
inp_e_dropout=0 inp_w_dropout=1/3 deep=2 pact='relu'
|
| ±0.012438 | ±0.011024 | ±0.022104 | ±0.025300 | ±0.042076 | ||
| -------------------------- | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- |
| rnn | 0.906453 | 0.875000 | 0.812159 | 0.822785 | 0.781521 |
inp_e_dropout=1/3 dropout=1/3
|
| ±0.012775 | ±0.005396 | ±0.008444 | ±0.008261 | ±0.011528 | ||
| cnn | 0.896200 | 0.856662 | 0.802337 | 0.821598 | 0.793560 |
inp_e_dropout=1/3 dropout=1/3
|
| ±0.018262 | ±0.005804 | ±0.006581 | ±0.006662 | ±0.006533 | ||
| rnncnn | 0.884963 | 0.860030 | 0.802535 | 0.816456 | 0.780175 |
inp_e_dropout=1/3 dropout=1/3
|
| ±0.009792 | ±0.006566 | ±0.007993 | ±0.009081 | ±0.012383 | ||
| attn1511 | 0.935063 | 0.877046 | 0.821878 | 0.816456 | 0.764327 |
focus_act='sigmoid/maxnorm' cnnact='relu'
|
| ±0.021065 | ±0.007928 | ±0.012047 | ±0.007572 | ±0.012779 | ||
| Ubu. rnn w/MLP | 0.950765 | 0.912425 | 0.865724 | 0.852057 | 0.804517 |
vocabt='ubuntu' pdim=1 ptscorer=B.mlp_ptscorer dropout=0 inp_e_dropout=0 task1_conf={'ptscorer':B.dot_ptscorer, 'f_add_kw':False} opt='rmsprop'
|
| ±0.016681 | ±0.004351 | ±0.007591 | ±0.008250 | ±0.015434 |
Variants with different relevance modelling - no relevance, or BM25 metric in the mix.
| Model | trn QAcc | val QAcc | val QF1 | tst QAcc | tst QF1 | settings |
|---|---|---|---|---|---|---|
| avg | 0.872660 | 0.822754 | 0.737748 | 0.745570 | 0.695175 | rel_mode=None |
| ±0.024101 | ±0.029963 | ±0.050214 | ±0.050540 | ±0.065933 | ||
| avg | 0.869323 | 0.821108 | 0.715507 | 0.752769 | 0.678428 |
prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] f_add_S2=['bm25']
|
| ±0.029043 | ±0.014130 | ±0.033405 | ±0.026164 | ±0.047926 | ||
| avg | 0.742412 | 0.742515 | nan | 0.670886 | nan |
prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] rel_mode='bm25'
|
| ±0.069382 | ±0.043924 | ±nan | ±0.042191 | ±nan | ||
| avg | 0.872828 | 0.816243 | 0.737179 | 0.769778 | 0.716633 |
prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] f_add_S1=['bm25'] rel_mode=None
|
| ±0.007759 | ±0.007922 | ±0.011952 | ±0.010984 | ±0.017375 | ||
| -------------------------- | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- |
| rnn | 0.914736 | 0.868263 | 0.803783 | 0.821994 | 0.786475 |
inp_e_dropout=1/3 dropout=1/3 rel_mode=None
|
| ±0.022191 | ±0.007080 | ±0.011420 | ±0.014596 | ±0.016089 | ||
| rnn | 0.905352 | 0.866018 | 0.797045 | 0.814082 | 0.772631 |
inp_e_dropout=1/3 dropout=1/3 prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] f_add_S2=['bm25']
|
| ±0.009524 | ±0.006593 | ±0.011930 | ±0.007909 | ±0.012046 | ||
| rnn | 0.758689 | 0.762725 | nan | 0.700158 | nan |
inp_e_dropout=1/3 dropout=1/3 prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] rel_mode='bm25'
|
| ±0.136521 | ±0.092000 | ±nan | ±0.093423 | ±nan | ||
| rnn | 0.901413 | 0.874251 | 0.815608 | 0.828323 | 0.790401 |
inp_e_dropout=1/3 dropout=1/3 prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] f_add_S1=['bm25'] rel_mode=None
|
| ±0.012837 | ±0.008671 | ±0.013353 | ±0.014834 | ±0.017081 | ||
| -------------------------- | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- |
| cnn | 0.889829 | 0.847305 | 0.788506 | 0.817247 | 0.787495 |
inp_e_dropout=1/3 dropout=1/3 rel_mode=None
|
| ±0.029667 | ±0.006622 | ±0.010412 | ±0.015526 | ±0.012572 | ||
| cnn | 0.746293 | 0.755240 | nan | 0.700158 | nan |
inp_e_dropout=1/3 dropout=1/3 prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] rel_mode='bm25'
|
| ±0.125400 | ±0.085816 | ±nan | ±0.093348 | ±nan | ||
| rnncnn | 0.885774 | 0.854790 | 0.795130 | 0.817247 | 0.786409 |
inp_e_dropout=1/3 dropout=1/3 rel_mode=None
|
| ±0.024995 | ±0.007816 | ±0.017369 | ±0.010726 | ±0.009953 | ||
| rnncnn | 0.778267 | 0.783683 | nan | 0.738924 | nan |
inp_e_dropout=1/3 dropout=1/3 prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] rel_mode='bm25'
|
| ±0.118755 | ±0.085010 | ±nan | ±0.098486 | ±nan | ||
| attn1511 | 0.928290 | 0.866018 | 0.809381 | 0.818829 | 0.773699 |
focus_act='sigmoid/maxnorm' cnnact='relu' rel_mode=None
|
| ±0.033857 | ±0.008278 | ±0.017342 | ±0.013474 | ±0.022847 | ||
| attn1511 | 0.811284 | 0.786677 | nan | 0.729430 | nan |
focus_act='sigmoid/maxnorm' cnnact='relu' prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] rel_mode='bm25'
|
| ±0.140512 | ±0.087390 | ±nan | ±0.091370 | ±nan | ||
| attn1511 | 0.932171 | 0.871257 | 0.820158 | 0.810918 | 0.763896 |
focus_act='sigmoid/maxnorm' cnnact='relu' prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] rel_mode=None f_add_S1=['bm25']
|
| ±0.017671 | ±0.007223 | ±0.009922 | ±0.012153 | ±0.018292 | ||
| -------------------------- | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- |
| Ubu. rnn w/MLP | 0.924930 | 0.912425 | 0.863092 | 0.846519 | 0.795959 |
vocabt='ubuntu' pdim=1 ptscorer=B.mlp_ptscorer dropout=0 inp_e_dropout=0 task1_conf={'ptscorer':B.dot_ptscorer, 'f_add':[]} opt='rmsprop' rel_mode=None
|
| ±0.015340 | ±0.003484 | ±0.005951 | ±0.009069 | ±0.013594 | ||
| Ubu. rnn w/MLP | 0.926437 | 0.906437 | 0.855672 | 0.830696 | 0.776912 |
vocabt='ubuntu' pdim=1 ptscorer=B.mlp_ptscorer dropout=0 inp_e_dropout=0 task1_conf={'ptscorer':B.dot_ptscorer, 'f_add':[]} opt='rmsprop' prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] rel_mode=None f_add_S1=['bm25']
|
| ±0.016855 | ±0.005562 | ±0.009583 | ±0.018089 | ±0.030604 | ||
| Ubu. rnn w/MLP | 0.799583 | 0.809880 | nan | 0.740506 | nan |
vocabt='ubuntu' pdim=1 ptscorer=B.mlp_ptscorer dropout=0 inp_e_dropout=0 task1_conf={'ptscorer':B.dot_ptscorer, 'f_add':[]} opt='rmsprop' prescoring='termfreq' prescoring_weightsf='weights-anssel-termfreq-3368350fbcab42e4-bestval.h5' prescoring_input='bm25' f_add=['bm25'] rel_mode='bm25'
|
| ±0.131607 | ±0.102343 | ±nan | ±0.099308 | ±nan |
(The QF1-nan'd measurements are troubled by many trainings which failed with nan loss early - so there's a lot of untrained-model accuracies in the averages for these; they should be rerun at some point.)
| Model | trn QAcc | val QAcc | val QF1 | tst QAcc | tst QF1 | settings |
|---|---|---|---|---|---|---|
| avg | 0.626815 | 0.670659 | nan | 0.621308 | nan | (defaults) |
| ±0.024750 | ±0.020524 | ±nan | ±0.026126 | ±nan | ||
| DAN | 0.913809 | 0.848303 | 0.787701 | 0.799578 | 0.754505 |
inp_e_dropout=0 inp_w_dropout=1/3 deep=2 pact='relu' l2reg=1e-5
|
| ±0.020632 | ±0.012912 | ±0.028049 | ±0.019803 | ±0.029174 | ||
| -------------------------- | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- |
| rnn | 0.930491 | 0.859281 | 0.793455 | 0.806962 | 0.763644 | (defaults) |
| ±0.044004 | ±0.007907 | ±0.009231 | ±0.020562 | ±0.030651 | ||
| cnn | 0.920297 | 0.862275 | 0.801795 | 0.819620 | 0.763181 | (defaults) |
| ±0.030030 | ±0.017017 | ±0.033116 | ±0.024479 | ±0.061008 | ||
| rnncnn | 0.922768 | 0.861277 | 0.804602 | 0.812236 | 0.765567 | (defaults) |
| ±0.040065 | ±0.009881 | ±0.017815 | ±0.014686 | ±0.025850 | ||
| attn1511 | 0.869632 | 0.841317 | 0.787862 | 0.812236 | 0.777503 | (defaults) |
| ±0.011013 | ±0.009426 | ±0.015477 | ±0.009129 | ±0.022301 |
(val QAcc is used for further tuning)
In attn1511, we previously considered sdim=2...
6x R_rg_2a51BV_EP100_s2 - 0.840319 (95% [0.835635, 0.845003]):
11289052.arien.ics.muni.cz.R_rg_2a51BV_EP100_s2 etc.
[0.844311, 0.838323, 0.844311, 0.832335, 0.838323, 0.844311, ]
The "After" version had masking disabled, after we repaired it:
6x R_rg_2avgBV_EP100_mask - 0.757485 (95% [0.707640, 0.807330]):
11290079.arien.ics.muni.cz.R_rg_2avgBV_EP100_mask etc.
[0.808383, 0.712575, 0.790419, 0.712575, 0.706587, 0.814371, ]
6x R_rg_2rnnBV_EP100_mask - 0.860279 (95% [0.855595, 0.864963]):
11290081.arien.ics.muni.cz.R_rg_2rnnBV_EP100_mask etc.
[0.856287, 0.862275, 0.856287, 0.868263, 0.862275, 0.856287, ]
6x R_rg_2cnnBV_EP100_mask - 0.852295 (95% [0.844457, 0.860133]):
11290082.arien.ics.muni.cz.R_rg_2cnnBV_EP100_mask etc.
[0.856287, 0.862275, 0.838323, 0.850299, 0.850299, 0.856287, ]
6x R_rg_2rnncnnBV_EP100_mask - 0.857285 (95% [0.839728, 0.874842]):
11290083.arien.ics.muni.cz.R_rg_2rnncnnBV_EP100_mask etc.
[0.850299, 0.832335, 0.856287, 0.868263, 0.886228, 0.850299, ]
6x R_rg_2a51BV_EP100_mask - 0.836327 (95% [0.827690, 0.844964]):
11290084.arien.ics.muni.cz.R_rg_2a51BV_EP100_mask etc.
[0.832335, 0.844311, 0.838323, 0.844311, 0.838323, 0.820359, ]
So it's important for avg performance, otherwise makes little discernable difference.
16x R_rg_2avgBV_EP100_mask_L1e-5 - 0.816242 (95% [0.808449, 0.824035]):
11290437.arien.ics.muni.cz.R_rg_2avgBV_EP100_mask_L1e-5 etc.
[0.820359, 0.790419, 0.838323, 0.826347, 0.832335, 0.820359, 0.808383, 0.784431, 0.814371, 0.808383, 0.832335, 0.826347, 0.802395, 0.820359, 0.826347, 0.808383, ]
16x R_rg_2danBV_EP100_mask_L1e-5_W13 - 0.821856 (95% [0.810832, 0.832880]):
11298376.arien.ics.muni.cz.R_rg_2danBV_EP100_mask_L1e-5_W13 etc.
[0.838323, 0.832335, 0.784431, 0.808383, 0.814371, 0.790419, 0.802395, 0.850299, 0.832335, 0.844311, 0.808383, 0.850299, 0.832335, 0.796407, 0.838323, 0.826347, ]
16x R_rg_2rnnBV_EP100_mask_L1e-4 - 0.846557 (95% [0.839706, 0.853407]):
11290087.arien.ics.muni.cz.R_rg_2rnnBV_EP100_mask_L1e-4 etc.
[0.844311, 0.850299, 0.850299, 0.844311, 0.844311, 0.862275, 0.832335, 0.868263, 0.838323, 0.856287, 0.850299, 0.832335, 0.814371, 0.862275, 0.850299, 0.844311, ]
6x R_rg_2cnnBV_EP100_mask_L1e-4 - 0.852295 (95% [0.838401, 0.866189]):
11290088.arien.ics.muni.cz.R_rg_2cnnBV_EP100_mask_L1e-4 etc.
[0.844311, 0.874251, 0.862275, 0.832335, 0.850299, 0.850299, ]
16x R_rg_2rnncnnBV_EP100_mask_L1e-4 - 0.861152 (95% [0.855601, 0.866704]):
11290089.arien.ics.muni.cz.R_rg_2rnncnnBV_EP100_mask_L1e-4 etc.
[0.856287, 0.850299, 0.856287, 0.874251, 0.856287, 0.868263, 0.880240, 0.874251, 0.850299, 0.856287, 0.868263, 0.856287, 0.862275, 0.868263, 0.862275, 0.838323, ]
16x R_rg_2a51BV_EP100_mask_L1e-4 - 0.851796 (95% [0.844186, 0.859406]):
11290090.arien.ics.muni.cz.R_rg_2a51BV_EP100_mask_L1e-4 etc.
[0.874251, 0.820359, 0.862275, 0.850299, 0.838323, 0.868263, 0.844311, 0.862275, 0.862275, 0.832335, 0.838323, 0.850299, 0.844311, 0.856287, 0.856287, 0.868263, ]
While there are no individual statistically significant results, it seems like moving back to the default l2reg of 1e-4 is not harmful at all anywhere (maybe RNN is a wee bit questionable) and that'll be better for consistency.
6x R_rg_2a51BV_EP100_mask - 0.836327 (95% [0.827690, 0.844964]):
6x R_rg_2a51BV_EP100_mask_1 - 0.675649 (95% [0.655640, 0.695658]):
11290441.arien.ics.muni.cz.R_rg_2a51BV_EP100_mask_1 etc.
[0.676647, 0.676647, 0.706587, 0.652695, 0.652695, 0.688623, ]
Oops, nada.
6x R_rg_2cnnBV_EP100_mask_L1e-4 - 0.852295 (95% [0.838401, 0.866189]):
11290088.arien.ics.muni.cz.R_rg_2cnnBV_EP100_mask_L1e-4 etc.
[0.844311, 0.874251, 0.862275, 0.832335, 0.850299, 0.850299, ]
9x R_rg_2cnnBV_EP100_mask_L1e-4_c121212 - 0.844976 (95% [0.832428, 0.857524]):
11298418.arien.ics.muni.cz.R_rg_2cnnBV_EP100_mask_L1e-4_c121212 etc.
[0.820359, 0.820359, 0.838323, 0.868263, 0.856287, 0.862275, 0.844311, 0.856287, 0.838323, ]
12x R_rg_2cnnSBV_EP100_mask_L1e-4 - 0.842315 (95% [0.826757, 0.857873]):
11298430.arien.ics.muni.cz.R_rg_2cnnSBV_EP100_mask_L1e-4 etc.
[0.814371, 0.808383, 0.814371, 0.808383, 0.856287, 0.856287, 0.838323, 0.862275, 0.838323, 0.862275, 0.880240, 0.868263, ]
Nah.
16x R_rg_2rnncnnBV_EP100_mask_L1e-4 - 0.861152 (95% [0.855601, 0.866704]):
8x R_rg_2rnncnnBV_EP100_mask_L1e-5 - 0.860778 (95% [0.852571, 0.868985]):
11304711.arien.ics.muni.cz.R_rg_2rnncnnBV_EP100_mask_L1e-5 etc.
[0.862275, 0.856287, 0.862275, 0.862275, 0.838323, 0.862275, 0.868263, 0.874251, ]
8x R_rg_2rnncnnSBV_EP100_mask_L1e-5 - 0.872754 (95% [0.867907, 0.877602]):
11304718.arien.ics.muni.cz.R_rg_2rnncnnSBV_EP100_mask_L1e-5 etc.
[0.874251, 0.880240, 0.880240, 0.868263, 0.874251, 0.874251, 0.868263, 0.862275, ]
8x R_rg_2rnncnnBV_EP100_mask_L1e-5_c121212 - 0.865269 (95% [0.855903, 0.874635]):
11304719.arien.ics.muni.cz.R_rg_2rnncnnBV_EP100_mask_L1e-5_c121212 etc.
[0.850299, 0.862275, 0.862275, 0.880240, 0.856287, 0.862275, 0.862275, 0.886228, ]
Nothing convincing. Siameseness questionably beneficial - we won't invest in a change anymore.
16x R_rg_2rnnBV_EP100_mask_L1e-4 - 0.846557 (95% [0.839706, 0.853407]):
8x R_rg_2rnnBV_EP100_mask_L1e-4_i13d13 - 0.871257 (95% [0.863342, 0.879173]):
11304810.arien.ics.muni.cz.R_rg_2rnnBV_EP100_mask_L1e-4_i13d13 etc.
[0.862275, 0.874251, 0.874251, 0.880240, 0.862275, 0.886228, 0.856287, 0.874251, ]
Awesome.
6x R_rg_2cnnBV_EP100_mask_L1e-4 - 0.852295 (95% [0.838401, 0.866189]):
8x R_rg_2cnnBV_EP100_mask_L1e-4_i13d13 - 0.861527 (95% [0.857619, 0.865434]):
11309618.arien.ics.muni.cz.R_rg_2cnnBV_EP100_mask_L1e-4_i13d13 etc.
[0.856287, 0.856287, 0.868263, 0.862275, 0.868263, 0.856287, 0.862275, 0.862275, ]
16x R_rg_2rnncnnBV_EP100_mask_L1e-4 - 0.861152 (95% [0.855601, 0.866704]):
8x R_rg_2rnncnnBV_EP100_mask_L1e-4_i13d13 - 0.860030 (95% [0.849137, 0.870922]):
11309632.arien.ics.muni.cz.R_rg_2rnncnnBV_EP100_mask_L1e-4_i13d13 etc.
[0.838323, 0.880240, 0.868263, 0.862275, 0.850299, 0.856287, 0.874251, 0.850299, ]
Inconclusive.
16x R_rg_2a51BV_EP100_mask_L1e-4 - 0.851796 (95% [0.844186, 0.859406]):
3x R_rg_2a51BV_EP100_mask_L1e-4_fasgmmn_crelu - 0.884232 (95% [0.853665, 0.914798]):
x.R_rg_2a51BV_EP100_mask_L1e-4_fasgmmn_crelu etc.
[0.898204, 0.886228, 0.868263, ]
3x R_rg_2a51BV_EP100_mask_L1e-4_fasgmmn - 0.870259 (95% [0.839693, 0.900826]):
x.R_rg_2a51BV_EP100_mask_L1e-4_fasgmmn etc.
[0.868263, 0.886228, 0.856287, ]
3x R_rg_2a51BV_EP100_mask_L1e-4_cl4_crelu - 0.856287 (95% [0.844142, 0.868432]):
x.R_rg_2a51BV_EP100_mask_L1e-4_cl4_crelu etc.
[0.856287, 0.850299, 0.862275, ]
3x R_rg_2a51BV_EP100_mask_L1e-4_fasgmmn_cl4_crelu - 0.880240 (95% [0.868093, 0.892386]):
x.R_rg_2a51BV_EP100_mask_L1e-4_fasgmmn_cl4_crelu etc.
[0.886228, 0.880240, 0.874251, ]
cnnact='relu' (maybe) and focus_act='sigmoid/maxnorm' are important; clen=4 not so much.
15x R_rg_2a51BV_EP100_mask_L1e-5_fasgmn_crelu - 0.877046 (95% [0.869118, 0.884974]):
11304952.arien.ics.muni.cz.R_rg_2a51BV_EP100_mask_L1e-5_fasgmn_crelu etc.
[0.880240, 0.892216, 0.898204, 0.868263, 0.844311, 0.880240, 0.886228, 0.850299, 0.886228, 0.874251, 0.892216, 0.874251, 0.880240, 0.880240, 0.868263, ]
16x R_urg11299592rnnBV_EP100_mask_rmsprop_mlp - 0.912426 (95% [0.908075, 0.916776]):
11305159.arien.ics.muni.cz.R_urg11299592rnnBV_EP100_mask_rmsprop_mlp etc.
[0.904192, 0.904192, 0.916168, 0.910180, 0.922156, 0.910180, 0.904192, 0.928144, 0.910180, 0.910180, 0.916168, 0.904192, 0.922156, 0.898204, 0.916168, 0.922156, ]
16x R_urg11299592rnnBV_EP100_mask_rmsprop_dot - 0.904940 (95% [0.896067, 0.913814]):
11305160.arien.ics.muni.cz.R_urg11299592rnnBV_EP100_mask_rmsprop_dot etc.
[0.904192, 0.916168, 0.880240, 0.910180, 0.904192, 0.880240, 0.910180, 0.874251, 0.898204, 0.928144, 0.898204, 0.928144, 0.928144, 0.898204, 0.922156, 0.898204, ]
MLP wins.
Try attn1511 model based on rnn pretraining...
8x R_urga51_11299592rnnBV_EP100_mask_rmsprop_mlp - 0.903443 (95% [0.898803, 0.908084]):
11310546.arien.ics.muni.cz.R_urga51_11299592rnnBV_EP100_mask_rmsprop_mlp etc.
[0.910180, 0.898204, 0.904192, 0.892216, 0.910180, 0.904192, 0.904192, 0.904192, ]
Nah.
3x R_rg_2a51BV_EP100_mask_L1e-4_fasgmmn_crelu - 0.884232 (95% [0.853665, 0.914798]):
3x R_rg_2a51BV_EP100_mask_L1e-4_fasgmmn_crelu_RF - 0.872255 (95% [0.817488, 0.927023]):
x.R_rg_2a51BV_EP100_mask_L1e-4_fasgmmn_crelu_RF etc.
[0.874251, 0.844311, 0.898204, ]
16x R_rg_2avg - 0.797530 (95% [0.790835, 0.804225]):
11173959.arien.ics.muni.cz.R_rg_2avg etc.
[0.814371, 0.796407, 0.802395, 0.790419, 0.778443, 0.802395, 0.772455, 0.790419, 0.790419, 0.802395, 0.808383, 0.814371, 0.808383, 0.802395, 0.808383, 0.778443, ]
16x R_rg_2dan_L1e-5 - 0.827095 (95% [0.811798, 0.842393]):
11173973.arien.ics.muni.cz.R_rg_2dan_L1e-5 etc.
[0.796407, 0.820359, 0.808383, 0.874251, 0.814371, 0.820359, 0.874251, 0.814371, 0.856287, 0.838323, 0.802395, 0.862275, 0.820359, 0.856287, 0.796407, 0.778443, ]
16x R_rg_2rnn - 0.854416 (95% [0.845340, 0.863491]):
11173961.arien.ics.muni.cz.R_rg_2rnn etc.
[0.880240, 0.862275, 0.838323, 0.886228, 0.832335, 0.868263, 0.838323, 0.856287, 0.820359, 0.850299, 0.862275, 0.850299, 0.850299, 0.874251, 0.850299, 0.850299, ]
16x R_rg_2cnn - 0.857410 (95% [0.852213, 0.862606]):
11173962.arien.ics.muni.cz.R_rg_2cnn etc.
[0.844311, 0.856287, 0.856287, 0.862275, 0.838323, 0.868263, 0.874251, 0.850299, 0.844311, 0.856287, 0.862275, 0.862275, 0.868263, 0.868263, 0.856287, 0.850299, ]
16x R_rg_2rnncnn - 0.852170 (95% [0.842550, 0.861790]):
11173963.arien.ics.muni.cz.R_rg_2rnncnn etc.
[0.850299, 0.856287, 0.838323, 0.874251, 0.862275, 0.868263, 0.832335, 0.832335, 0.832335, 0.874251, 0.874251, 0.814371, 0.844311, 0.856287, 0.874251, 0.850299, ]
16x R_rg_2a51 - 0.834206 (95% [0.824653, 0.843760]):
11173964.arien.ics.muni.cz.R_rg_2a51 etc.
[0.838323, 0.850299, 0.850299, 0.832335, 0.844311, 0.856287, 0.838323, 0.850299, 0.808383, 0.814371, 0.820359, 0.826347, 0.802395, 0.856287, 0.808383, 0.850299, ]
4x R_rg_2dan - 0.694611 (95% [0.579087, 0.810135]):
4x R_rg_2dan_L1e-5 - 0.824850 (95% [0.777507, 0.872193]):
11173973.arien.ics.muni.cz.R_rg_2dan_L1e-5 etc.
[0.796407, 0.820359, 0.808383, 0.874251, ]
4x R_rg_2rnn - 0.866766 (95% [0.837109, 0.896424]):
4x R_rg_2rnn_L1e-5 - 0.875749 (95% [0.857451, 0.894046]):
11173974.arien.ics.muni.cz.R_rg_2rnn_L1e-5 etc.
[0.886228, 0.880240, 0.856287, 0.880240, ]
16x R_rg_2rnn_D0.5s2L1e-5 - 0.866766 (95% [0.857567, 0.875966]):
11206383.arien.ics.muni.cz.R_rg_2rnn_D0.5s2L1e-5 etc.
[0.832335, 0.880240, 0.856287, 0.868263, 0.892216, 0.862275, 0.850299, 0.898204, 0.874251, 0.874251, 0.868263, 0.844311, 0.868263, 0.874251, 0.880240, 0.844311, ]
8x R_rg_2rnn_S25 - 0.854042 (95% [0.839896, 0.868187]):
11215661.arien.ics.muni.cz.R_rg_2rnn_S25 etc.
[0.868263, 0.844311, 0.880240, 0.868263, 0.844311, 0.832335, 0.832335, 0.862275, ]
8x R_rg_2rnn_s1S25 - 0.850299 (95% [0.838559, 0.862039]):
11215960.arien.ics.muni.cz.R_rg_2rnn_s1S25 etc.
[0.856287, 0.868263, 0.862275, 0.850299, 0.856287, 0.850299, 0.820359, 0.838323, ]
8x R_rg_2rnn_s1E300S25 - 0.828593 (95% [0.770215, 0.886970]):
11215962.arien.ics.muni.cz.R_rg_2rnn_s1E300S25 etc.
[0.868263, 0.838323, 0.868263, 0.874251, 0.652695, 0.808383, 0.844311, 0.874251, ]
16x R_rg_2a51 - 0.834206 (95% [0.824653, 0.843760]):
16x R_rg_2a51_a1 - 0.832709 (95% [0.827016, 0.838402]):
11192762.arien.ics.muni.cz.R_rg_2a51_a1 etc.
[0.850299, 0.826347, 0.844311, 0.838323, 0.838323, 0.844311, 0.814371, 0.832335, 0.838323, 0.832335, 0.808383, 0.832335, 0.838323, 0.832335, 0.832335, 0.820359, ]
16x R_rg_2a51_p1 - 0.819985 (95% [0.808962, 0.831007]):
11192763.arien.ics.muni.cz.R_rg_2a51_p1 etc.
[0.790419, 0.808383, 0.808383, 0.826347, 0.820359, 0.820359, 0.808383, 0.844311, 0.856287, 0.802395, 0.814371, 0.814371, 0.856287, 0.790419, 0.850299, 0.808383, ]
16x R_rg_2a51_p1D0.5 - 0.833084 (95% [0.825697, 0.840470]):
11192767.arien.ics.muni.cz.R_rg_2a51_p1D0.5 etc.
[0.808383, 0.820359, 0.838323, 0.844311, 0.826347, 0.850299, 0.838323, 0.832335, 0.844311, 0.838323, 0.802395, 0.838323, 0.826347, 0.838323, 0.826347, 0.856287, ]
16x R_rg_2a51_D0.5 - 0.825224 (95% [0.817310, 0.833139]):
11200187.arien.ics.muni.cz.R_rg_2a51_D0.5 etc.
[0.796407, 0.820359, 0.820359, 0.814371, 0.820359, 0.832335, 0.814371, 0.838323, 0.838323, 0.838323, 0.814371, 0.826347, 0.826347, 0.808383, 0.862275, 0.832335, ]
16x R_rg_2a51_D0.5s2 - 0.848428 (95% [0.843547, 0.853309]):
11200190.arien.ics.muni.cz.R_rg_2a51_D0.5s2 etc.
[0.856287, 0.850299, 0.850299, 0.838323, 0.850299, 0.850299, 0.850299, 0.844311, 0.856287, 0.826347, 0.862275, 0.850299, 0.856287, 0.856287, 0.844311, 0.832335, ]
16x R_rg_2a51_D0.5s2L1e-5 - 0.849550 (95% [0.845200, 0.853901]):
11206381.arien.ics.muni.cz.R_rg_2a51_D0.5s2L1e-5 etc.
[0.838323, 0.868263, 0.850299, 0.850299, 0.856287, 0.856287, 0.844311, 0.850299, 0.838323, 0.850299, 0.850299, 0.838323, 0.844311, 0.850299, 0.844311, 0.862275, ]
16x R_rg_2a51_s2 - 0.842066 (95% [0.835309, 0.848822]):
11215612.arien.ics.muni.cz.R_rg_2a51_s2 etc.
[0.868263, 0.832335, 0.850299, 0.820359, 0.844311, 0.838323, 0.844311, 0.868263, 0.844311, 0.838323, 0.832335, 0.844311, 0.832335, 0.826347, 0.850299, 0.838323, ]
16x R_rg_2cnn - 0.857410 (95% [0.852213, 0.862606]):
16x R_rg_2cnnS - 0.842440 (95% [0.836500, 0.848379]):
11235913.arien.ics.muni.cz.R_rg_2cnnS etc.
[0.838323, 0.832335, 0.850299, 0.820359, 0.832335, 0.844311, 0.838323, 0.856287, 0.844311, 0.856287, 0.844311, 0.832335, 0.844311, 0.838323, 0.868263, 0.838323, ]
16x R_rg_2rnn - 0.854416 (95% [0.845340, 0.863491]):
We retrained the Ubuntu Dialogue model with embdim=50 to be compatible. As before, it's pdim=1, d0. The 11226219 is also already trained to classify as MLP with Ddim=0.
4x R_urg11226212rnn_mlp - 0.889222 (95% [0.874930, 0.903514]):
11240005.arien.ics.muni.cz.R_urg11226212rnn_mlp etc.
[0.880240, 0.904192, 0.886228, 0.886228, ]
16x R_urg11226212rnn_rmsprop_mlp - 0.894087 (95% [0.888710, 0.899464]):
11240004.arien.ics.muni.cz.R_urg11226212rnn_rmsprop_mlp etc.
[0.892216, 0.898204, 0.892216, 0.892216, 0.910180, 0.892216, 0.886228, 0.898204, 0.892216, 0.886228, 0.874251, 0.904192, 0.904192, 0.904192, 0.904192, 0.874251, ]
4x R_urg11226219rnn_mlp - 0.880240 (95% [0.862413, 0.898066]):
11240500.arien.ics.muni.cz.R_urg11226219rnn_mlp etc.
[0.898204, 0.874251, 0.880240, 0.868263, ]
4x R_urg11226219rnn_rmsprop_mlp - 0.881737 (95% [0.873836, 0.889638]):
11240498.arien.ics.muni.cz.R_urg11226219rnn_rmsprop_mlp etc.
[0.886228, 0.880240, 0.886228, 0.874251, ]