Help wanted with fresh colab run of bert_sngp on clinc

[Jordy-VL]

Hi,

I wanted to try out SNGP with the script as provided in baselines.
Current command with custom flags:

python3 uncertainty-baselines/baselines/clinc_intent/sngp.py --data_dir "/root/tensorflow_datasets/clinc_oos/0.1.0" --use_gpu --bert_dir "./bert3/cased_L-12_H-768_A-12"

I first had some issues to get CLINC_OOS data in the right, expected format, but managed to fix that.
Now I am stuck trying to get a BERT checkpoint loaded correctly with the flags provided.

It might help others as well to get a working "collab-fresh" example, so I am sharing my progress so far:
https://colab.research.google.com/drive/1HVMcZm7opre4FRaaoXRAocXMw4WSrhvD?usp=sharing

Can anyone provide short assistance to get a proper BERT checkpoint loaded?
I'm happy to share the final (cleaned) working example, once I can get it to run end-to-end on a fresh environment :)

Cheers,

Jordy

[jereliu]

Hey Jordy,

Here's a draft colab that may be helpful:

https://github.com/tensorflow/models/blob/master/official/colab/uncertainty_quantification_with_sngp_bert.ipynb

It is using the SNGP implementation from TF model garden, but effectively replicates the performance of BERT_SNGP.

[Jordy-VL]

Hey Jeremiah,

Thanks for sharing that resource, exactly what I needed!

In that same Colab I see a link to an SNGP tutorial, but the link gives me a 404. Would you mind sharing that one as well? :)

I am planning on testing SNGP on different architectures than BERT, so any added intuition would certainly help me.

[jereliu]

Hey Jordy,

Sure thing! The link to the basic tutorial is here:

https://www.tensorflow.org/tutorials/understanding/sngp

Sorry for the broken link...it should be fixed soon.

[dustinvtran]

For posterity, how about providing these links somewhere in the Uncertainty Baselines documentation? For example, in a uncertainty-baselines/baselines/notebooks/README.md.

[Jordy-VL]

Hi Jeremiah and Dustin,

Today, I finished implementing a wrapper SNGP class for simple tf.keras models, which I tested (in hardcoded fashion) with MNIST and a simple CNN model.

Just to share my experience, I could not properly train the SNGP if I did not put this argument: use_custom_random_features=True.

See my colab with running code here: https://colab.research.google.com/drive/1xcVPUEbFPgUoUmtA47N3jbwpJ-wGUncZ?usp=sharing

Would any of you mind having a look at the model wrapper and that it performs as expected from an SNGP model?
Thanks a ton! I based most of the code on the last two tutorials you shared, this was very helpful to get me started. :)

Cheers!

[jereliu]

Hey Jordy,

For image recognition type of tasks, I recommend look into the SNGP hyperparameter setting for, say, CIFAR, and the associated WideResNet-SNGP used in that example.

Yes, in general it is recommended to set use_custom_random_features=True, and for CNN-type models, we found the spectral norm should be set to an larger bound (e.g., norm_multiplier=6.).

Thanks!