v2.0.0

The v2.0.0 release adds unlabeled data to 8 datasets and several new algorithms for taking advantage of the unlabeled data. It also updates the standard data augmentations used for the image datasets.

All labeled (training, validation, and test) datasets are exactly the same. Evaluation metrics are also exactly the same. All results on the datasets in v1.x are therefore still current and directly comparable to results obtained on v2.

For more information, please read our paper on the unlabeled data.

New datasets with unlabeled data

We have added unlabeled data to the following datasets:

1. iwildcam
2. camelyon17
3. ogb-molpcba
4. globalwheat
5. civilcomments
6. fmow
7. poverty
8. amazon

The following datasets have no unlabeled data and have not been changed:

1. rxrx1
2. py150

The labeled training, validation, and test data in all datasets have been kept exactly the same.

The unlabeled data comes from the same underlying sources as the original labeled data, and they can be from the source, validation, extra, or target domains. We describe each dataset in detail in our paper.

Each unlabeled dataset has its own corresponding data loader defined in wilds/datasets/unlabeled. Please see the README for more details on how to use them.

New algorithms for using unlabeled data

In our scripts in the examples folder, we have updated and/or added new algorithms that make use of the unlabeled data:

1. CORAL (Sun and Saenko, 2016)
2. DANN (Ganin et al., 2016)
3. AFN (Xu et al., 2019)
4. Pseudo-Label (Lee, 2013)
5. FixMatch (Sohn et al., 2020)
6. Noisy Student (Xie et al., 2020)
7. SwAV pre-training (Caron et al., 2020)
8. Masked language model pre-training (Devlin et al., 2019)

Other changes

GlobalWheat v1.0 -> v1.1

We have corrected some errors in the metadata for the previous version of the GlobalWheat (labeled) dataset.
Users who did not explicitly make use of the location or stage metadata (which should be most users) will not be affected.
All baseline results are unchanged.

DomainNet support

We have included data loaders for the DomainNet dataset (Peng at al., 2019) as a means of benchmarking the algorithms we implemented on existing datasets.

Data augmentation

We have added support for RandAugment (Cubuk et al., 2019) for RGB images, and we have also implemented a set of data augmentations for the multi-spectral Poverty dataset. These augmentations are used in all of the algorithms for unlabeled data listed above.

Hyperparameters

In our experiments to benchmark the algorithms for using unlabeled data, we tuned hyperparameters by random search instead of grid search. The default hyperparameters in examples/configs/datasets.py still work well but do not reflect the exact hyperparameters we used for our experiments. To see those, please view our CodaLab worksheet.

Miscellaneous

• In our example scripts, we have added support for gradient accumulation by specifying the gradient_accumulation_steps parameter.
• We have also added support for logging using Weights and Biases.