Skip to content

Commit

Permalink
Update TFDS version to 3.2.0
Browse files Browse the repository at this point in the history 
API:

 * Add a `tfds.ImageFolder` and `tfds.TranslateFolder` to easily create custom datasets with your custom data.
 * Add a `tfds.ReadConfig(input_context=)` to shard dataset, for better multi-worker compatibility (#1426).
 * The default `data_dir` can be controlled by the `TFDS_DATA_DIR` environment variable.
 * Better usability when developing datasets outside TFDS
   * Downloads are always cached
   * Checksum are optional
 * Added a `tfds.show_statistics(ds_info)` to display [FACETS OVERVIEW](https://pair-code.github.io/facets/). Note: This require the dataset to have been generated with the statistics.
 * Open source various scripts to help deployment/documentation (Generate catalog documentation, export all metadata files,...)

Documentation:

 * Catalog display images ([example](https://www.tensorflow.org/datasets/catalog/sun397#sun397standard-part2-120k))
 * Catalog shows which dataset have been recently added and are only available in `tfds-nightly` <span class="material-icons">nights_stay</span>

Breaking compatibility change:

 * Fix deterministic example order on Windows when path was used as key (this only impact a few datasets). Now example order should be the same on all platforms.
 * Remove `tfds.load('image_label_folder')` in favor of the more user-friendly `tfds.ImageFolder`

Other:

 * Various performances improvements for both generation and reading (e.g. use `__slot__`, fix parallelisation bug in `tf.data.TFRecordReader`,...)
 * Various fixes (typo, types annotations, better error messages, fixing dead links, better windows compatibility,...)

PiperOrigin-RevId: 320672697
  • Loading branch information
@Conchylicultor @copybara-github
Conchylicultor authored and copybara-github committed Jul 10, 2020
1 parent 84ac202 commit 61ce68c
Show file tree
Hide file tree
Showing 3 changed files with 180 additions and 69 deletions.
245 changes: 178 additions & 67 deletions tensorflow_datasets/stable_versions.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ amazon_us_reviews/Video_Games_v1_00/0.1.0
amazon_us_reviews/Video_v1_00/0.1.0
amazon_us_reviews/Watches_v1_00/0.1.0
amazon_us_reviews/Wireless_v1_00/0.1.0
anli/r1/0.1.0
anli/r2/0.1.0
anli/r3/0.1.0
arc/2019-12-06/1.0.0
bair_robot_pushing_small/2.0.0
beans/0.1.0
Expand Down Expand Up @@ -161,18 +164,90 @@ celeb_a_hq/4/2.0.0
celeb_a_hq/512/2.0.0
celeb_a_hq/64/2.0.0
celeb_a_hq/8/2.0.0
cfq/mcd1/1.0.1
cfq/mcd2/1.0.1
cfq/mcd3/1.0.1
cfq/query_complexity_split/1.0.1
cfq/query_pattern_split/1.0.1
cfq/question_complexity_split/1.0.1
cfq/question_pattern_split/1.0.1
cfq/random_split/1.0.1
cfq/cd0.1_r1/1.2.0
cfq/cd0.1_r2/1.2.0
cfq/cd0.1_r3/1.2.0
cfq/cd0.1_r4/1.2.0
cfq/cd0.1_r5/1.2.0
cfq/cd0.1_r6/1.2.0
cfq/cd0.1_r7/1.2.0
cfq/cd0.1_r8/1.2.0
cfq/cd0.1_r9/1.2.0
cfq/cd0.2_r1/1.2.0
cfq/cd0.2_r2/1.2.0
cfq/cd0.2_r3/1.2.0
cfq/cd0.2_r4/1.2.0
cfq/cd0.2_r5/1.2.0
cfq/cd0.2_r6/1.2.0
cfq/cd0.2_r7/1.2.0
cfq/cd0.2_r8/1.2.0
cfq/cd0.2_r9/1.2.0
cfq/cd0.3_r1/1.2.0
cfq/cd0.3_r2/1.2.0
cfq/cd0.3_r3/1.2.0
cfq/cd0.3_r4/1.2.0
cfq/cd0.3_r5/1.2.0
cfq/cd0.3_r6/1.2.0
cfq/cd0.3_r7/1.2.0
cfq/cd0.3_r8/1.2.0
cfq/cd0.3_r9/1.2.0
cfq/cd0.4_r1/1.2.0
cfq/cd0.4_r2/1.2.0
cfq/cd0.4_r3/1.2.0
cfq/cd0.4_r4/1.2.0
cfq/cd0.4_r5/1.2.0
cfq/cd0.4_r6/1.2.0
cfq/cd0.4_r7/1.2.0
cfq/cd0.4_r8/1.2.0
cfq/cd0.4_r9/1.2.0
cfq/cd0.5_r1/1.2.0
cfq/cd0.5_r2/1.2.0
cfq/cd0.5_r3/1.2.0
cfq/cd0.5_r4/1.2.0
cfq/cd0.5_r5/1.2.0
cfq/cd0.5_r6/1.2.0
cfq/cd0.5_r7/1.2.0
cfq/cd0.5_r8/1.2.0
cfq/cd0.5_r9/1.2.0
cfq/cd0.6_r1/1.2.0
cfq/cd0.6_r2/1.2.0
cfq/cd0.6_r3/1.2.0
cfq/cd0.6_r4/1.2.0
cfq/cd0.6_r5/1.2.0
cfq/cd0.6_r6/1.2.0
cfq/cd0.6_r7/1.2.0
cfq/cd0.6_r8/1.2.0
cfq/cd0.6_r9/1.2.0
cfq/cd0_r1/1.2.0
cfq/cd0_r2/1.2.0
cfq/cd0_r3/1.2.0
cfq/cd0_r4/1.2.0
cfq/cd0_r5/1.2.0
cfq/cd0_r6/1.2.0
cfq/cd0_r7/1.2.0
cfq/cd0_r8/1.2.0
cfq/cd0_r9/1.2.0
cfq/cd1_r1/1.2.0
cfq/cd1_r2/1.2.0
cfq/cd1_r3/1.2.0
cfq/cd1_r4/1.2.0
cfq/cd1_r5/1.2.0
cfq/cd1_r6/1.2.0
cfq/cd1_r7/1.2.0
cfq/cd1_r8/1.2.0
cfq/cd1_r9/1.2.0
cfq/mcd1/1.2.0
cfq/mcd2/1.2.0
cfq/mcd3/1.2.0
cfq/query_complexity_split/1.2.0
cfq/query_pattern_split/1.2.0
cfq/question_complexity_split/1.2.0
cfq/question_pattern_split/1.2.0
cfq/random_split/1.2.0
cifar10/3.0.2
cifar100/3.0.2
cifar10_1/v4/1.0.0
cifar10_1/v6/1.0.0
cifar10_1/v4/1.1.0
cifar10_1/v6/1.1.0
cifar10_corrupted/brightness_1/1.0.0
cifar10_corrupted/brightness_2/1.0.0
cifar10_corrupted/brightness_3/1.0.0
Expand Down Expand Up @@ -269,8 +344,10 @@ cifar10_corrupted/zoom_blur_3/1.0.0
cifar10_corrupted/zoom_blur_4/1.0.0
cifar10_corrupted/zoom_blur_5/1.0.0
citrus_leaves/0.1.0
civil_comments/0.9.0
civil_comments/CivilComments/1.0.0
civil_comments/CivilCommentsIdentities/1.0.0
clevr/3.1.0
clinc_oos/0.1.0
cmaterdb/bangla/1.0.0
cmaterdb/devanagari/1.0.0
cmaterdb/telugu/1.0.0
Expand Down Expand Up @@ -302,6 +379,8 @@ common_voice/tr/1.0.0
common_voice/tt/1.0.0
common_voice/zh-TW/1.0.0
cos_e/0.0.1
cosmos_qa/1.0.0
covid19sum/1.0.0
crema_d/1.0.0
curated_breast_imaging_ddsm/original-calc/2.0.0
curated_breast_imaging_ddsm/original-mass/2.0.0
Expand Down Expand Up @@ -338,7 +417,7 @@ dmlab/2.0.0
downsampled_imagenet/32x32/2.0.0
downsampled_imagenet/64x64/2.0.0
dsprites/2.0.0
dtd/3.0.0
dtd/3.0.1
duke_ultrasound/1.0.0
emnist/balanced/3.0.0
emnist/byclass/3.0.0
Expand All @@ -357,6 +436,8 @@ flores/neen_plain_text/1.1.0
flores/sien_plain_text/1.1.0
food101/2.0.0
forest_fires/0.0.1
fuss/reverberant/1.2.0
fuss/unprocessed/1.2.0
gap/0.1.0
geirhos_conflict_stimuli/1.0.0
german_credit_numeric/1.0.0
Expand All @@ -381,7 +462,6 @@ groove/full-midionly/2.0.1
higgs/2.0.0
horses_or_humans/3.0.0
i_naturalist2017/0.1.0
image_label_folder/2.0.0
imagenet2012/5.0.0
imagenet2012_corrupted/brightness_1/3.1.0
imagenet2012_corrupted/brightness_2/3.1.0
Expand Down Expand Up @@ -478,6 +558,7 @@ imagenet2012_corrupted/zoom_blur_2/3.1.0
imagenet2012_corrupted/zoom_blur_3/3.1.0
imagenet2012_corrupted/zoom_blur_4/3.1.0
imagenet2012_corrupted/zoom_blur_5/3.1.0
imagenet2012_real/1.0.0
imagenet2012_subset/10pct/5.0.0
imagenet2012_subset/1pct/5.0.0
imagenet_resized/16x16/0.1.0
Expand All @@ -497,6 +578,7 @@ imdb_reviews/bytes/1.0.0
imdb_reviews/plain_text/1.0.0
imdb_reviews/subwords32k/1.0.0
imdb_reviews/subwords8k/1.0.0
irc_disentanglement/2.0.0
iris/2.0.0
kitti/3.2.0
kmnist/3.0.1
Expand Down Expand Up @@ -582,6 +664,7 @@ math_dataset/polynomials__expand/1.0.0
math_dataset/polynomials__simplify_power/1.0.0
math_dataset/probability__swr_p_level_set/1.0.0
math_dataset/probability__swr_p_sequence/1.0.0
mctaco/1.0.0
mnist/3.0.1
mnist_corrupted/brightness/1.0.0
mnist_corrupted/canny_edges/1.0.0
Expand All @@ -599,6 +682,16 @@ mnist_corrupted/spatter/1.0.0
mnist_corrupted/stripe/1.0.0
mnist_corrupted/translate/1.0.0
mnist_corrupted/zigzag/1.0.0
movie_lens/100k-movies/0.1.0
movie_lens/100k-ratings/0.1.0
movie_lens/1m-movies/0.1.0
movie_lens/1m-ratings/0.1.0
movie_lens/20m-movies/0.1.0
movie_lens/20m-ratings/0.1.0
movie_lens/25m-movies/0.1.0
movie_lens/25m-ratings/0.1.0
movie_lens/latest-small-movies/0.1.0
movie_lens/latest-small-ratings/0.1.0
movie_rationales/0.1.0
moving_mnist/1.0.0
multi_news/1.0.0
Expand All @@ -609,15 +702,24 @@ newsroom/1.0.0
nsynth/full/2.3.2
nsynth/gansynth_subset.f0_and_loudness/2.3.2
nsynth/gansynth_subset/2.3.2
nyu_depth_v2/0.0.1
omniglot/3.0.0
open_images_challenge2019_detection/200k/1.0.0
open_images_challenge2019_detection/300k/1.0.0
open_images_v4/200k/2.0.0
open_images_v4/300k/2.0.0
open_images_v4/original/2.0.0
openbookqa/0.1.0
opinion_abstracts/idebate/1.0.0
opinion_abstracts/rotten_tomatoes/1.0.0
opinosis/1.0.0
oxford_flowers102/2.1.0
oxford_iiit_pet/3.1.0
opus/IT/0.1.0
opus/koran/0.1.0
opus/law/0.1.0
opus/medical/0.1.0
opus/subtitles/0.1.0
oxford_flowers102/2.1.1
oxford_iiit_pet/3.2.0
para_crawl/enbg_plain_text/1.0.0
para_crawl/encs_plain_text/1.0.0
para_crawl/enda_plain_text/1.0.0
Expand All @@ -643,6 +745,7 @@ para_crawl/ensl_plain_text/1.0.0
para_crawl/ensv_plain_text/1.0.0
patch_camelyon/2.0.0
pet_finder/1.0.0
pg19/0.1.1
places365_small/2.0.0
plant_leaves/0.1.0
plant_village/1.0.0
Expand All @@ -669,6 +772,7 @@ qa4mre/2013.main.ES/0.1.0
qa4mre/2013.main.RO/0.1.0
quickdraw_bitmap/3.0.0
reddit/1.0.0
reddit_disentanglement/2.0.0
reddit_tifu/long/1.1.0
reddit_tifu/short/1.1.0
resisc45/3.0.0
Expand Down Expand Up @@ -758,7 +862,7 @@ ted_multi_translate/plain_text/1.0.0
tedlium/release1/1.0.1
tedlium/release2/1.0.1
tedlium/release3/1.0.1
tf_flowers/3.0.0
tf_flowers/3.0.1
the300w_lp/1.0.0
tiny_shakespeare/1.0.0
titanic/2.0.0
Expand All @@ -771,64 +875,67 @@ ucf101/ucf101_1/2.0.0
ucf101/ucf101_1_256/2.0.0
ucf101/ucf101_2/2.0.0
ucf101/ucf101_3/2.0.0
vctk/mic1/1.0.0
vctk/mic2/1.0.0
vgg_face2/1.0.0
visual_domain_decathlon/aircraft/1.1.0
visual_domain_decathlon/cifar100/1.1.0
visual_domain_decathlon/daimlerpedcls/1.1.0
visual_domain_decathlon/dtd/1.1.0
visual_domain_decathlon/gtsrb/1.1.0
visual_domain_decathlon/imagenet12/1.1.0
visual_domain_decathlon/omniglot/1.1.0
visual_domain_decathlon/svhn/1.1.0
visual_domain_decathlon/ucf101/1.1.0
visual_domain_decathlon/vgg-flowers/1.1.0
visual_domain_decathlon/aircraft/1.2.0
visual_domain_decathlon/cifar100/1.2.0
visual_domain_decathlon/daimlerpedcls/1.2.0
visual_domain_decathlon/dtd/1.2.0
visual_domain_decathlon/gtsrb/1.2.0
visual_domain_decathlon/imagenet12/1.2.0
visual_domain_decathlon/omniglot/1.2.0
visual_domain_decathlon/svhn/1.2.0
visual_domain_decathlon/ucf101/1.2.0
visual_domain_decathlon/vgg-flowers/1.2.0
voc/2007/4.0.0
voc/2012/4.0.0
voxceleb/1.1.1
voxforge/1.0.0
waymo_open_dataset/0.1.0
web_questions/1.0.0
wider_face/0.1.0
wiki40b/Wiki40B.ar/1.1.0
wiki40b/Wiki40B.bg/1.1.0
wiki40b/Wiki40B.ca/1.1.0
wiki40b/Wiki40B.cs/1.1.0
wiki40b/Wiki40B.da/1.1.0
wiki40b/Wiki40B.de/1.1.0
wiki40b/Wiki40B.el/1.1.0
wiki40b/Wiki40B.en/1.1.0
wiki40b/Wiki40B.es/1.1.0
wiki40b/Wiki40B.et/1.1.0
wiki40b/Wiki40B.fa/1.1.0
wiki40b/Wiki40B.fi/1.1.0
wiki40b/Wiki40B.fr/1.1.0
wiki40b/Wiki40B.he/1.1.0
wiki40b/Wiki40B.hi/1.1.0
wiki40b/Wiki40B.hr/1.1.0
wiki40b/Wiki40B.hu/1.1.0
wiki40b/Wiki40B.id/1.1.0
wiki40b/Wiki40B.it/1.1.0
wiki40b/Wiki40B.ja/1.1.0
wiki40b/Wiki40B.ko/1.1.0
wiki40b/Wiki40B.lt/1.1.0
wiki40b/Wiki40B.lv/1.1.0
wiki40b/Wiki40B.ms/1.1.0
wiki40b/Wiki40B.nl/1.1.0
wiki40b/Wiki40B.no/1.1.0
wiki40b/Wiki40B.pl/1.1.0
wiki40b/Wiki40B.pt/1.1.0
wiki40b/Wiki40B.ro/1.1.0
wiki40b/Wiki40B.ru/1.1.0
wiki40b/Wiki40B.sk/1.1.0
wiki40b/Wiki40B.sl/1.1.0
wiki40b/Wiki40B.sr/1.1.0
wiki40b/Wiki40B.sv/1.1.0
wiki40b/Wiki40B.th/1.1.0
wiki40b/Wiki40B.tl/1.1.0
wiki40b/Wiki40B.tr/1.1.0
wiki40b/Wiki40B.uk/1.1.0
wiki40b/Wiki40B.vi/1.1.0
wiki40b/Wiki40B.zh-cn/1.1.0
wiki40b/Wiki40B.zh-tw/1.1.0
wiki40b/ar/1.3.0
wiki40b/bg/1.3.0
wiki40b/ca/1.3.0
wiki40b/cs/1.3.0
wiki40b/da/1.3.0
wiki40b/de/1.3.0
wiki40b/el/1.3.0
wiki40b/en/1.3.0
wiki40b/es/1.3.0
wiki40b/et/1.3.0
wiki40b/fa/1.3.0
wiki40b/fi/1.3.0
wiki40b/fr/1.3.0
wiki40b/he/1.3.0
wiki40b/hi/1.3.0
wiki40b/hr/1.3.0
wiki40b/hu/1.3.0
wiki40b/id/1.3.0
wiki40b/it/1.3.0
wiki40b/ja/1.3.0
wiki40b/ko/1.3.0
wiki40b/lt/1.3.0
wiki40b/lv/1.3.0
wiki40b/ms/1.3.0
wiki40b/nl/1.3.0
wiki40b/no/1.3.0
wiki40b/pl/1.3.0
wiki40b/pt/1.3.0
wiki40b/ro/1.3.0
wiki40b/ru/1.3.0
wiki40b/sk/1.3.0
wiki40b/sl/1.3.0
wiki40b/sr/1.3.0
wiki40b/sv/1.3.0
wiki40b/th/1.3.0
wiki40b/tl/1.3.0
wiki40b/tr/1.3.0
wiki40b/uk/1.3.0
wiki40b/vi/1.3.0
wiki40b/zh-cn/1.3.0
wiki40b/zh-tw/1.3.0
wikihow/all/1.2.0
wikihow/sep/1.2.0
wikipedia/20190301.aa/1.0.0
Expand Down Expand Up @@ -1431,6 +1538,8 @@ wikipedia/20200301.zh-min-nan/1.0.0
wikipedia/20200301.zh-yue/1.0.0
wikipedia/20200301.zh/1.0.0
wikipedia/20200301.zu/1.0.0
wikipedia_toxicity_subtypes/0.2.0
winogrande/1.1.0
wmt14_translate/cs-en/1.0.0
wmt14_translate/de-en/1.0.0
wmt14_translate/fr-en/1.0.0
Expand Down Expand Up @@ -1477,6 +1586,8 @@ wmt19_translate/lt-en/1.0.0
wmt19_translate/ru-en/1.0.0
wmt19_translate/zh-en/1.0.0
wmt_t2t_translate/de-en/1.0.0
wordnet/WN18/0.1.0
wordnet/WN18RR/0.1.0
xnli/plain_text/1.0.0
xsum/1.1.0
yelp_polarity_reviews/bytes/0.1.0
Expand Down
2 changes: 1 addition & 1 deletion tensorflow_datasets/version.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@

# We follow Semantic Versioning (https://semver.org/)
_MAJOR_VERSION = '3'
_MINOR_VERSION = '1'
_MINOR_VERSION = '2'
_PATCH_VERSION = '0'

__version__ = '.'.join([
Expand Down
Loading

0 comments on commit 61ce68c

Please sign in to comment.