I have problem with creating tfrecord file #6

Zhangsihao-Yang · 2018-08-25T03:07:53Z

I have received an assert error when I try to create the tfrecord. Could you help?
I could provide more details.

metatl · 2018-11-15T01:25:30Z

Hi,
I have a related question. Is there any instruction to create tfrecords formatted datasets from our own data, i.e. non-standard data? I didn't see it in the README.

Thanks!

Iris6713 · 2018-11-30T07:31:08Z

I have same question about creating our own data.

skjerns · 2018-12-14T19:28:18Z

you can do so by python dataset_tools create_from_images target_dir source_dir, whre source_dir contains the images that you want to add

liminalitythree · 2018-12-26T02:48:18Z

@skjerns

When I do this I get this error

Loading images from "../imgset"
Creating dataset "imagedataset"
Added 0 images.
Traceback (most recent call last):
  File "dataset_tool.py", line 738, in <module>
    execute_cmdline(sys.argv)
  File "dataset_tool.py", line 733, in execute_cmdline
    func(**vars(args))
  File "dataset_tool.py", line 621, in create_from_images
    img = img.transpose(2, 0, 1) # HWC => CHW
ValueError: axes don't match array

skjerns · 2019-01-02T12:20:22Z

what is the exact command you are running?
which format are your images in?
which size are they, are they all the same size?

try using the absolute path

amrbenattia · 2019-03-05T09:00:37Z

Still have AssertionError while using dataset_tool.py for Cifar10 dataset. any one could help?

Traceback (most recent call last):
File "dataset_tool.py", line 740, in
execute_cmdline(sys.argv)
File "dataset_tool.py", line 735, in execute_cmdline
func(**vars(args))
File "dataset_tool.py", line 341, in create_cifar10
assert labels.shape == (50000,) and labels.dtype == np.int32
AssertionError

nandaw · 2019-03-22T16:35:31Z

Edit: I managed to solve myself. In my case some of my images were sRGB instead of RGB

Did someone managed to solve the "Assertion Error" issue? I'm having the same problem as @amrbenattia

Creating dataset "/hdd/Alice-Dataset/Experiments/Pneumothorax/prog_gan_imgs/"
Added 4 images.                         
Traceback (most recent call last):
  File "dataset_tool.py", line 738, in <module>
    execute_cmdline(sys.argv)
  File "dataset_tool.py", line 733, in execute_cmdline
    func(**vars(args))
  File "dataset_tool.py", line 622, in create_from_images
    tfr.add_image(img)
  File "dataset_tool.py", line 75, in add_image
    assert img.shape == self.shape
AssertionError

keiboy · 2019-04-11T14:42:07Z

Thank you for your work. I have a related question. when I user cifar 10 dataset , I change max_label_size = 'full', code can run with labels. and I can also get img under different labels. when I create tfrecords formatted datasets from our own dataset,I can get *tfrecords and *labels,but I can not get img under different labels.The img from different labels are the same. any one could help me?

test4fest · 2019-04-17T18:48:04Z

@amrbenattia
I resolved this issue with changing the following code:
labels.dtype == np.int64

tvshow5727 · 2019-08-27T10:54:21Z

when i run python import_emample.py in linux ,there are small mistakes as follows:
when i add
import os
os.environ['CUDA_VISIBLE_DEVICES'] = "0"
but,,,it doesn't work .thanks for your suggestions.
@skjerns @skjerns

2019-08-27 18:45:01.035566: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-27 18:45:01.043830: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-08-27 18:45:01.144797: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55898d0 executing computations on platform CUDA. Devices:
2019-08-27 18:45:01.144829: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2019-08-27 18:45:01.163683: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3399985000 Hz
2019-08-27 18:45:01.164503: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55fb910 executing computations on platform Host. Devices:
2019-08-27 18:45:01.164541: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2019-08-27 18:45:01.165868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:03:00.0
2019-08-27 18:45:01.166020: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:
2019-08-27 18:45:01.166127: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:
2019-08-27 18:45:01.166217: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:
2019-08-27 18:45:01.166304: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:
2019-08-27 18:45:01.166390: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:
2019-08-27 18:45:01.166476: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:
2019-08-27 18:45:01.170958: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-08-27 18:45:01.170989: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1663] Cannot dlopen some GPU libraries. Skipping registering GPU devices...
2019-08-27 18:45:01.171011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-27 18:45:01.171025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-08-27 18:45:01.171036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-08-27 18:45:01.817931: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
Traceback (most recent call last):
File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation G_paper_1/Run/G_paper_1/latents_in: {{node G_paper_1/Run/G_paper_1/latents_in}}was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0 ]. Make sure the device specification refers to a valid device.
[[G_paper_1/Run/G_paper_1/latents_in]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "import_example-src.py", line 29, in
images = Gs.run(latents, labels)
File "/media/qcit-server/ai_install_lib/progressive_growing_of_gans-master-tensorflow/tfutil.py", line 687, in run
mb_out = tf.compat.v1.get_default_session().run(out_expr, dict(zip(self.input_templates, mb_in)))
File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation G_paper_1/Run/G_paper_1/latents_in: node G_paper_1/Run/G_paper_1/latents_in (defined at /media/qcit-server/ai_install_lib/progressive_growing_of_gans-master-tensorflow/tfutil.py:526) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0 ]. Make sure the device specification refers to a valid device.
[[G_paper_1/Run/G_paper_1/latents_in]]

Errors may have originated from an input operation.
Input Source operations connected to node G_paper_1/Run/G_paper_1/latents_in:
G_paper_1/Run/split (defined at /media/qcit-server/ai_install_lib/progressive_growing_of_gans-master-tensorflow/tfutil.py:659)

doantientai · 2019-10-04T10:13:42Z

In my case, I create a new .py file and do this:
from dataset_tool import create_from_images create_from_images_resize(tfrecord_dir='path_output', image_dir='path_dir_imgs', shuffle=True)
Make sure all images in the path_dir_imgs are in the same size

yamand16 · 2020-02-24T16:29:05Z

When I try to run create_celebahq function, the error about cryptography.fernet.InvalidToken occurs. I could not install libjpeg 8d, because this version is not available in Conda. It cannot find it. How can I solve this problem or how can I create Celeba_hq dataset from .dat files?

bfallc · 2020-04-30T00:49:15Z

Edit: I managed to solve myself. In my case some of my images were sRGB instead of RGB

Did someone managed to solve the "Assertion Error" issue? I'm having the same problem as @amrbenattia

Creating dataset "/hdd/Alice-Dataset/Experiments/Pneumothorax/prog_gan_imgs/"
Added 4 images.                         
Traceback (most recent call last):
  File "dataset_tool.py", line 738, in <module>
    execute_cmdline(sys.argv)
  File "dataset_tool.py", line 733, in execute_cmdline
    func(**vars(args))
  File "dataset_tool.py", line 622, in create_from_images
    tfr.add_image(img)
  File "dataset_tool.py", line 75, in add_image
    assert img.shape == self.shape
AssertionError

I had this exact error too. I don't think sRGB is a problem (I ran a dataset with sRGB images which converted fine). Also, if images aren't RGB, the script will output an appropriate error message ("Input images must be stored as RGB or grayscale").

My resolution to this message was finding non-square images among thousands in the dataset. I had preprocessed my dataset images in photoshop before converting. I hadn't noticed that some of the images (that were pulled from around the internet) had non-square pixel aspect ratios. Photoshop had automatically corrected them to square (with Pixel Aspect Ratio Correction) while I was processing the images so that the aspect ratio wasn't immediately obvious. Once these images were squared up everything was fine.

If you want to alter this script to tell you the image it fails on, add print(image_filenames[order[idx]]) after line 616 of dataset_tool.py

for idx in range(order.size):
            print(image_filenames[order[idx]])
            img = np.asarray(PIL.Image.open(image_filenames[order[idx]]))

Hope this helps future travelers.

samuro95 · 2021-10-13T14:02:30Z

when i run python import_emample.py in linux ,there are small mistakes as follows: when i add import os os.environ['CUDA_VISIBLE_DEVICES'] = "0" but,,,it doesn't work .thanks for your suggestions. @skjerns @skjerns

2019-08-27 18:45:01.035566: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2019-08-27 18:45:01.043830: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1 2019-08-27 18:45:01.144797: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55898d0 executing computations on platform CUDA. Devices: 2019-08-27 18:45:01.144829: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1 2019-08-27 18:45:01.163683: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3399985000 Hz 2019-08-27 18:45:01.164503: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55fb910 executing computations on platform Host. Devices: 2019-08-27 18:45:01.164541: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , 2019-08-27 18:45:01.165868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582 pciBusID: 0000:03:00.0 2019-08-27 18:45:01.166020: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64: 2019-08-27 18:45:01.166127: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64: 2019-08-27 18:45:01.166217: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64: 2019-08-27 18:45:01.166304: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64: 2019-08-27 18:45:01.166390: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64: 2019-08-27 18:45:01.166476: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64: 2019-08-27 18:45:01.170958: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2019-08-27 18:45:01.170989: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1663] Cannot dlopen some GPU libraries. Skipping registering GPU devices... 2019-08-27 18:45:01.171011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-08-27 18:45:01.171025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2019-08-27 18:45:01.171036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2019-08-27 18:45:01.817931: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile. Traceback (most recent call last): File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call return fn(*args) File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation G_paper_1/Run/G_paper_1/latents_in: {{node G_paper_1/Run/G_paper_1/latents_in}}was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0 ]. Make sure the device specification refers to a valid device. [[G_paper_1/Run/G_paper_1/latents_in]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "import_example-src.py", line 29, in images = Gs.run(latents, labels) File "/media/qcit-server/ai_install_lib/progressive_growing_of_gans-master-tensorflow/tfutil.py", line 687, in run mb_out = tf.compat.v1.get_default_session().run(out_expr, dict(zip(self.input_templates, mb_in))) File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 950, in run run_metadata_ptr) File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1173, in _run feed_dict_tensor, options, run_metadata) File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run run_metadata) File "/home/zff/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation G_paper_1/Run/G_paper_1/latents_in: node G_paper_1/Run/G_paper_1/latents_in (defined at /media/qcit-server/ai_install_lib/progressive_growing_of_gans-master-tensorflow/tfutil.py:526) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0 ]. Make sure the device specification refers to a valid device. [[G_paper_1/Run/G_paper_1/latents_in]]

Errors may have originated from an input operation. Input Source operations connected to node G_paper_1/Run/G_paper_1/latents_in: G_paper_1/Run/split (defined at /media/qcit-server/ai_install_lib/progressive_growing_of_gans-master-tensorflow/tfutil.py:659)

+1

tkarras added 6 commits February 23, 2018 13:50

Add a note about versions. Update the YouTube link.

3367435

Color tags are not supported in GitHub

c21687f

Update CelebA-HQ and LSUN configs to match final version of the paper

18d4116

Update README after releasing the TensorFlow version

82c0b36

Add contact for press and other inquiries

34ae24c

Update contacts

2c76604

Zhangsihao-Yang closed this Aug 25, 2018

Zhangsihao-Yang reopened this Aug 25, 2018

Update Google Drive links

1a05598

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

I have problem with creating tfrecord file #6

I have problem with creating tfrecord file #6

Zhangsihao-Yang commented Aug 25, 2018 •

edited

Loading

metatl commented Nov 15, 2018

Iris6713 commented Nov 30, 2018

skjerns commented Dec 14, 2018

liminalitythree commented Dec 26, 2018 •

edited

Loading

skjerns commented Jan 2, 2019

amrbenattia commented Mar 5, 2019

nandaw commented Mar 22, 2019 •

edited

Loading

keiboy commented Apr 11, 2019

test4fest commented Apr 17, 2019

tvshow5727 commented Aug 27, 2019

doantientai commented Oct 4, 2019

yamand16 commented Feb 24, 2020

bfallc commented Apr 30, 2020

samuro95 commented Oct 13, 2021

I have problem with creating tfrecord file #6

Are you sure you want to change the base?

I have problem with creating tfrecord file #6

Conversation

Zhangsihao-Yang commented Aug 25, 2018 • edited Loading

metatl commented Nov 15, 2018

Iris6713 commented Nov 30, 2018

skjerns commented Dec 14, 2018

liminalitythree commented Dec 26, 2018 • edited Loading

skjerns commented Jan 2, 2019

amrbenattia commented Mar 5, 2019

nandaw commented Mar 22, 2019 • edited Loading

keiboy commented Apr 11, 2019

test4fest commented Apr 17, 2019

tvshow5727 commented Aug 27, 2019

doantientai commented Oct 4, 2019

yamand16 commented Feb 24, 2020

bfallc commented Apr 30, 2020

samuro95 commented Oct 13, 2021

Zhangsihao-Yang commented Aug 25, 2018 •

edited

Loading

liminalitythree commented Dec 26, 2018 •

edited

Loading

nandaw commented Mar 22, 2019 •

edited

Loading