About Step1 Training MagicPoint on Synthetic Shapes Problem

[LSK0821]

Dear Author, Hello!
I am using TF1.2 and python3.6 under linux system.Having an out-of-index problem while executing step 1 to extract gaussian_noise. I changed line 184 in the file /superpoint/superpoint/synthetic_shapes.py to tf.contrib.data.Dataset.map_parallel = lambda self, fn: self.map( fn, num_parallel_calls=config['num_parallel_calls']), because I see that tf1.4 and below use data with contrib, the other parts have not been modified, but look at the content of the error report from this sentence into.
The following is the content of the console error reporting：

(test) liusikang@4029GP-TRT:~/cord/superpoint/superpoint$ python experiment.py train configs/magic-point_shapes.yaml magic-point_synth
/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:458: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:459: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:460: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:461: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:462: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:465: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
experiment.py:155: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(f)
[12/05/2022 10:17:55 INFO] Running command TRAIN
[12/05/2022 10:17:55 INFO] Number of GPUs detected: 1
INFO:tensorflow:Extracting archive for primitive draw_lines.
[12/05/2022 10:17:58 INFO] Extracting archive for primitive draw_lines.
INFO:tensorflow:Extracting archive for primitive draw_polygon.
[12/05/2022 10:18:02 INFO] Extracting archive for primitive draw_polygon.
INFO:tensorflow:Extracting archive for primitive draw_multiple_polygons.
[12/05/2022 10:18:05 INFO] Extracting archive for primitive draw_multiple_polygons.
INFO:tensorflow:Extracting archive for primitive draw_ellipses.
[12/05/2022 10:18:08 INFO] Extracting archive for primitive draw_ellipses.
INFO:tensorflow:Extracting archive for primitive draw_star.
[12/05/2022 10:18:11 INFO] Extracting archive for primitive draw_star.
INFO:tensorflow:Extracting archive for primitive draw_checkerboard.
[12/05/2022 10:18:14 INFO] Extracting archive for primitive draw_checkerboard.
INFO:tensorflow:Extracting archive for primitive draw_stripes.
[12/05/2022 10:18:17 INFO] Extracting archive for primitive draw_stripes.
INFO:tensorflow:Extracting archive for primitive draw_cube.
[12/05/2022 10:18:20 INFO] Extracting archive for primitive draw_cube.
INFO:tensorflow:Extracting archive for primitive gaussian_noise.
[12/05/2022 10:18:23 INFO] Extracting archive for primitive gaussian_noise.
Traceback (most recent call last):
File "experiment.py", line 162, in
args.func(config, output_dir, args)
File "experiment.py", line 99, in _cli_train
train(config, config['train_iter'], output_dir, pretrained_dir)
File "experiment.py", line 22, in train
with _init_graph(config) as net:
File "/home/liusikang/anaconda3/envs/test/lib/python3.6/contextlib.py", line 82, in enter
return next(self.gen)
File "experiment.py", line 73, in _init_graph
dataset = get_dataset(config['data']['name'])(**config['data'])
File "/home/liusikang/cord/superpoint/superpoint/datasets/base_dataset.py", line 108, in init
self.tf_splits[n] = self._get_data(self.dataset, n, **self.config)
File "/home/liusikang/cord/superpoint/superpoint/datasets/synthetic_shapes.py", line 191, in _get_data
(filenames[split_name]['images'], filenames[split_name]['points'])).batch(batch_size=32)
File "/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 473, in from_tensor_slices
return TensorSliceDataset(tensors)
File "/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 896, in init
batch_dim = flat_tensors[0].get_shape()[0]
File "/home/liusikang/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/framework/tensor_shape.py", line 500, in getitem
return self._dims[key]
IndexError: list index out of range

Looking forward to your reply!

[LSK0821]

My apologies. I tried to specify the batch size by adding .batch(batch_size=32)
"/home/liusikang/cord/superpoint/superpoint/datasets/synthetic_shapes.py", line 191, in _get_data
(filenames[split_name]['images'], filenames[split_name]['points'])).batch(batch_size=32), still no solve. So I removed batch(batch_size=32) and it still showed IndexError: list index out of range.

[rpautrat]

Hi, why are you using such an old version of Tensorflow? I suggest you to upgrade, e.g. to 1.12 as in #173 (comment)

[vrahnos3]

Hello, thanks for the great job,
I followed the steps of #173
I have a problem with loss nan after extracting all syntetic shapes. Any ideas?
Here is my error message:
/home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/panosvrach/anaconda3/envs/superpoint/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
[02/17/2023 16:50:21 INFO] Running command TRAIN
[02/17/2023 16:50:21 INFO] Number of GPUs detected: 1
[02/17/2023 16:50:23 INFO] Extracting archive for primitive draw_lines.
[02/17/2023 16:50:26 INFO] Extracting archive for primitive draw_polygon.
[02/17/2023 16:50:29 INFO] Extracting archive for primitive draw_multiple_polygons.
[02/17/2023 16:50:32 INFO] Extracting archive for primitive draw_ellipses.
[02/17/2023 16:50:35 INFO] Extracting archive for primitive draw_star.
[02/17/2023 16:50:38 INFO] Extracting archive for primitive draw_checkerboard.
[02/17/2023 16:50:41 INFO] Extracting archive for primitive draw_stripes.
[02/17/2023 16:50:43 INFO] Extracting archive for primitive draw_cube.
[02/17/2023 16:50:46 INFO] Extracting archive for primitive gaussian_noise.
[02/17/2023 16:50:50 INFO] Caching data, fist access will take some time.
[02/17/2023 16:50:51 INFO] Caching data, fist access will take some time.
[02/17/2023 16:50:51 INFO] Caching data, fist access will take some time.
2023-02-17 16:50:51.923674: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2023-02-17 16:50:52.270209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.785
pciBusID: 0000:3b:00.0
totalMemory: 23.70GiB freeMemory: 23.43GiB
2023-02-17 16:50:52.270250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2023-02-17 16:55:07.976783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-02-17 16:55:07.976819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2023-02-17 16:55:07.976825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2023-02-17 16:55:07.976957: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2023-02-17 16:55:07.976989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22711 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:3b:00.0, compute capability: 8.6)
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:08 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
[02/17/2023 16:55:09 INFO] Scale of 0 disables regularizer.
2023-02-17 16:55:09.704916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2023-02-17 16:55:09.704969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-02-17 16:55:09.704973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2023-02-17 16:55:09.704977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2023-02-17 16:55:09.705063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22711 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:3b:00.0, compute capability: 8.6)
[02/17/2023 16:55:14 INFO] Start training
[02/17/2023 17:14:51 INFO] Iter 0: loss 4.1759, precision 0.0005, recall 0.0482
/home/panosvrach/Desktop/SuperPoint/superpoint/models/base_model.py:387: RuntimeWarning: Mean of empty slice
metrics = {m: np.nanmean(metrics[m], axis=0) for m in metrics}
[02/17/2023 17:15:10 INFO] Iter 1000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:15:30 INFO] Iter 2000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:15:49 INFO] Iter 3000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:16:08 INFO] Iter 4000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:16:27 INFO] Iter 5000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:16:47 INFO] Iter 6000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:17:06 INFO] Iter 7000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:17:25 INFO] Iter 8000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:17:44 INFO] Iter 9000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:18:03 INFO] Iter 10000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:18:23 INFO] Iter 11000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:18:42 INFO] Iter 12000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:19:01 INFO] Iter 13000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:19:20 INFO] Iter 14000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:19:39 INFO] Iter 15000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:19:58 INFO] Iter 16000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:20:17 INFO] Iter 17000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:20:37 INFO] Iter 18000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:20:56 INFO] Iter 19000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:21:15 INFO] Iter 20000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:21:34 INFO] Iter 21000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:21:53 INFO] Iter 22000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:22:13 INFO] Iter 23000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:22:32 INFO] Iter 24000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:22:51 INFO] Iter 25000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:23:10 INFO] Iter 26000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:23:30 INFO] Iter 27000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:23:49 INFO] Iter 28000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:24:08 INFO] Iter 29000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:24:27 INFO] Iter 30000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:24:46 INFO] Iter 31000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:25:05 INFO] Iter 32000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:25:25 INFO] Iter 33000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:25:44 INFO] Iter 34000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:26:03 INFO] Iter 35000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:26:22 INFO] Iter 36000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:26:41 INFO] Iter 37000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:27:01 INFO] Iter 38000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:27:20 INFO] Iter 39000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:27:39 INFO] Iter 40000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:27:58 INFO] Iter 41000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:28:18 INFO] Iter 42000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:28:37 INFO] Iter 43000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:28:56 INFO] Iter 44000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:29:15 INFO] Iter 45000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:29:34 INFO] Iter 46000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:29:54 INFO] Iter 47000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:30:13 INFO] Iter 48000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:30:32 INFO] Iter 49000: loss nan, precision nan, recall 0.0000
[02/17/2023 17:30:47 INFO] Training finished
[02/17/2023 17:30:47 INFO] Saving checkpoint for iteration #50000
2023-02-17 17:30:47.452008: W tensorflow/core/kernels/data/cache_dataset_ops.cc:770] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the datasetwill be discarded. This can happen if you have an input pipeline similar to dataset.cache().take(k).repeat(). You should use dataset.take(k).cache().repeat() instead.

[rpautrat]

Hi, this seems to be very related to this issue: #189

In this other case, it was due to an incompatibility of ampere technology (SM_86) and old cuda + tf versions. Could it be the same for you?

[vrahnos3]

I use cuda 11.4 so probably its an incompatibility of cuda-tf version. I will try the pytorch implementation as mentioned in #189 otherwise I will try to downgrade cuda version.
Thanks a lot for your fast reply.

[1z2213]

Dear author， Hello！
Hi, I followed your instructions of #173 and now I am trying to run the 1st step. python experiment.py train configs/magic-point_shapes.yaml magic-point_synth. I have the same problem with loss nan after extracting all syntetic shapes. I desperately need your help. Looking forward to your reply.
Here are the instructions and all the information displayed when I run the code.
sunlab@sunlab-ThinkStation-P520:~$ source ~/anaconda3/bin/activate
(base) sunlab@sunlab-ThinkStation-P520:~$ conda create -n linenv python=3.6.3
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
current version: 4.10.1
latest version: 23.5.0

Please update conda by running

$ conda update -n base -c defaults conda

Package Plan

environment location: /home/sunlab/anaconda3/envs/linenv

added / updated specs:
- python=3.6.3

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
pip-21.2.2                 |   py36h06a4308_0         1.8 MB  defaults
python-3.6.3               |       h6c0c0dc_5        25.5 MB  defaults
setuptools-58.0.4          |   py36h06a4308_0         788 KB  defaults
------------------------------------------------------------
                                       Total:        28.1 MB

The following NEW packages will be INSTALLED:

_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
_openmp_mutex pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
ca-certificates pkgs/main/linux-64::ca-certificates-2023.05.30-h06a4308_0
certifi pkgs/main/linux-64::certifi-2021.5.30-py36h06a4308_0
libedit pkgs/main/linux-64::libedit-3.1.20221030-h5eee18b_0
libffi pkgs/main/linux-64::libffi-3.2.1-hf484d3e_1007
libgcc-ng pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1
libgomp pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1
ncurses pkgs/main/linux-64::ncurses-6.4-h6a678d5_0
openssl pkgs/main/linux-64::openssl-1.0.2u-h7b6447c_0
pip pkgs/main/linux-64::pip-21.2.2-py36h06a4308_0
python pkgs/main/linux-64::python-3.6.3-h6c0c0dc_5
readline pkgs/main/linux-64::readline-7.0-h7b6447c_5
setuptools pkgs/main/linux-64::setuptools-58.0.4-py36h06a4308_0
sqlite pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
tk pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0
wheel pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0
xz pkgs/main/linux-64::xz-5.4.2-h5eee18b_0
zlib pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0

Proceed ([y]/n)? y

Downloading and Extracting Packages
python-3.6.3 | 25.5 MB | ##################################### | 100%
setuptools-58.0.4 | 788 KB | ##################################### | 100%
pip-21.2.2 | 1.8 MB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

To activate this environment, use

$ conda activate linenv

To deactivate an active environment, use

$ conda deactivate

(base) sunlab@sunlab-ThinkStation-P520:~$ conda activate linenv
(linenv) sunlab@sunlab-ThinkStation-P520:~$ conda install tensorflow-gpu=1.13
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
current version: 4.10.1
latest version: 23.5.0

Please update conda by running

$ conda update -n base -c defaults conda

Package Plan

environment location: /home/sunlab/anaconda3/envs/linenv

added / updated specs:
- tensorflow-gpu=1.13

The following NEW packages will be INSTALLED:

_tflow_select pkgs/main/linux-64::_tflow_select-2.1.0-gpu
absl-py pkgs/main/noarch::absl-py-0.15.0-pyhd3eb1b0_0
astor pkgs/main/linux-64::astor-0.8.1-py36h06a4308_0
blas pkgs/main/linux-64::blas-1.0-mkl
c-ares pkgs/main/linux-64::c-ares-1.19.0-h5eee18b_0
cudatoolkit pkgs/main/linux-64::cudatoolkit-10.0.130-0
cudnn pkgs/main/linux-64::cudnn-7.6.5-cuda10.0_0
cupti pkgs/main/linux-64::cupti-10.0.130-0
dataclasses pkgs/main/noarch::dataclasses-0.8-pyh4f3eec9_6
gast pkgs/main/noarch::gast-0.5.3-pyhd3eb1b0_0
grpcio pkgs/main/linux-64::grpcio-1.14.1-py36h9ba97e2_0
h5py pkgs/main/linux-64::h5py-2.10.0-py36hd6299e0_1
hdf5 pkgs/main/linux-64::hdf5-1.10.6-hb1b8bf9_0
intel-openmp pkgs/main/linux-64::intel-openmp-2022.1.0-h9e868ea_3769
keras-applications pkgs/main/noarch::keras-applications-1.0.8-py_1
keras-preprocessi~ pkgs/main/noarch::keras-preprocessing-1.1.2-pyhd3eb1b0_0
libgfortran-ng pkgs/main/linux-64::libgfortran-ng-7.5.0-ha8ba4b0_17
libgfortran4 pkgs/main/linux-64::libgfortran4-7.5.0-ha8ba4b0_17
libprotobuf pkgs/main/linux-64::libprotobuf-3.17.2-h4ff587b_1
markdown pkgs/main/linux-64::markdown-3.1.1-py36_0
mkl pkgs/main/linux-64::mkl-2020.2-256
mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py36he8ac12f_0
mkl_fft pkgs/main/linux-64::mkl_fft-1.3.0-py36h54f3939_0
mkl_random pkgs/main/linux-64::mkl_random-1.1.1-py36h0573a6f_0
mock pkgs/main/noarch::mock-4.0.3-pyhd3eb1b0_0
numpy pkgs/main/linux-64::numpy-1.19.2-py36h54aff64_0
numpy-base pkgs/main/linux-64::numpy-base-1.19.2-py36hfa32c7d_0
protobuf pkgs/main/linux-64::protobuf-3.17.2-py36h295c915_0
scipy pkgs/main/linux-64::scipy-1.5.2-py36h0b6359f_0
six pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1
tensorboard pkgs/main/linux-64::tensorboard-1.13.1-py36hf484d3e_0
tensorflow pkgs/main/linux-64::tensorflow-1.13.1-gpu_py36h3991807_0
tensorflow-base pkgs/main/linux-64::tensorflow-base-1.13.1-gpu_py36h8d69cac_0
tensorflow-estima~ pkgs/main/noarch::tensorflow-estimator-1.13.0-py_0
tensorflow-gpu pkgs/main/linux-64::tensorflow-gpu-1.13.1-h0d30ee6_0
termcolor pkgs/main/linux-64::termcolor-1.1.0-py36h06a4308_1
werkzeug pkgs/main/noarch::werkzeug-2.0.3-pyhd3eb1b0_0

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(linenv) sunlab@sunlab-ThinkStation-P520:$ cd superpoint/SuperPoint-master
(linenv) sunlab@sunlab-ThinkStation-P520:/superpoint/SuperPoint-master$ make install
pip3 install -r requirements.txt
Requirement already satisfied: numpy in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.19.2)
Requirement already satisfied: scipy in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (1.5.2)
Collecting opencv-python==3.4.2.16
Using cached opencv_python-3.4.2.16-cp36-cp36m-manylinux1_x86_64.whl (25.0 MB)
Collecting opencv-contrib-python==3.4.2.16
Using cached opencv_contrib_python-3.4.2.16-cp36-cp36m-manylinux1_x86_64.whl (30.6 MB)
Collecting tqdm
Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting pyyaml
Using cached PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (603 kB)
Collecting flake8
Using cached flake8-5.0.4-py2.py3-none-any.whl (61 kB)
Collecting jupyter
Using cached jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB)
Collecting importlib-resources
Using cached importlib_resources-5.4.0-py3-none-any.whl (28 kB)
Collecting mccabe<0.8.0,>=0.7.0
Using cached mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB)
Collecting pycodestyle<2.10.0,>=2.9.0
Using cached pycodestyle-2.9.1-py2.py3-none-any.whl (41 kB)
Collecting importlib-metadata<4.3,>=1.1.0
Using cached importlib_metadata-4.2.0-py3-none-any.whl (16 kB)
Collecting pyflakes<2.6.0,>=2.5.0
Using cached pyflakes-2.5.0-py2.py3-none-any.whl (66 kB)
Collecting ipykernel
Using cached ipykernel-5.5.6-py3-none-any.whl (121 kB)
Collecting nbconvert
Using cached nbconvert-6.0.7-py3-none-any.whl (552 kB)
Collecting ipywidgets
Using cached ipywidgets-7.7.5-py2.py3-none-any.whl (123 kB)
Collecting notebook
Using cached notebook-6.4.10-py3-none-any.whl (9.9 MB)
Collecting qtconsole
Using cached qtconsole-5.2.2-py3-none-any.whl (120 kB)
Collecting jupyter-console
Using cached jupyter_console-6.4.3-py3-none-any.whl (22 kB)
Collecting typing-extensions>=3.6.4
Using cached typing_extensions-4.1.1-py3-none-any.whl (26 kB)
Collecting zipp>=0.5
Using cached zipp-3.6.0-py3-none-any.whl (5.3 kB)
Collecting jupyter-client
Using cached jupyter_client-7.1.2-py3-none-any.whl (130 kB)
Collecting traitlets>=4.1.0
Using cached traitlets-4.3.3-py2.py3-none-any.whl (75 kB)
Collecting ipython>=5.0.0
Using cached ipython-7.16.3-py3-none-any.whl (783 kB)
Collecting tornado>=4.2
Using cached tornado-6.1-cp36-cp36m-manylinux2010_x86_64.whl (427 kB)
Collecting ipython-genutils
Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
Using cached prompt_toolkit-3.0.36-py3-none-any.whl (386 kB)
Collecting pexpect
Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting backcall
Using cached backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools>=18.5 in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->jupyter->-r requirements.txt (line 8)) (58.0.4)
Collecting pygments
Using cached Pygments-2.14.0-py3-none-any.whl (1.1 MB)
Collecting decorator
Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting jedi<=0.17.2,>=0.10
Using cached jedi-0.17.2-py2.py3-none-any.whl (1.4 MB)
Collecting pickleshare
Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting parso<0.8.0,>=0.7.0
Using cached parso-0.7.1-py2.py3-none-any.whl (109 kB)
Collecting wcwidth
Using cached wcwidth-0.2.6-py2.py3-none-any.whl (29 kB)
Requirement already satisfied: six in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from traitlets>=4.1.0->ipykernel->jupyter->-r requirements.txt (line 8)) (1.16.0)
Collecting jupyterlab-widgets<3,>=1.0.0
Using cached jupyterlab_widgets-1.1.4-py3-none-any.whl (246 kB)
Collecting widgetsnbextension~=3.6.4
Using cached widgetsnbextension-3.6.4-py2.py3-none-any.whl (1.6 MB)
Collecting prometheus-client
Using cached prometheus_client-0.17.0-py3-none-any.whl (60 kB)
Collecting Send2Trash>=1.8.0
Using cached Send2Trash-1.8.2-py3-none-any.whl (18 kB)
Collecting jinja2
Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting argon2-cffi
Using cached argon2_cffi-21.3.0-py3-none-any.whl (14 kB)
Collecting nbformat
Using cached nbformat-5.1.3-py3-none-any.whl (178 kB)
Collecting nest-asyncio>=1.5
Using cached nest_asyncio-1.5.6-py3-none-any.whl (5.2 kB)
Collecting pyzmq>=17
Using cached pyzmq-25.1.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.1 MB)
Collecting terminado>=0.8.3
Using cached terminado-0.12.1-py3-none-any.whl (15 kB)
Collecting jupyter-core>=4.6.1
Using cached jupyter_core-4.9.2-py3-none-any.whl (86 kB)
Collecting python-dateutil>=2.1
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting entrypoints
Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Collecting defusedxml
Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting pandocfilters>=1.4.1
Using cached pandocfilters-1.5.0-py2.py3-none-any.whl (8.7 kB)
Collecting testpath
Using cached testpath-0.6.0-py3-none-any.whl (83 kB)
Collecting mistune<2,>=0.8.1
Using cached mistune-0.8.4-py2.py3-none-any.whl (16 kB)
Collecting bleach
Using cached bleach-4.1.0-py2.py3-none-any.whl (157 kB)
Collecting nbclient<0.6.0,>=0.5.0
Using cached nbclient-0.5.9-py3-none-any.whl (69 kB)
Collecting jupyterlab-pygments
Using cached jupyterlab_pygments-0.1.2-py2.py3-none-any.whl (4.6 kB)
Collecting MarkupSafe>=2.0
Using cached MarkupSafe-2.0.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB)
Collecting async-generator
Using cached async_generator-1.10-py3-none-any.whl (18 kB)
Collecting jsonschema!=2.5.0,>=2.4
Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting attrs>=17.4.0
Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting pyrsistent>=0.14.0
Using cached pyrsistent-0.18.0-cp36-cp36m-manylinux1_x86_64.whl (117 kB)
Collecting ptyprocess
Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Requirement already satisfied: dataclasses in /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages (from argon2-cffi->notebook->jupyter->-r requirements.txt (line 8)) (0.8)
Collecting argon2-cffi-bindings
Using cached argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (86 kB)
Collecting cffi>=1.0.1
Using cached cffi-1.15.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (402 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting webencodings
Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting packaging
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.7-py3-none-any.whl (98 kB)
Collecting qtpy
Using cached QtPy-2.0.1-py3-none-any.whl (65 kB)
WARNING: The candidate selected for download or install is a yanked version: 'opencv-python' candidate (version 3.4.2.16 at https://files.pythonhosted.org/packages/fa/7d/5042b668a8ed41d2a80b8c172f5efcd572e3c046c75ae029407e19b7fc68/opencv_python-3.4.2.16-cp36-cp36m-manylinux1_x86_64.whl#sha256=d75f60baced5086300a19c8ba63e75d059e8dce333795ef02084b9be6ec61516 (from https://pypi.org/simple/opencv-python/))
Reason for being yanked: Release deprecated
WARNING: The candidate selected for download or install is a yanked version: 'opencv-contrib-python' candidate (version 3.4.2.16 at https://files.pythonhosted.org/packages/08/f1/66330f4042c4fb3b2d77a159db8e8916d9cdecc29bc8c1f56bc7f8a9bec9/opencv_contrib_python-3.4.2.16-cp36-cp36m-manylinux1_x86_64.whl#sha256=8de56394a9a3cf8788559032c2139c622ffdc7e37c32215ec865b4e1cd2ca70d (from https://pypi.org/simple/opencv-contrib-python/))
Reason for being yanked: Release deprecated
Installing collected packages: zipp, typing-extensions, ipython-genutils, decorator, traitlets, pyrsistent, importlib-metadata, attrs, wcwidth, tornado, pyzmq, python-dateutil, pyparsing, pycparser, ptyprocess, parso, nest-asyncio, jupyter-core, jsonschema, entrypoints, webencodings, pygments, prompt-toolkit, pickleshare, pexpect, packaging, nbformat, MarkupSafe, jupyter-client, jedi, cffi, backcall, async-generator, testpath, pandocfilters, nbclient, mistune, jupyterlab-pygments, jinja2, ipython, defusedxml, bleach, argon2-cffi-bindings, terminado, Send2Trash, prometheus-client, nbconvert, ipykernel, argon2-cffi, notebook, widgetsnbextension, qtpy, jupyterlab-widgets, qtconsole, pyflakes, pycodestyle, mccabe, jupyter-console, ipywidgets, importlib-resources, tqdm, pyyaml, opencv-python, opencv-contrib-python, jupyter, flake8
Successfully installed MarkupSafe-2.0.1 Send2Trash-1.8.2 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 async-generator-1.10 attrs-22.2.0 backcall-0.2.0 bleach-4.1.0 cffi-1.15.1 decorator-5.1.1 defusedxml-0.7.1 entrypoints-0.4 flake8-5.0.4 importlib-metadata-4.2.0 importlib-resources-5.4.0 ipykernel-5.5.6 ipython-7.16.3 ipython-genutils-0.2.0 ipywidgets-7.7.5 jedi-0.17.2 jinja2-3.0.3 jsonschema-3.2.0 jupyter-1.0.0 jupyter-client-7.1.2 jupyter-console-6.4.3 jupyter-core-4.9.2 jupyterlab-pygments-0.1.2 jupyterlab-widgets-1.1.4 mccabe-0.7.0 mistune-0.8.4 nbclient-0.5.9 nbconvert-6.0.7 nbformat-5.1.3 nest-asyncio-1.5.6 notebook-6.4.10 opencv-contrib-python-3.4.2.16 opencv-python-3.4.2.16 packaging-21.3 pandocfilters-1.5.0 parso-0.7.1 pexpect-4.8.0 pickleshare-0.7.5 prometheus-client-0.17.0 prompt-toolkit-3.0.36 ptyprocess-0.7.0 pycodestyle-2.9.1 pycparser-2.21 pyflakes-2.5.0 pygments-2.14.0 pyparsing-3.0.7 pyrsistent-0.18.0 python-dateutil-2.8.2 pyyaml-6.0 pyzmq-25.1.0 qtconsole-5.2.2 qtpy-2.0.1 terminado-0.12.1 testpath-0.6.0 tornado-6.1 tqdm-4.64.1 traitlets-4.3.3 typing-extensions-4.1.1 wcwidth-0.2.6 webencodings-0.5.1 widgetsnbextension-3.6.4 zipp-3.6.0
pip3 install -e .
Obtaining file:///home/sunlab/superpoint/SuperPoint-master
Installing collected packages: superpoint
Running setup.py develop for superpoint
Successfully installed superpoint-0.0
sh setup.sh
Path of the directory where datasets are stored and read: /home/sunlab//superpoint/SuperPoint-master/DATA_DIR
Path of the directory where experiments data (logs, checkpoints, configs) are written: /home/sunlab//superpoint/SuperPoint-master/EXPER_DIR
(linenv) sunlab@sunlab-ThinkStation-P520:/superpoint/SuperPoint-master$ cd ./superpoint
(linenv) sunlab@sunlab-ThinkStation-P520:/superpoint/SuperPoint-master/superpoint$ export TMPDIR=/tmp/
(linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master/superpoint$ export TF_FORCE_GPU_ALLOW_GROWTH=true
(linenv) sunlab@sunlab-ThinkStation-P520:/superpoint/SuperPoint-master/superpoint$ python experiment.py train configs/magic-point_shapes.yaml magic-point_synth/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
[07/06/2023 23:37:58 INFO] Running command TRAIN
Traceback (most recent call last):
File "experiment.py", line 160, in
args.func(config, output_dir, args)
File "experiment.py", line 97, in _cli_train
train(config, config['train_iter'], output_dir, pretrained_dir)
File "experiment.py", line 22, in train
with _init_graph(config) as net:
File "/home/sunlab/anaconda3/envs/linenv/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "experiment.py", line 68, in _init_graph
n_gpus = get_num_gpus()
File "experiment.py", line 62, in get_num_gpus
return len(os.environ['CUDA_VISIBLE_DEVICES'].split(','))
File "/home/sunlab/anaconda3/envs/linenv/lib/python3.6/os.py", line 669, in getitem
raise KeyError(key) from None
KeyError: 'CUDA_VISIBLE_DEVICES'
**(linenv) sunlab@sunlab-ThinkStation-P520:/superpoint/SuperPoint-master/superpoint$ export CUDA_VISIBLE_DEVICES=0
(linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master/superpoint$ python experiment.py train** configs/magic-point_shapes.yaml magic-point_synth/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
[07/06/2023 23:38:09 INFO] Running command TRAIN
[07/06/2023 23:38:09 INFO] Number of GPUs detected: 1
[07/06/2023 23:38:12 INFO] Extracting archive for primitive draw_lines.
[07/06/2023 23:38:15 INFO] Extracting archive for primitive draw_polygon.
[07/06/2023 23:38:19 INFO] Extracting archive for primitive draw_multiple_polygons.
[07/06/2023 23:38:22 INFO] Extracting archive for primitive draw_ellipses.
[07/06/2023 23:38:26 INFO] Extracting archive for primitive draw_star.
[07/06/2023 23:38:30 INFO] Extracting archive for primitive draw_checkerboard.
[07/06/2023 23:38:33 INFO] Extracting archive for primitive draw_stripes.
[07/06/2023 23:38:36 INFO] Extracting archive for primitive draw_cube.
[07/06/2023 23:38:40 INFO] Extracting archive for primitive gaussian_noise.
[07/06/2023 23:38:45 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/datasets/synthetic_shapes.py:189: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, use
tf.py_function, which takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.

[07/06/2023 23:38:45 INFO] Caching data, fist access will take some time.
[07/06/2023 23:38:45 WARNING] From /home/sunlab/anaconda3/envs/linenv/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py:423: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[07/06/2023 23:38:45 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/homographies.py:218: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
[07/06/2023 23:38:45 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/homographies.py:277: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
[07/06/2023 23:38:46 INFO] Caching data, fist access will take some time.
[07/06/2023 23:38:46 INFO] Caching data, fist access will take some time.
2023-07-06 23:38:46.500189: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2023-07-06 23:38:46.688675: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2023-07-06 23:38:46.701881: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x557760fccac0 executing computations on platform Host. Devices:
2023-07-06 23:38:46.702032: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2023-07-06 23:38:46.823689: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x557760fce720 executing computations on platform CUDA. Devices:
2023-07-06 23:38:46.823750: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): NVIDIA GeForce RTX 3060, Compute Capability 8.6
2023-07-06 23:38:46.823998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: NVIDIA GeForce RTX 3060 major: 8 minor: 6 memoryClockRate(GHz): 1.837
pciBusID: 0000:65:00.0
totalMemory: 11.76GiB freeMemory: 10.53GiB
2023-07-06 23:38:46.824031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2023-07-06 23:38:46.845678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-07-06 23:38:46.845739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2023-07-06 23:38:46.845756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2023-07-06 23:38:46.845924: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2023-07-06 23:38:46.845989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10245 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:65:00.0, compute capability: 8.6)
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:47 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/backbones/vgg.py:10: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
[07/06/2023 23:38:47 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/backbones/vgg.py:14: batch_normalization (from tensorflow.python.layers.normalization) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.batch_normalization instead.
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:47 WARNING] From /home/sunlab/superpoint/SuperPoint-master/superpoint/models/backbones/vgg.py:28: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:47 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
[07/06/2023 23:38:48 INFO] Scale of 0 disables regularizer.
2023-07-06 23:38:48.493726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2023-07-06 23:38:48.493769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-07-06 23:38:48.493774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2023-07-06 23:38:48.493778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2023-07-06 23:38:48.493822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10245 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:65:00.0, compute capability: 8.6)
[07/06/2023 23:38:53 INFO] Start training
[07/07/2023 00:01:33 INFO] Iter 0: loss 4.1784, precision 0.0006, recall 0.0583
/home/sunlab/superpoint/SuperPoint-master/superpoint/models/base_model.py:387: RuntimeWarning: Mean of empty slice
metrics = {m: np.nanmean(metrics[m], axis=0) for m in metrics}
[07/07/2023 00:01:45 INFO] Iter 1000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:01:56 INFO] Iter 2000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:02:08 INFO] Iter 3000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:02:20 INFO] Iter 4000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:02:32 INFO] Iter 5000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:02:44 INFO] Iter 6000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:02:56 INFO] Iter 7000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:03:08 INFO] Iter 8000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:03:20 INFO] Iter 9000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:03:32 INFO] Iter 10000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:03:44 INFO] Iter 11000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:03:56 INFO] Iter 12000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:04:08 INFO] Iter 13000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:04:20 INFO] Iter 14000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:04:32 INFO] Iter 15000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:04:44 INFO] Iter 16000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:04:55 INFO] Iter 17000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:05:07 INFO] Iter 18000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:05:19 INFO] Iter 19000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:05:31 INFO] Iter 20000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:05:42 INFO] Iter 21000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:05:54 INFO] Iter 22000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:06:06 INFO] Iter 23000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:06:17 INFO] Iter 24000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:06:29 INFO] Iter 25000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:06:41 INFO] Iter 26000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:06:52 INFO] Iter 27000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:07:04 INFO] Iter 28000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:07:15 INFO] Iter 29000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:07:27 INFO] Iter 30000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:07:39 INFO] Iter 31000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:07:54 INFO] Iter 32000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:08:10 INFO] Iter 33000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:08:41 INFO] Iter 34000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:09:14 INFO] Iter 35000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:09:47 INFO] Iter 36000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:10:15 INFO] Iter 37000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:10:32 INFO] Iter 38000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:10:50 INFO] Iter 39000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:11:06 INFO] Iter 40000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:11:23 INFO] Iter 41000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:11:39 INFO] Iter 42000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:11:57 INFO] Iter 43000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:12:14 INFO] Iter 44000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:12:31 INFO] Iter 45000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:12:49 INFO] Iter 46000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:13:06 INFO] Iter 47000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:13:23 INFO] Iter 48000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:13:41 INFO] Iter 49000: loss nan, precision nan, recall 0.0000
[07/07/2023 00:13:54 INFO] Training finished
[07/07/2023 00:13:55 INFO] Saving checkpoint for iteration #50000
2023-07-07 00:13:56.578198: W tensorflow/core/kernels/data/cache_dataset_ops.cc:810] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the datasetwill be discarded. This can happen if you have an input pipeline similar to dataset.cache().take(k).repeat(). You should use dataset.take(k).cache().repeat() instead.
(linenv) sunlab@sunlab-ThinkStation-P520:~/superpoint/SuperPoint-master/superpoint$ python experiment.py train configs/magic-point_shapes.yaml magic-point_synth

[00ohnim]

I ran into this problem,too,but I don't know how to solve it.

[1z2213]

Add heading textAdd bold text, <Ctrl+b>Add italic text, <Ctrl+i>
Add a quote, <Ctrl+Shift+.>Add code, <Ctrl+e>Add a link, <Ctrl+k>
Add a bulleted list, <Ctrl+Shift+8>Add a numbered list, <Ctrl+Shift+7>Add a task list, <Ctrl+Shift+l>
Directly mention a user or team
Reference an issue, pull request, or discussion
Add saved reply
Slash commands
It seems like a lot of people experienced this problem, but those who experienced it don't seem to have a solution for it.

[rpautrat]

Hi, this might be due to the warning about the cache returned by tensorflow at the end of your output. Could you try training without caching the data, i.e. by setting 'cache_in_memory' to false in superpoint/configs/magic-point_shapes.yaml?

Alternatively, you could also add the field 'on-the-fly' in a new line after 'cache_in_memory' and set it to true. This would generate the shapes on the fly instead of pre-generating them, and could be a good way to check if the error is caused by caching issues.

[1z2213]

Hi, this might be due to the warning about the cache returned by tensorflow at the end of your output. Could you try training without caching the data, i.e. by setting 'cache_in_memory' to false in superpoint/configs/magic-point_shapes.yaml?

Alternatively, you could also add the field 'on-the-fly' in a new line after 'cache_in_memory' and set it to true. This would generate the shapes on the fly instead of pre-generating them, and could be a good way to check if the error is caused by caching issues.

I apologize for responding to your message so late, I tried your method but it didn't solve my problem. This proves that the error is not caused by caching issues.