Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

loss is nan #16

Open
xunfeng2zkj opened this issue Oct 31, 2023 · 0 comments
Open

loss is nan #16

xunfeng2zkj opened this issue Oct 31, 2023 · 0 comments

Comments

@xunfeng2zkj
Copy link

Model: "graphmodel_1"

Layer (type) Output Shape Param # Connected to

atom (InputLayer) (None, 1) 0

squeeze_1 (Squeeze) (None,) 0 atom_index[0][0]
atom[0][0]
n_pro[0][0]

atom_embedding (Embedding) (None, 256) 2560 squeeze_1[1][0]

dense_1 (Dense) (None, 256) 65536 atom_embedding[0][0]

connectivity (InputLayer) (None, 2) 0

gather_atom_to_bond_1 (GatherAt (None, 256) 0 dense_1[0][0]
connectivity[0][0]

gather_atom_to_bond_2 (GatherAt (None, 256) 0 dense_1[0][0]
connectivity[0][0]

distance_rbf (InputLayer) (None, 256) 0

concatenate_1 (Concatenate) (None, 768) 0 gather_atom_to_bond_1[0][0]
gather_atom_to_bond_2[0][0]
distance_rbf[0][0]

dense_2 (Dense) (None, 512) 393728 concatenate_1[0][0]

dense_3 (Dense) (None, 256) 131328 dense_2[0][0]

dense_4 (Dense) (None, 256) 65792 dense_3[0][0]

dense_5 (Dense) (None, 256) 65792 dense_4[0][0]

add_1 (Add) (None, 256) 0 dense_5[0][0]
distance_rbf[0][0]

multiply_1 (Multiply) (None, 256) 0 gather_atom_to_bond_1[0][0]
add_1[0][0]

reduce_bond_to_atom_1 (ReduceBo (None, 256) 0 multiply_1[0][0]
connectivity[0][0]

dense_6 (Dense) (None, 256) 65792 reduce_bond_to_atom_1[0][0]

dense_7 (Dense) (None, 256) 65792 dense_6[0][0]

add_2 (Add) (None, 256) 0 dense_1[0][0]
dense_7[0][0]

dense_8 (Dense) (None, 256) 65536 add_2[0][0]

gather_atom_to_bond_3 (GatherAt (None, 256) 0 dense_8[0][0]
connectivity[0][0]

gather_atom_to_bond_4 (GatherAt (None, 256) 0 dense_8[0][0]
connectivity[0][0]

concatenate_2 (Concatenate) (None, 768) 0 gather_atom_to_bond_3[0][0]
gather_atom_to_bond_4[0][0]
add_1[0][0]

dense_9 (Dense) (None, 512) 393728 concatenate_2[0][0]

dense_10 (Dense) (None, 256) 131328 dense_9[0][0]

dense_11 (Dense) (None, 256) 65792 dense_10[0][0]

dense_12 (Dense) (None, 256) 65792 dense_11[0][0]

add_3 (Add) (None, 256) 0 dense_12[0][0]
add_1[0][0]

multiply_2 (Multiply) (None, 256) 0 gather_atom_to_bond_3[0][0]
add_3[0][0]

reduce_bond_to_atom_2 (ReduceBo (None, 256) 0 multiply_2[0][0]
connectivity[0][0]

dense_13 (Dense) (None, 256) 65792 reduce_bond_to_atom_2[0][0]

dense_14 (Dense) (None, 256) 65792 dense_13[0][0]

add_4 (Add) (None, 256) 0 dense_8[0][0]
dense_14[0][0]

dense_15 (Dense) (None, 256) 65536 add_4[0][0]

gather_atom_to_bond_5 (GatherAt (None, 256) 0 dense_15[0][0]
connectivity[0][0]

gather_atom_to_bond_6 (GatherAt (None, 256) 0 dense_15[0][0]
connectivity[0][0]

concatenate_3 (Concatenate) (None, 768) 0 gather_atom_to_bond_5[0][0]
gather_atom_to_bond_6[0][0]
add_3[0][0]

dense_16 (Dense) (None, 512) 393728 concatenate_3[0][0]

dense_17 (Dense) (None, 256) 131328 dense_16[0][0]

dense_18 (Dense) (None, 256) 65792 dense_17[0][0]

dense_19 (Dense) (None, 256) 65792 dense_18[0][0]

add_5 (Add) (None, 256) 0 dense_19[0][0]
add_3[0][0]

multiply_3 (Multiply) (None, 256) 0 gather_atom_to_bond_5[0][0]
add_5[0][0]

reduce_bond_to_atom_3 (ReduceBo (None, 256) 0 multiply_3[0][0]
connectivity[0][0]

dense_20 (Dense) (None, 256) 65792 reduce_bond_to_atom_3[0][0]

dense_21 (Dense) (None, 256) 65792 dense_20[0][0]

atom_index (InputLayer) (None, 1) 0

n_pro (InputLayer) (None, 1) 0

add_6 (Add) (None, 256) 0 dense_15[0][0]
dense_21[0][0]

reduce_atom_to_pro_1 (ReduceAto (None, 256) 0 add_6[0][0]
squeeze_1[0][0]
squeeze_1[2][0]

dense_22 (Dense) (None, 256) 65792 reduce_atom_to_pro_1[0][0]

dense_23 (Dense) (None, 256) 65792 dense_22[0][0]

dense_24 (Dense) (None, 128) 32896 dense_23[0][0]

atomwise_shift (Embedding) (None, 1) 10 squeeze_1[1][0]

dense_25 (Dense) (None, 1) 129 dense_24[0][0]

reduce_atom_to_pro_2 (ReduceAto (None, 1) 0 atomwise_shift[0][0]
squeeze_1[0][0]
squeeze_1[2][0]

add_7 (Add) (None, 1) 0 dense_25[0][0]
reduce_atom_to_pro_2[0][0]

Total params: 2,728,459
Trainable params: 2,728,459
Non-trainable params: 0

2023-10-31 06:41:41.641883: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2023-10-31 06:41:41.773098: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-31 06:41:41.773275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.725
pciBusID: 0000:05:00.0
totalMemory: 23.69GiB freeMemory: 23.43GiB
2023-10-31 06:41:41.773292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2023-10-31 06:41:42.223463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-10-31 06:41:42.223495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2023-10-31 06:41:42.223501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2023-10-31 06:41:42.223603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22717 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:05:00.0, compute capability: 8.6)
{'n_pro': array([24, 21, 7, 27, 12, 10, 10, 20]), 'atom': array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 2, 3, 2, 2, 2, 2, 2,
2, 2, 2, 2, 3, 2, 2, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 3, 2, 2, 2, 2, 2, 2, 2,
2, 3, 2, 3, 2, 2, 3, 2, 3, 2, 2, 2, 3, 2, 2, 2, 4, 2, 4, 2, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 2, 2, 2, 2, 5, 3, 2, 2, 2, 3, 3, 4, 6, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 2, 2, 2, 2, 2, 2, 6, 2, 2, 2, 2, 2,
2, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 6, 2, 6, 2, 5, 2, 2, 2, 5,
2, 2, 2, 2, 2, 3, 3, 2, 2, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 6, 2, 6, 2, 5, 2, 2, 3, 2, 2, 2, 2, 5,
2, 2, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 2, 2, 3, 2, 2,
2, 3, 3, 3, 6, 2, 3, 2, 6, 6, 2, 2, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
6, 2, 2, 2, 2, 3, 3, 2, 2, 6, 6, 2, 2, 3, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4]), 'bond': array([2, 3, 4, ..., 0, 0, 0]), 'connectivity': array([[ 0, 31],
[ 0, 5],
[ 0, 1],
...,
[342, 298],
[342, 310],
[342, 322]]), 'atom_index': array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, -1,
-1, -1, 12, -1, 13, 14, 15, 16, 17, 18, 19, 20, 21,
-1, 22, 23, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, 24, -1, 25, 26, 27, 28, 29, 30,
31, 32, -1, 33, -1, 34, 35, -1, 36, -1, 37, 38, 39,
-1, 40, 41, 42, -1, 43, -1, 44, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 45, 46, 47,
48, -1, -1, 49, 50, 51, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 52, 53,
54, 55, 56, 57, 58, -1, 59, 60, 61, 62, 63, 64, -1,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77,
78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, 79, -1, 80, -1, 81,
82, 83, -1, 84, 85, 86, 87, 88, -1, -1, 89, 90, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, 91, -1, 92, -1,
93, 94, -1, 95, 96, 97, 98, -1, 99, 100, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 101, 102,
103, -1, 104, 105, 106, -1, -1, -1, -1, 107, -1, 108, -1,
-1, 109, 110, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 111, 112, 113,
114, 115, 116, 117, 118, 119, 120, 121, 122, -1, 123, 124, 125,
126, -1, -1, 127, 128, -1, -1, 129, 130, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1]), 'node_graph_indices': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7]), 'distance_rbf': array([[8.28731687e-06, 6.52437027e-05, 4.20537142e-04, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
[5.55421160e-09, 7.92017194e-08, 9.24672414e-07, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
[3.65965223e-09, 5.37801536e-08, 6.47061206e-07, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
...,
[6.50888555e-97, 7.20740461e-93, 6.53419777e-89, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
[4.27906304e-97, 4.78068100e-93, 4.37292421e-89, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
[3.26498268e-99, 4.04323416e-95, 4.09937841e-91, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00]])}
[[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]] (131, 1)
/root/miniconda3/envs/my-rdkit-env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Epoch 1/1200
139/807 [====>.........................] - ETA: 20s - loss: nan
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@xunfeng2zkj