non-deterministic data values returned when reading from a BP5 file #1571

BenWibking · 2023-12-15T22:31:41Z

BenWibking
Dec 15, 2023

Describe the bug
I get different, incorrect values when reading a BP5 file in Python I created using the C++ API.
I also get different values each time I run the script to read the data(!).

To Reproduce
Compile-able/executable code example to reproduce the problem:

Python:

import openpmd_api as io

series = io.Series("plt00000.bp", io.Access.read_only)
my_iter = series.iterations[0]
mesh = my_iter.meshes
comp = mesh['gasDensity']['\x0bScalar']

my_list = list(comp)

print("first try:")
# this gives one value
print(comp[0])

print("second try (produces different output from the above, and different on each run):")
# this is different from the above
print(comp[0])

Reading from these attached files:
quokka_bp5.tar.gz

Expected behavior
It should produce the same output every time it's run!

Software Environment

version of openPMD-api: 0.15.2
installed openPMD-api via: brew
operating system: macOS 14.2
machine: MacBook Pro (Apple M1 Pro)
name and version of Python implementation: CPython 3.12.1
version of HDF5: 1.14.3
version of ADIOS2: 2.9.2
name and version of MPI: OpenMPI 5.0.0

Additional context
Written using Quokka: quokka-astro/quokka@f7f34fc

Answered by ax3l

Dec 16, 2023

Hi @BenWibking ,

I think you forgot to flush the read:
https://openpmd-api.readthedocs.io/en/0.15.2/usage/firstread.html#register-chunk

import openpmd_api as io

series = io.Series("plt00000.bp", io.Access.read_only)
my_iter = series.iterations[0]
mesh = my_iter.meshes

# note 1: do not use our magic value :D
# note 2: select the data you want to read
comp = mesh['gasDensity'][io.Mesh_Record_Component.SCALAR][()]

# add this to read the selected data from disk
series.flush()

my_list = list(comp)

print("first try:")
# this gives one value
print(comp[0])

print("second try (produces different output from the above, and different on each run):")
# this is different from the above
print(comp[0

View full answer

BenWibking · 2023-12-15T22:32:39Z

BenWibking
Dec 15, 2023
Author

Example output from the above script:

✗ ipython read_openpmd.py
first try:
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0.]
second try (produces different output from the above, and different on each run):
[ 0.00000000e+000 -9.45745593e-124  0.00000000e+000  2.11222945e-319
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  2.16797252e-314
  2.09483834e-321  2.25878313e-314  2.16113605e-314  2.16787765e-314
  2.16117536e-314  2.16117533e-314  2.16117543e-314  2.16113558e-314
  2.16113603e-314  2.16113607e-314  2.16117546e-314  2.16117539e-314
  2.16117549e-314  2.16113631e-314  2.16117552e-314  2.16113650e-314
  2.16117555e-314  2.16113648e-314  2.16113657e-314  2.16113660e-314
  2.16113662e-314  2.16117558e-314  2.16117562e-314  2.16113664e-314
  2.16113667e-314  2.16117565e-314  2.16113669e-314  2.16113671e-314
  2.16117568e-314  2.16117571e-314  2.16117574e-314  2.16117577e-314
  2.16117581e-314  2.16113674e-314  2.16113676e-314  2.16113679e-314
  2.16113681e-314  2.16117584e-314  2.16113683e-314  2.16113686e-314
  2.16113688e-314  2.16113690e-314  2.16113693e-314  2.16113695e-314
  2.16113698e-314  2.16113700e-314  2.16113702e-314  2.16113705e-314
  2.16113707e-314  2.16117587e-314  2.16113709e-314  2.16113712e-314
  2.16117590e-314  2.16113714e-314  2.16113717e-314  2.16117593e-314
  2.16113719e-314  2.16113721e-314  2.16117596e-314  2.16113724e-314
  2.16113726e-314  2.16117600e-314  2.16113728e-314  2.16113731e-314
  2.16117603e-314  2.16113733e-314  2.16113736e-314  2.16117606e-314
  2.16113738e-314  2.16113740e-314  2.16117609e-314  2.16113743e-314
  2.16113745e-314  2.16117612e-314  2.16113747e-314  2.16113750e-314
  2.16113752e-314  2.16117615e-314  2.16113754e-314  2.16113757e-314
  2.16117618e-314  2.16117622e-314  2.16113759e-314  2.16113762e-314
  2.16113764e-314  2.16117625e-314  2.16113766e-314  2.16113769e-314
  2.16113771e-314  2.16117628e-314  2.16117631e-314  2.16117634e-314
  2.16113773e-314  2.16113776e-314  2.16113778e-314  2.16117637e-314
  2.16113781e-314  2.16113783e-314  2.16113785e-314  2.16117641e-314
  2.16113788e-314  2.16113790e-314  2.16113792e-314  2.16117644e-314
  2.16113795e-314  2.16113797e-314  2.16113800e-314  2.16117647e-314
  2.16113802e-314  2.16113804e-314  2.16113807e-314  2.16117650e-314
  2.16113809e-314  2.16113811e-314  2.16113814e-314  2.16117653e-314
  2.16113816e-314  2.16113819e-314  2.16113821e-314  2.16117656e-314
  2.16113823e-314  2.16113826e-314  2.16113828e-314  2.16113830e-314
  2.16117660e-314  2.16113833e-314  2.16113835e-314  2.16113837e-314
  2.16117663e-314  2.16113840e-314  2.16113842e-314  2.16113845e-314
  2.16117666e-314  2.16113847e-314  2.16113849e-314  2.16113852e-314
  2.16117669e-314  2.16113854e-314  2.16113856e-314  2.16113859e-314
  2.16117672e-314  2.16113861e-314  2.16113864e-314  2.16113866e-314
  2.16117675e-314  2.16113868e-314  2.16113871e-314  2.16113873e-314
  2.16113875e-314  2.16117679e-314  2.16113878e-314  2.16113880e-314
  2.16113883e-314  2.16113885e-314  2.16113887e-314  2.16117682e-314
  2.16113890e-314  2.16113892e-314  2.16113894e-314  2.16113897e-314
  2.16113899e-314  2.16113902e-314  2.16113904e-314  2.16117685e-314
  2.16117688e-314  2.16117691e-314  2.16117694e-314  2.16117698e-314
  2.16117701e-314  2.16113906e-314  2.16113909e-314  2.16113911e-314
  2.16113913e-314  2.16113916e-314  2.16113918e-314  2.16113920e-314
  2.16113923e-314  2.16117704e-314  2.16117707e-314  2.16113925e-314
  2.16117710e-314  2.16117713e-314  2.16113928e-314  2.16113930e-314
  2.16113932e-314  2.16117717e-314  2.16113935e-314  2.16113937e-314
  2.16117720e-314  2.16117723e-314  2.16117726e-314  2.16117729e-314
  2.16113939e-314  2.16113942e-314  2.16113944e-314  2.16113947e-314
  2.16068901e-314  2.16117732e-314  2.16113949e-314  2.16117735e-314
  2.16113951e-314  2.16117739e-314  2.16113954e-314  2.16113956e-314
  2.16113958e-314  2.16113961e-314  2.16113963e-314  2.16117742e-314
  2.16117745e-314  2.16117748e-314  2.16117751e-314  2.16117754e-314
  2.16117758e-314  2.16117761e-314  2.16117764e-314  2.16117767e-314
  2.16117770e-314  2.16117773e-314  2.16117777e-314  2.16117780e-314
  2.16117783e-314  2.16117786e-314  2.16117789e-314  2.16117792e-314
  2.16117796e-314  2.16117799e-314  2.16117802e-314  2.16117805e-314
  2.16117808e-314  2.16117811e-314  2.16117815e-314  2.16117818e-314
  2.16117821e-314  2.16117824e-314  2.16117827e-314  2.16117830e-314
  2.16113966e-314  2.16113968e-314  2.16113970e-314  2.16117833e-314
  2.16117837e-314  2.16117840e-314  2.16117843e-314  2.16117846e-314
  2.16113973e-314  2.16113975e-314  2.16117849e-314  2.16117852e-314
  2.16117856e-314  2.16117859e-314  2.16117862e-314  2.16113977e-314
  2.16113980e-314  2.16117865e-314  2.16117868e-314  2.16113982e-314
  2.16113985e-314  2.16117871e-314  2.16117875e-314  2.16117878e-314
  2.16117881e-314  2.16113987e-314  2.16113989e-314  2.16117884e-314
  2.16117887e-314  2.16117890e-314  2.16117894e-314  2.16117897e-314
  2.16113992e-314  2.16113994e-314  2.16117900e-314  2.16117903e-314
  2.16117906e-314  2.16117909e-314  2.16117913e-314  2.16113996e-314
  2.16113999e-314  2.16117916e-314  2.16117919e-314  2.16117922e-314
  2.16117925e-314  2.16117928e-314  2.16114001e-314  2.16114004e-314
  2.16117932e-314  2.16117935e-314  2.16117938e-314  2.16117941e-314
  2.16117944e-314  2.16117947e-314  2.16114006e-314  2.16117950e-314
  2.16117954e-314  2.16117957e-314  2.16117960e-314  2.16117963e-314
  2.16117966e-314  2.16117969e-314  2.16117973e-314  2.16117976e-314
  2.16117979e-314  2.16117982e-314  2.16114008e-314  2.16114011e-314
  2.16117985e-314  2.16117988e-314  2.16117992e-314  2.16117995e-314
  2.16117998e-314  2.16114013e-314  2.16118001e-314  2.16118004e-314
  2.16118007e-314  2.16118011e-314  2.16118014e-314  2.16118017e-314
  2.16118020e-314  2.16118023e-314  2.16114015e-314  2.16114018e-314
  2.16114020e-314  2.16114022e-314  2.16114025e-314  2.16114027e-314
  2.16114030e-314  2.16114032e-314  2.16114034e-314  2.16114037e-314
  2.16118026e-314  2.16114039e-314  2.16114041e-314  2.16114044e-314
  2.16118030e-314  2.16114046e-314  2.16114049e-314  2.16118033e-314
  2.16118036e-314  2.16118039e-314  2.16118042e-314  2.16114051e-314
  2.16114053e-314  2.16825740e-314  2.16118045e-314  2.16118049e-314
  2.16118052e-314  2.16118055e-314  2.16118058e-314  2.16118061e-314
  2.16118064e-314  2.16118067e-314  2.16118071e-314  2.16118074e-314
  2.16118077e-314  2.16118080e-314  2.16118083e-314  2.16118086e-314
  2.16118090e-314  2.16114056e-314  2.16113536e-314  2.16114058e-314
  2.16118093e-314  2.16114060e-314  2.16114063e-314  2.16114065e-314
  2.16118096e-314  2.16114068e-314  2.16118099e-314  2.16118102e-314
  2.16118105e-314  2.16114070e-314  2.16114072e-314  2.16118109e-314
  2.16118112e-314  2.16118115e-314  2.16114075e-314  2.16114077e-314
  2.16118118e-314  2.16118121e-314  2.16118124e-314  2.16118128e-314
  2.16118131e-314  2.16118134e-314  2.16118137e-314  2.16118140e-314
  2.16114079e-314  2.16114082e-314  2.16118143e-314  2.16118147e-314
  2.16118150e-314  2.16118153e-314  2.16114084e-314  2.16114087e-314
  2.16114089e-314  2.16118156e-314  2.16118159e-314  2.16118162e-314
  2.16118166e-314  2.16118169e-314  2.16118172e-314  2.16114091e-314
  2.16114094e-314  2.16114096e-314  2.16118175e-314  2.16118178e-314
  2.16118181e-314  2.16118184e-314  2.16114098e-314  2.16114101e-314
  2.16114103e-314  2.16118188e-314  2.16114105e-314  2.16114108e-314
  2.16114110e-314  2.16114113e-314  2.16118191e-314  2.16118194e-314
  2.16114115e-314  2.16114117e-314  2.16118197e-314  2.16778611e-314
  2.16118200e-314  2.16787853e-314  2.16118203e-314  2.16113265e-314
  2.16797926e-314  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000
  0.00000000e+000  0.00000000e+000  0.00000000e+000  0.00000000e+000]

0 replies

BenWibking · 2023-12-15T22:33:11Z

BenWibking
Dec 15, 2023
Author

Example output from another run of the above script:

✗ ipython read_openpmd.py
first try:
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0.]
second try (produces different output from the above, and different on each run):
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0.]

0 replies

BenWibking · 2023-12-15T22:36:32Z

BenWibking
Dec 15, 2023
Author

This seems like an uninitialized value bug...

0 replies

ax3l · 2023-12-16T00:13:32Z

ax3l
Dec 16, 2023
Maintainer

Hi @BenWibking ,

I think you forgot to flush the read:
https://openpmd-api.readthedocs.io/en/0.15.2/usage/firstread.html#register-chunk

import openpmd_api as io

series = io.Series("plt00000.bp", io.Access.read_only)
my_iter = series.iterations[0]
mesh = my_iter.meshes

# note 1: do not use our magic value :D
# note 2: select the data you want to read
comp = mesh['gasDensity'][io.Mesh_Record_Component.SCALAR][()]

# add this to read the selected data from disk
series.flush()

my_list = list(comp)

print("first try:")
# this gives one value
print(comp[0])

print("second try (produces different output from the above, and different on each run):")
# this is different from the above
print(comp[0])

4 replies

BenWibking Dec 16, 2023
Author

Maybe it could return an exception when it's accessed before a flush()?

ax3l Dec 16, 2023
Maintainer

I am not sure that is possible, because we literally allocate and return a numpy array and just hold a view on it...

But I am definitely eager to improve this part of the API with better ideas, too error prone. The features we would like to keep:

allow to write to pre-allocated memory
async if desired

Maybe something similar to a C++ future via asyncio would work as well...

franzpoeschel Dec 16, 2023
Maintainer

This is the most common problem users face when using our Python API, and by now I consider it a design error on our end that the Python API even gives you access to uninitialized memory.

Suggestion:

Default mode: synchronous. Every load_chunk() call or similar does a flush and you immediately get the values you want.
Opt-in mode: async. load_chunk() does not even return a numpy array, but a request handle instead:
```
data_pending = E_x[:, :, :]
np_array = data_pending.request()
```
This goes very close to the std::future thing that Axel mentioned.

In both modes, any Numpy array that the user sees contains valid data.
This would also solve another common pitfall:

arr = np.zero(...)
# load into allocated buffer
arr[:, :, :] = E_x[:, :, :]
# better though: E_x.load_chunk(arr[:, :, :])

This pattern currently gives garbage data since E_x[:, :, :] is not flushed yet, and there's nothing we can do about it, since both sides of the assignment are Numpy arrays.

The suggestion is slightly breaking since it turns load_chunk() into a potentially synchronous call.

BenWibking Dec 16, 2023
Author

An async API based on futures or request handles would be very nice to have.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

non-deterministic data values returned when reading from a BP5 file #1571

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

non-deterministic data values returned when reading from a BP5 file #1571

BenWibking Dec 15, 2023

Replies: 4 comments · 4 replies

BenWibking Dec 15, 2023 Author

BenWibking Dec 15, 2023 Author

BenWibking Dec 15, 2023 Author

ax3l Dec 16, 2023 Maintainer

BenWibking Dec 16, 2023 Author

ax3l Dec 16, 2023 Maintainer

franzpoeschel Dec 16, 2023 Maintainer

BenWibking Dec 16, 2023 Author

BenWibking
Dec 15, 2023

Replies: 4 comments 4 replies

BenWibking
Dec 15, 2023
Author

BenWibking
Dec 15, 2023
Author

BenWibking
Dec 15, 2023
Author

ax3l
Dec 16, 2023
Maintainer

BenWibking Dec 16, 2023
Author

ax3l Dec 16, 2023
Maintainer

franzpoeschel Dec 16, 2023
Maintainer

BenWibking Dec 16, 2023
Author