New Backends, Extension, Perf. & Memory

ADIOS2 Joined arrays, API simplification, HDF5 subfiling, TOML backend, wildcard file extensions, Performance & Memory: This release adds support for additional I/O features in all backends, and additionally a completely new TOML backend, useful for openPMD-formatted configuration files in scientific workflows. The ADIOS1 backend, previously deprecated, has been removed in favor of ADIOS2.

The openPMD-api has been simplified by no longer requiring explicit specification for scalar components via RecordComponent::SCALAR in its object model. The addition of wildcard filename extensions (e.g. simData.%E or simData_%T.%E) simplifies the generic implementation of backend-independent logic, such as post-processing routines.

Thanks to Franz Pöschel @franzpoeschel, Axel Huebl @ax3l, David Grote @dpgrote, Davide Terzani @titoiride, Ilian Kara-Mostefa @IlianCS, Junmin Gu @guj, Luca Fedeli @lucafedeli88, Nils Schild @DerNils-git, Paweł Ordyna @pordyna and Ulrik Günther @skalarproduktraum for contributing to this release!

New Contributors

• @dpgrote made their first contribution in #1442
• @skalarproduktraum made their first contribution in #1364
• @pordyna made their first contribution in #1477
• @IlianCS made their first contribution in #1510
• @lucafedeli88 made their first contribution in #1532
• @titoiride made their first contribution in #1558

Python, ADIOS2 and HDF5 Fixes

This release fixed regressions in the Python frontend as well as the ADIOS2 and HDF5 backends.
Supported macOS versions are now 11.0+ and Python versions are 3.8+.

Thanks to Franz Poeschel, Axel Huebl, David P. Grote, Ulrik Günther and Ilian Kara-Mostefa for contributing to this release!

Build Regressions

This release fixes build regressions and minor documentation updates for the 0.15.0 release.

Thanks to Axel Huebl and Franz Poeschel for contributing to this release!

C++17, Error Recovery, ADIOS2 BP5, Append & Read-Linear Modes, Performance & Memory

This release adds error recovery mechanisms, in order to access erroneous datasets, created e.g. by crashing simulations. The BP5 engine of ADIOS2 v2.9 is fully supported by this release, including access to its various features for more fine-grained control of memory usage. Various I/O performance improvements for HDF5 are activated by default. Runtime configuration of openPMD and its backends, e.g. selection of backends and compression, is now consistently done via JSON, and alternatively via TOML for better readability. The data storage/retrieval API now consistently supports all common C++ pointer types (raw and smart pointers), implementing automatic memory optimizations for ADIOS2 BP5 if using unique pointers.

The miminum required C++ version is now C++17. Supported Python versions add Python 3.10 and 3.11.

Thanks to Axel Huebl, Franz Poeschel, Jean Luc Bez, Junmin Gu, Bernhard Manfred Gruber, and Nils Schild for contributing to this release!

Improve Series Parsing, Python & Fix Backend Bugs

This release improves reading back iterations that overflow the specified zero-pattern. ADIOS1, ADIOS2 and HDF5 backend stability and performance were improved. Python bindings got additional wheel platform support and various smaller issues were fixed.

Thanks to Axel Huebl, Franz Poeschel, and Jean Luca Bez for contributing to this release!

Increased Compatibility & Python Install Bug

This release fixes various read/parsing bugs and increases compatibility with upcoming versions of ADIOS and old releases of Intel icpc. An installation issue for pip-based installs from source in the last release was fixed and Python 3.10 support added. Various documentation and installation warnings have been fixed.

Thanks to Axel Huebl and Franz Poeschel for contributing to this release!

Read Bugs, C++17 Mixing and HDF5 Performance

This release makes reads more robust by fixing small API, file-based parsing and test bugs. Building the library in C++14 and using it in C++17 will not result in incompatible ABIs anymore. HDF5 1.10.1+ performance was improved significantly.

Thanks to Axel Huebl, Franz Poeschel and Jean Luca Bez for contributing to this release!

Various Reader Fixes

This releases fixes regressions in reads, closing files properly, avoiding inefficient parsing and allowing more permissive casts in attribute reads. (Inofficial) support for HDF5 vlen string reads has been fixed.

Thanks to Franz Poeschel and Axel Huebl for contributing to this release!

ADIOS2 Close Regressions & ADIOS1 Build

Fix a regression with file handling for ADIOS2 when using explicit close logic, especially with interleaved writes to multiple iterations. Also fix an issue with ADIOS1 builds that potentially picked up headers from older, installed openPMD-api versions.

Thanks to Franz Poeschel and Axel Huebl for contributing to this release!

Resize, Dask, openpmd-pipe and new ADIOS2 Iteration Encoding

This release adds support for resizable data sets. For data-processing, support for Dask (parallel) and Pandas (serial) are added and lazy reader parsing of iterations is now supported. ADIOS2 adds an experimental variable-based iteration encoding. An openPMD Series can now be flushed from non-Series objects and write buffers can be requested upfront to avoid unnecessary data copies in some situations.

Thanks to Axel Huebl, Franz Poeschel, Erik Schnetter, Junmin Gu, John Kirkham, and Dmitry Ganyushin for contributing to this release!