Skip to content

Commit

Permalink
Fixed issues with nanovdb and hdf5 outputs
Browse files Browse the repository at this point in the history
  • Loading branch information
mattloulou committed Sep 9, 2024
1 parent 9bc6e61 commit d16eba0
Show file tree
Hide file tree
Showing 5 changed files with 78 additions and 6 deletions.
2 changes: 2 additions & 0 deletions cmake/LoadingExternalGitRepositories.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ if(${NEON_USE_NANOVDB})
COMMAND ${CMAKE_COMMAND} --build ${nanovdb_BINARY_DIR} --target install
WORKING_DIRECTORY ${nanovdb_SOURCE_DIR}
)

include_directories(${nanovdb_SOURCE_DIR}/nanovdb)
endif ()
endif ()

Expand Down
31 changes: 31 additions & 0 deletions docs/learn/io/01-NanoVDB-output.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# NanoVDB

## What is NanoVDB?

As the name indicates it's a mini-version of the much bigger OpenVDB library, both in terms of functionality and scope. In fact, a stand-alone C++11 implementation of NanoVDB is available in the file NanoVDB.h and the C99 equivalent in the files CNanoVDB.h, and PNanoVDB.h. However, NanoVDB offers one major advantage over OpenVDB, namely support for GPUs. In short, NanoVDB is a standalone static-topology implementation of the well-known sparse volumetric VDB data structure. In other words, while values can be modified in a NanoVDB grid its tree topology cannot.

Additionally, it also can have no external dependencies.

*The explanation is taken from their documentation*

## How do I compile a program that uses NanoVDB?:

When making Neon, you must cmake it with the setting ` -DNEON_USE_NANOVDB=ON `.

For example, you can make Neon with ` cmake -DNEON_USE_NANOVDB=ON .. `, which will then download and install NanoVDB on your computer.

## Where is the tool located?:

At Neon/libNeonCore/include/Neon/core/tools/io/ioToNanoVDB.h

## How do I use the tool?:

For the user, you simply have to instantiate the object `Neon::ioToNanoVDB`. At the time of writing, its constructor has 7 arguments.

1. `filename`: The name of the file you want to outupt to. For example, if it is `coolGrid`, the output file will be written to `coolGrid.nvdb`.
2. `dim`: The dimension of the output. If you set it to `(10, 10, 10)`, there will be 1000 datapoints outputted.
3. `fun`: This is an anonymous function which takes in an index and a cardinality (of types `Neon::Integer_3d<intType_ta>` and `int`, respectively), and should output the value you want to be stored at the corresponding index in the output. This function allows this tool to access internal values for your grid/field in the way you specify.
4. `card`: The cardinality of the output. Currently, only cardinalities of `1`, `3`, or `4` are supported.
5. `scalingData`: This is a scalar which scales the voxels in the output by the amount given.
6. `origin`: This is the index where the output starts at. The indices stored will be from `origin` to `origin + dim - (1,1,1)`.
7. `mask`: The anonymous function detailing which indices inside the inclusive range [`origin`, `origin + dim - (1,1,1)`] should be included in the output. This allows for sparse matrices to be stored. It should return `true` for indices that should be outputted, and `false` for those that shouldn't. You only need to consider indices inside that range.
39 changes: 39 additions & 0 deletions docs/learn/io/02-HDF5-output.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# HDF5

## What is HDF5?

Hierarchical Data Format (HDF) is a set of file formats (HDF4, HDF5) designed to store and organize large amounts of data. Originally developed at the U.S. National Center for Supercomputing Applications, it is supported by The HDF Group, a non-profit corporation whose mission is to ensure continued development of HDF5 technologies and the continued accessibility of data stored in HDF.

*The definition was taken from Wikipedia https://en.wikipedia.org/wiki/Hierarchical_Data_Format*


## What is HighFive?

It is a modern header-only C++11 friendly interface for libhdf5. It is used in this context for HDF5 output.

You can read its documentation here: https://bluebrain.github.io/HighFive/

## How do I compile a program that uses HDF5?:

When making Neon, you must cmake it with the setting ` -NEON_USE_HDF5=ON `.

For example, you can make Neon with ` cmake -NEON_USE_HDF5=ON .. `, which will then download and install HighFive on your computer

You must also have HDF5 and Boost installed. For Ubuntu, you can do `sudo apt install libhdf5-dev` and `sudo apt install libboost-all-dev`

## Where is the tool located?:

At Neon/libNeonCore/include/Neon/core/tools/io/ioToHDF5.h

## How do I use the tool?:

For the user, you simply have to instantiate the object `Neon::ioToHDF5`. At the time of writing, its constructor has 7 arguments.

1. `filename`: The name of the file you want to outupt to. For example, if it is `coolGrid`, the output file will be written to `coolGrid.nvdb`.
2. `dim`: The dimension of the output. If you set it to `(10, 10, 10)`, there will be 1000 datapoints outputted.
3. `fun`: This is an anonymous function which takes in an index and a cardinality (of types `Neon::Integer_3d<intType_ta>` and `int`, respectively), and should output the value you want to be stored at the corresponding index in the output. This function allows this tool to access internal values for your grid/field in the way you specify.
4. `card`: The cardinality of the output. Currently, only cardinalities of `1`, `3`, or `4` are supported.
5. `scalingData`: This is a scalar which scales the voxels in the output by the amount given.
6. `origin`: This is the index where the output starts at. The indices stored will be from `origin` to `origin + dim - (1,1,1)`.
7. `chunking`: This is an integer 3d which stores the dimensions for which the HDF5 output should be chunked. You can play around with it to get different optimization results based on your use case.
8. `mask`: The anonymous function detailing which indices inside the inclusive range [`origin`, `origin + dim - (1,1,1)`] should be included in the output. This allows for sparse matrices to be stored. It should return `true` for indices that should be outputted, and `false` for those that shouldn't. You only need to consider indices inside that range.
6 changes: 3 additions & 3 deletions libNeonCore/include/Neon/core/tools/io/ioToHDF5.h
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,9 @@ void ioToHDF5(const ioToHDF5ns::UserFieldInformation<intType_ta, real_tt>&
for (int i = origin.x; i < origin.x + dim.x; ++i) {
for (int j = origin.y; j < origin.y + dim.y; ++j) {
for (int k = origin.z; k < origin.z + dim.z; ++k) {
if (!mask(Neon::Integer_3d<intType_ta>(i, j, k)) {
if (!mask(Neon::Integer_3d<intType_ta>(i, j, k))) {
dataset.select({i - origin.x, j - origin.y, k - origin.z}, {1, 1, 1}).write(fieldData.m_userFieldAccessGenericFunction(Neon::Integer_3d<intType_ta>(i, j, k), 0));
})
}
}
}
}
Expand All @@ -121,7 +121,7 @@ struct ioToHDF5
{
ioToHDF5(const std::string& filename /*! File name */,
const Neon::Integer_3d<intType_ta>& dim /*! IoDense dimension of the field */,
const std::function<real_tt(const Neon::Integer_3d<intType_ta>&, int componentIdx)>& fun /*! Implicit defintion of the user field */,
const ioToHDF5ns::UserFieldAccessGenericFunction_t<real_tt, intType_ta>& fun /*! Implicit defintion of the user field */,
const nComponent_t card /*! Field cardinality */,
const double scalingData = 1.0 /*! Spacing, i.e. size of a voxel */,
const Neon::Integer_3d<intType_ta>& origin = Neon::Integer_3d<intType_ta>(0, 0, 0) /*! Minimum Corner && Origin */,
Expand Down
6 changes: 3 additions & 3 deletions libNeonCore/include/Neon/core/tools/io/ioToNanoVDB.h
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@


#include <nanovdb/NanoVDB.h>
#include <nanovdb/util/CreateNanoGrid.h>
#include <nanovdb/util/IO.h>
#include <nanovdb/tools/CreateNanoGrid.h>
#include <nanovdb/io/IO.h>
#include <nanovdb/tools/GridBuilder.h>
#include <nanovdb/tools/CreateNanoGrid.h>

Expand Down Expand Up @@ -232,7 +232,7 @@ struct ioToNanoVDB
{
ioToNanoVDB(const std::string& filename /*! File name */,
const Neon::Integer_3d<intType_ta>& dim /*! IoDense dimension of the field */,
const std::function<real_tt(const Neon::Integer_3d<intType_ta>&, int componentIdx)>& fun /*! Implicit defintion of the user field */,
const ioToNanoVDBns::UserFieldAccessGenericFunction_t<real_tt, intType_ta>& fun /*! Implicit defintion of the user field */,
const nComponent_t card /*! Field cardinality */,
const double scalingData = 1.0 /*! Spacing, i.e. size of a voxel */,
const Neon::Integer_3d<intType_ta>& origin = Neon::Integer_3d<intType_ta>(0, 0, 0) /*! Minimum Corner && Origin */,
Expand Down

0 comments on commit d16eba0

Please sign in to comment.