Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentation about using files in memory #1118

Merged
merged 4 commits into from
Sep 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,36 @@ mtx_dset = create_dataset(g, "M", datatype(Float64), dataspace(100, 1000),
mtx = HDF5.readmmap(mtx_dset) # succeeds immediately
```

## In-memory HDF5 files

It is possible to use HDF5 files without writing or reading from disk. This is useful when receiving or sending data over the network. Typically, when sending data, one might want to
1) Create a new file in memory. This can be achieved by passing `Drivers.Core(; backing_store=false)` to `h5open(...)`
2) Add data to the `HDF5.File` object
3) Get a representation of the file as a byte vector. This can be achieved by calling `Vector{UInt8}(...)` on the file object.

This is illustrated on the example below
```julia
using HDF5

# Creates a new file object without storing to disk by setting `backing_store=false`
file_as_bytes = h5open("AnyName_InMemory", "w"; driver=Drivers.Core(; backing_store=false)) do fid
fid["MyData"] = randn(5, 5) # add some data
return Vector{UInt8}(fid) # get a byte vector to send, e.g., using HTTP, MQTT or similar.
end
```

The same way, when receiving data as a vector of bytes that represent a HDF5 file, one can use `h5open(...)` with the byte vector as first argument to get a file object.
Creating a file object from a byte vector will also by default open the file in memory, without saving a copy on disk.
```julia
using HDF5

...
h5open(file_as_bytes, "r"; name = "in_memory.h5") do fid
... # Do things with the data
end
...
```

## Supported data types

`HDF5.jl` knows how to store values of the following types: signed and unsigned integers of 8, 16, 32, and 64 bits, `Float32`, `Float64`; `Complex` versions of these numeric types; `Array`s of these numeric types (including complex versions); `String`; and `Array`s of `String`.
Expand Down
2 changes: 1 addition & 1 deletion src/drivers/drivers.jl
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ const DRIVERS = Dict{API.hid_t,Type{<:Driver}}()
# Arguments

* `increment`: specifies the increment by which allocated memory is to be increased each time more memory is required. (default: 8192)
* `backing_store`: Boolean flag indicating whether to write the file contents to disk when the file is closed. (default: false)
* `backing_store`: Boolean flag indicating whether to write the file contents to disk when the file is closed. (default: true)
* `write_tracking`: Boolean flag indicating whether write tracking is enabled. (default: false)
* `page_size`: Size, in bytes, of write aggregation pages. (default: 524288)
"""
Expand Down