Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CRUD operations #14

Open
carlogambi opened this issue Mar 28, 2022 · 2 comments
Open

CRUD operations #14

carlogambi opened this issue Mar 28, 2022 · 2 comments

Comments

@carlogambi
Copy link

carlogambi commented Mar 28, 2022

Hi all,
I’m trying to do CRUD OPS in order to manage and populate a dataset: writing and reading on it are straightforwared, but i cannot understand how to update or delete attributes, group or dataset.
I’m working with a stream of data in a nest js server, where elements are:

{
_id,
timestamp,
value
}

I need to create a dataset for every _id that contains timestamp and value (1st and 2nd columns in a n x 2 table).
I’m working with a big volume of data, so i cannot collect everything server-side before creating the dataset, so i need to create the dataset and update it at every step.
As my last point, i'm not aware if the creation of the dataset using dynamic length is possible; but I can retrieve the final length of the dataset.
Kind Regards, Carlo

@bmaranville
Copy link
Member

Hello, Carlo -
Currently there is no resizing or appending possible with h5wasm. The only available option right now is to create a (whole, finished) HDF5 dataset from an existing array of values.

It's certainly possible to implement using parts of the HDF5 API, which provides a way of making resizable datasets - but it is not currently implemented. Implementation would involve:

at Dataset creation:

  • setting chunksize at dataset creation
  • setting the maxsize for each dimension (could include axes with unlimited max size)

when writing a point:

  • calling H5Dextend to resize the dataset
  • writing to a slice for the last point

None of these features is currently implemented in h5wasm (slice is enabled for reading, but not writing - and no option for passing chunk sizes, maxsize are used in the create_dataset function)

Out of curiosity, are you hoping to do this in the browser (where you would be accumulating points in memory no matter how you do it) or in node.js, where you could write directly to disk?

@bmaranville
Copy link
Member

You can create resizable datasets (must specify chunks and maxshape for create_dataset), and you can resize resizable datasets, and you can overwrite sections of data (so if you resize, you can write to the new extended region) in v0.4.11 released just now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants