Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify Dataset Class #370

Open
jadball opened this issue Jan 10, 2025 · 2 comments
Open

Simplify Dataset Class #370

jadball opened this issue Jan 10, 2025 · 2 comments

Comments

@jadball
Copy link
Contributor

jadball commented Jan 10, 2025

The dataset class/file has, at a minimum:

  • Data
    • Motor positions
  • File paths
    • dataroot
    • analysisroot
    • sample name
    • dataset name

Right now, we include a bunch of other stuff:

  • File paths to processed data
    • Messy and not very phase-aware, don't like this at all right now
  • spatial files
  • parameter files
    • pointer to json
      • itself a pointer to .par files

Additionally the H5 file is not human-readable as text...
Instead I propose this:

  • Dataset file
    • This becomes ONLY the data needed to construct a sinogram (dty/omega positions)
  • Single Analysis Schema / Project Schema / Whatever file

This can be partially autogenerated during the acquisition stage or when the first notebook (/ewoks) is run. Detector parameters filled in on the BLISS side, etc

The file paths issue is then solved by keeping the notebooks in the same folder as the files we create, so relative paths only are needed
If you make a peaks1.h5 and a peaks2.h5, you just need to update the indexing notebook to say which peaks file you want to use
This means we can copy the entire analysispath to different locations and nothing breaks.

@jadball
Copy link
Contributor Author

jadball commented Jan 10, 2025

Tasks needed:

  • Remove file management stuff from the dataset class
  • Update notebooks accordingly

@jadball
Copy link
Contributor Author

jadball commented Jan 10, 2025

Should wait until #371 is complete before we do this.

@jadball jadball changed the title Processing file management overhaul Simplify Dataset Class Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant