Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to Data Packaging Service #119

Closed
wants to merge 4 commits into from
Closed

Conversation

elmiomar
Copy link
Contributor

@elmiomar elmiomar commented Dec 19, 2024

This PR updates to the Distribution Service, particularly the Data Packaging Service, to address the feedback from Greg.

Changes

  1. Update Max Zip Size
  • The distrib.packaging.maxpackagesize parameter already handles requested zip size. (Current setting is 4GB)
  1. File Name Updates for RPA
  • Renamed zip file to: NIST-Data-[timestamp]. Timestamp is in a human readable format indicating the date and time of download.
  • Note: This change is implemented in the datacart frontend code and will have its own PR.
  1. Status Renaming
  • Renamed status filenames for clarity:
    • PackagingSuccessful → DownloadSuccessful
    • PackagingErrors → DownloadErrors
  • Replaced all occurrences of "Packaging" with "Download".
  1. Update Package Hierarchy
  • Removed ark:/88434/ from the package hierarchy.
  • New hierarchy: The dataset name makes a folder at the root of the zip file, with data files organized within this folder.
  • New logic:
    • If the dataset name contains ark, the dataset ID is used as the folder name.
    • If the dataset ID is a 32-34 character string (e.g., "3A1EE2F169DD3B8CE0531A570681DB5D1491"), only the last 2 or 4 characters are retained and prefixed with "mds2-00" or "mds2-".

Testing:

  • All updates were tested locally using oar-docker to verify the changes work as intended.

@elmiomar elmiomar requested a review from RayPlante December 19, 2024 14:35
@elmiomar elmiomar self-assigned this Dec 19, 2024
RayPlante added a commit that referenced this pull request Jan 19, 2025
Combine PRs #113, #116, and #119:  
  * Migrate code to Java 21, JUnit 5, and Spring 3
  * Support pre-approved RPA
  * update data packaging for more human-friendly zip hierarchy
@RayPlante
Copy link
Collaborator

This PR has been deprecated by #117 which combines these changes with the migration to Java 21/Spring 3; closing without merging.

@RayPlante RayPlante closed this Jan 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants