Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Implement retrieval from LTS #59

Open
phwissmann opened this issue Jun 6, 2024 · 1 comment
Open

[Feature] Implement retrieval from LTS #59

phwissmann opened this issue Jun 6, 2024 · 1 comment
Assignees
@phwissmann
Labels
enhancement New feature or request
Milestone
Milestone I

Comments

@phwissmann
Copy link
Collaborator

phwissmann commented Jun 6, 2024 •
edited
Loading

Description
Based on the flow implemented in #16 the actual retrieval from the LTS can be implemented. As a user I want to schedule a job to retrieve an existing dataset.

Definition of Done

  • Flow deployed on scopem-openem.ethz.ch
  • manual test for successful retrieval

Additional context
Notes from meeting with storage team:

  • limit of 3 parallel reads for all LTS jobs (all of ETHZ)
    • sequential jobs
  • steps
    • file available on nfs share (with T flag)
    • start copy with file path
    • tape will be mounted
    • file available in nfs share
      • T-flag disappears
    • copy will begin
  • T flag indicates if file is on tape but not on landing zone
@phwissmann phwissmann added the enhancement New feature or request label Jun 6, 2024
@phwissmann phwissmann self-assigned this Jun 6, 2024
@phwissmann phwissmann changed the title [Feature] Implement and test retrieval against LTS [Feature] Implement retrieval from LTS Jun 6, 2024
@phwissmann phwissmann added this to the Milestone I milestone Jun 27, 2024
@phwissmann
Copy link
Collaborator Author

phwissmann commented Jul 2, 2024 •
edited
Loading

  • Concurrency limit not respected (needs deployment step)
  • Add target tarball size as variable
  • Test large data set
  • minio url cannot have path

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phwissmann phwissmann

Labels
enhancement New feature or request
Projects
None yet
Milestone
Milestone I
Development

No branches or pull requests
1 participant
@phwissmann