Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: remove optionally using operating system in the manifest file name hash #552

Draft
wants to merge 1 commit into
base: mainline
Choose a base branch
from

Conversation

godobyte
Copy link
Contributor

@godobyte godobyte commented Jan 11, 2025

What was the problem/requirement? (What/Why)

attachment download and attachment upload commands support upload and download in a batch. They use the premise manifest file name contains the hash of the root path where the manifest was generated for to map manifest file to its corresponding root.

However, file_system_location_name is used in certain cases when passed in for some reason. This lead to the following problems at the time of downloading or uploading

  1. manifests and root paths approach has no visibility of file_system_location_name
  2. path mapping approach potentially has no visibility of file_system_location_name

What was the solution? (How)

By tracing the code, there is no indication file_system_location_name need to be required in hash. Manifest should be unique to its root. Additionally, there is no hash check on the file name, which guarantees this change does not introduce regression.

Input manifest is already persisted as part of job

attachments:
  manifests:
  - rootPath: /bealine-clients/deadline-cloud-worker-agent/scripts/submit_jobs/asset_example
    rootPathFormat: posix
    outputRelativeDirectories:
    - .
    inputManifestPath: farm-XXX/queue-XXX/Inputs/43fc2f90cec04ba5a2aa3706c50fb786/0bb7eb91fdf8780c4a7e6174de6dfc5e_input

Output manifest is discovered using formated s3 partition.

What is the impact of this change?

How was this change tested?

  • Have you run the unit tests? [Y]
  • Ran integ tests [Y]
    • 28 passed, 2 skipped, 5 warnings in 74.34s (0:01:14)
  • Have you made changes to the download or asset_sync modules? If so, then it is highly recommended
    that you ensure that the docker-based unit tests pass. [Y]
    • 32 passed, 12 skipped in 14.59s

Was this change documented?

  • Are relevant docstrings in the code base updated? [N/A]
  • Has the README.md been updated? If you modified CLI arguments, for instance. [N/A]

Does this PR introduce new dependencies?

  • This PR adds one or more new dependency Python packages. I acknowledge I have reviewed the considerations for adding dependencies in DEVELOPMENT.md.
  • This PR does not add any new dependencies.

Is this a breaking change?

This PR changes the way manifest file name is generated to be the hash of root path.

Does this change impact security?

No


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@godobyte godobyte requested a review from a team as a code owner January 11, 2025 00:56
Copy link

@jericht jericht marked this pull request as draft January 11, 2025 01:36
@jericht jericht added the job attachments For an issue with job attachments label Jan 13, 2025
@@ -224,16 +224,15 @@ def _gather_upload_metadata(
manifest: BaseAssetManifest,
source_root: Path,
manifest_name_suffix: str,
# TODO - remove file_system_location_name after ASSET_SYNC_JOB_USER_FEATURE completion
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a tracking ticket so we don't forget?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
job attachments For an issue with job attachments
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants