Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FC-41030] batou_ext.file.SymlinkAndCleanup: add option "etag_suffix" #214

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Ma27
Copy link
Member

@Ma27 Ma27 commented Jan 22, 2025

Given the following deployment:

self += s3.Download(..., target=f"{version}.tar.gz")
archive = self._.target
self += SymlinkAndCleanup(target, pattern="*.tar.gz")

With this, the tarballs that are neither linked by last or current are removed. However, the .etag files from s3.Download are not.

Doing pattern="*.tar.gz*" is not an option here since this would remove ALL etag files. This means that the next deployment would download the tarball from S3 again since its etag file is missing.

This patch adds an attribute called etag_suffix that tries to delete all files matching the glob pattern pattern + etag_suffix unless pattern is equal to the file linked by current or last.

Given the following deployment:

    self += s3.Download(..., target=f"{version}.tar.gz")
    archive = self._.target
    self += SymlinkAndCleanup(target, pattern="*.tar.gz")

With this, the tarballs that are neither linked by `last` or `current`
are removed. However, the `.etag` files from `s3.Download` are not.

Doing `pattern="*.tar.gz*"` is not an option here since this would
remove ALL etag files. This means that the next deployment would
download the tarball from S3 again since its etag file is missing.

This patch adds an attribute called `etag_suffix` that tries to delete
all files matching the glob pattern `pattern + etag_suffix` unless
`pattern` is equal to the file linked by current or last.

FC-41030
@Ma27
Copy link
Member Author

Ma27 commented Jan 22, 2025

Fwiw I tested this with

from batou.component import Component
from batou.lib.download import Download
from batou.lib.file import File
from batou_ext.archive import SingleUntar
from batou_ext.file import SymlinkAndCleanup


class DownloadWithEtag(Component):
    def configure(self):
        download_data = [
            ("517b30657aa12089289aa0f963fc1a6d3626e2e4d15ef81711f554f74e896f52", "069ca2a21a1be741b359053c6d6c7a9e85837ea6"),
            ("308237fba2842f5cc1c225adf6e0a32194ca8a7696cb6dd9548f9232e1b9929a", "7af6329005deda244b90a54beb1f341159a67222"),
            ("32851337df95888fcda1d1a7e12c323e5b0c547b7897ab8a40c6b947af259f88", "21f2e291762a41f55edcbb2c0789d3b741f1bad5")
        ]
        index = 2

        checksum, rev = download_data[index]
        file = f"{rev}.tar.gz"

        self += Download(
            f"https://codeload.github.com/NixOS/nix/tar.gz/{rev}",
            target=file,
            checksum=f"sha256:{checksum}"
        )
        self += File(f"{file}.etag", content="etag dummy")
        self += SymlinkAndCleanup(
            self.map(file), pattern="*.tar.gz"
        )

Didn't bother to build my own S3 bucket just for this, so I decided to generate dummy etag files.
Multiple downloads were done by changing the index variable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants