Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete a temporary file, when a distributor finishes downloading it, but it ends up having an unexpected size #26

Open
daniil-berg opened this issue Sep 13, 2024 · 0 comments
Assignees
Labels
error handling More helpful and transparent error handling/logging question Further information is requested

Comments

@daniil-berg
Copy link
Contributor

Currently we throw an UnexpectedFileSize error, if the number of bytes loaded does not exactly match the size that the distributor was "promised", when its /api/distributor/copy/... endpoint was called. The file is then not considered "controlled" by that node, but it remains present on disk at its temporary path.

There is no cleanup task like there is with the storage file controller and since the file is not considered controlled by the node, the periodic DistributionController.free_up_dist_nodes task does not clean it up either. I have not tested this, but I think in such a case the temporary file would remain there indefinitely. In the worst case, the disk could end up littered with "data corpses".

@daniil-berg daniil-berg self-assigned this Sep 13, 2024
@daniil-berg daniil-berg added question Further information is requested error handling More helpful and transparent error handling/logging labels Sep 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
error handling More helpful and transparent error handling/logging question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant