Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new startup process that scans for broken imports and retries them. #227

Open
Foxcapades opened this issue Feb 7, 2024 · 0 comments
Labels
internals Changes to VDI internals that are not visible to any external system.

Comments

@Foxcapades
Copy link
Member

Foxcapades commented Feb 7, 2024

There should be a new process that fires before the trigger handlers are started that scans postgres for datasets that have an import status of queued or in-progress.

For datasets that are in the queued status:

Check if an import ready file exists in MinIO.

If the import ready file does exist, fire an import event

If the import ready file does not exist:

  • mark the dataset as failed
  • write an import message indicating that the upload was lost due to unexpected service shutdown

For datasets that are in the in-progress status:

  • set them back to queued and fire an import event.
@Foxcapades Foxcapades added the internals Changes to VDI internals that are not visible to any external system. label Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
internals Changes to VDI internals that are not visible to any external system.
Projects
None yet
Development

No branches or pull requests

1 participant