Improve startup robustness and monitoring for connected services #185
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The application writing results will ensure that the containers are set up correctly.
Since we (currently) rely on the algorithms writing data to create the container, it may not have yet existing on startup, so we need to re-initialize the Tiled client for result retrieval.
To be discussed if that is sensible and should be implemented in other places
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new affix to the interface indicating the reachability of infrastructure including access to Tiled for input data, masks, and segmentation results, as well as Prefect server and Prefect worker. Additionally, infrastructure components not being available no longer hinders startup of the application, but instead issues a warning.
Reachability is queried once per minute. Since Prefect is also querying the state of its workers at an interval of a minute, there may be a delay in recognizing if the Prefect worker is up an running.
The state is shown with colored icons in the affix on hover and also summarized by color (red if anything is unreachable, gray if everything is reachable).
Adresses #180.