diff --git a/docs/source/prefect.rst b/docs/source/prefect.rst index 0ff63f46..6d09cc30 100644 --- a/docs/source/prefect.rst +++ b/docs/source/prefect.rst @@ -14,15 +14,6 @@ In HPC, you can use the following command to create a work pool. If the prefect server is running, but the workpool is not available, then you can create a workpool by going to the website where the prefect server is hosted. Go to `Work Pools` tab and create a workpool with the name `workpool`. This is the name of the workpool that is defined in [prefect.yaml > definitions > work_pools > name](https://github.com/niaid/image_portal_workflows/pull/353/files#diff-b49a6f022232810a70f1a0c2feffbbe84d018b2418a7996e52430c6063ada3a3R23) file. -Deploying workflows -------------------- - -Once the prefect server is running and workpool is created. You can login to respective *BigSky* instance (dev, qa, prod) and deploy the workflows. - -.. code-block:: - - prefect deploy --all - Continuous Deployment (dev to qa to prod) ----------------------------------------- @@ -41,3 +32,45 @@ Afterwards, we can deploy the aws infrastructure for `qa` as, SPACES_SOLUTION_ENV=qa spaces task -f hedwig.spaces-solution.yaml build-deploy This can then again be applied for the `qa` to `prod` changes. + +Managing Prefect Worker +======================= + +Deploying workflows +------------------- + +Once the prefect server is running and workpool is created. You can login to respective *BigSky* instance (dev, qa, prod) and deploy the workflows. + +Make sure the configurations are correct: + +1. Update prefect.yaml to change the user and/or directory names + + .. code-block:: + + # by default pull.directory setting is set for prod environment + directory: /gs1/home/hedwig_prod/image_portal_workflows + + # change it to dev or qa, based on your environment + +2. Check prefect config with view + + .. code-block:: + + prefect config view + + # Update config iff required + export PREFECT_API_KEY=xyz + export PREFECT_API_URL=abc.com + +3. Deploy flows with prefect deploy + + .. code-block:: + + prefect deploy + # Or deploy all based on prefect.yaml using the following setting + # However, this will also deploy pytest_runner workflow in other envs (where it's not needed) + # prefect deploy --all + +4. Run worker (properly via the helper_scripts/.service file) + + The service files should restarts the worker when killed. Normally, we would need to do this step diff --git a/docs/source/usage.rst b/docs/source/usage.rst index b9f19551..b7cc6981 100644 --- a/docs/source/usage.rst +++ b/docs/source/usage.rst @@ -8,7 +8,7 @@ Workflow servers are deployed in `development schema.json +Pipeline Response +----------------- + +All pipelines responses follow the same schema. Currently, all schema are documented in yaml files as such: + +.. literalinclude:: ../../api_schema/PipelineCallback.yaml + :language: yaml + :emphasize-lines: 1-5,13-21 + :linenos: + +You can explore more on the yaml files in the `Github `_ repo. + CLI/SDK Submission ------------------