An archiver service that allows uploading dataset and registering it with SciCat. It is built on Prefect.io to orchestrate the asynchronous jobs (called flows) to archive and retrieve datasets to/from the ETH LTS.
The full setup is containerized and requires a SciCat instance (or a mocked instance) to run correctly.
Refer to the Github pages for more details.
Build all the services:
docker compose --env-file .env --env-file .development.env buildStarting up all services for development:
docker compose --env-file .env --env-file .development.env up -dFor development and debugging, a local process can serve flows, for example by running python -m archiver.flows. However, some more configuration is required to fully integration with the other services; therefore a VS Code launch command Prefect Flows can be used in launch.json). This allows to debug flows locally and the registered flows have a prefix, DEV_.
The latest documentation can be found in the GitHub Pages
Github pages are built on mkdocs.
Build docker image first:
docker build . -f docs.Dockerfile -t ghcr.io/swissopenem/scopemarchiver-docs:latestStart development server
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs ghcr.io/swissopenem/scopemarchiver-docs:latestBuild documentation
docker run --rm -it -p 8000:8000 -v ${PWD}:/docs ghcr.io/swissopenem/scopemarchiver-docs:latest buildDeploy documentation to GitHub Pages
docker run --rm -it -v ~/.ssh:/root/.ssh -v ${PWD}:/docs ghcr.io/swissopenem/scopemarchiver-docs:latest gh-deploy