Use DANE version with no-tar option (WIP)

[Veldhoen]

Instead of tarring the worker's output before uploading to S3, files are added separately with prefixes to maintain the directory structure. This mostly relies on a proposed update in DANE core; https://github.com/CLARIAH/DANE/tree/23-update-s3_util-to-allow-for-untarred-uploading

• Make sure to have a .env file with AWS credentials to be able to write to the bucket you specify in config
• Run either locally:
  • install the environment with poetry install
  • Copy ./config/config.yl to the root of the repo (.) and adjust the necessary settings: BASE_MOUNT and PATHS , S3_endpoint_URL, TRANSFER_ON_COMPLETION
  • poetry run python worker.py --run-test-file
• Or run containerized:
  • Build the image docker build -t dane-visual-feature-extraction-worker .
  • Adjust the config in ./config/config.yl (set S3_endpoint_URL and TRANSFER_ON_COMPLETION)
  • Run docker-compose up
• Run with TAR_OUTPUT: false and TAR_OUTPUT: true and confirm that the output is transferred in the appropriate format

Bonus points: check out the DANE PR too (https://github.com/CLARIAH/DANE/tree/23-update-s3_util-to-allow-for-untarred-uploading)