Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow worker to write files to a bucket in prod, local filesystem in dev #25

Open
phargogh opened this issue Jun 22, 2022 · 2 comments
Open
Assignees
Labels
backend Mainly a backend issue / task
Milestone

Comments

@phargogh
Copy link
Member

We should be able to have a consistent interface for where files are stored whether we're in a development environment or in production.

Possibility for helping to separate dev/prod configuration: https://pypi.org/project/python-decouple

@phargogh phargogh added the backend Mainly a backend issue / task label Jun 22, 2022
@phargogh phargogh self-assigned this Jul 18, 2022
@phargogh
Copy link
Member Author

phargogh commented Jul 18, 2022

For the prod part, it appears that we can access private GCS resources from within a container on GCS with no extra work! I'm relieved. To reproduce on our VM:

$ docker run --rm -ti urban-online-workflow_worker /bin/bash
# gdalinfo /vsigs/natcap-urban-online-datasets/nlud.tif
Driver: GTiff/GeoTIFF
Files: /vsigs/natcap-urban-online-datasets/nlud.tif
       /vsigs/natcap-urban-online-datasets/nlud.tif.ovr
Size is 153811, 96525
Coordinate System is:
PROJCRS["Albers_Conical_Equal_Area",
...

writing to the bucket, however, appears to be a different task entirely and right now I'm struggling to give the worker VM's service account the permissions needed to be able to write to the bucket.

  • Use cloud VM's credentials to push to a bucket.

@phargogh phargogh changed the title Abstract the storage backend away to a config file Allow worker to write files to a bucket in prod, local filesystem in dev Jul 29, 2022
@dcdenu4 dcdenu4 assigned dcdenu4 and unassigned phargogh Jun 6, 2023
@davemfish davemfish added this to the release milestone May 30, 2024
@dcdenu4
Copy link
Member

dcdenu4 commented Sep 20, 2024

@davemfish, let's revisit what we'd like to accomplish with this and more generally with a dev vs production setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Mainly a backend issue / task
Projects
Status: No status
Development

No branches or pull requests

3 participants