Define our data storage and bucket schema

[phargogh]

We'll need the ability to store a variety of files during execution. Define what these filenames should be or what pattern they should take.

Related, which buckets should these files use? Where should the core datasets (e.g. NLUD) be stored?

[phargogh]

Here's what we're thinking

So, <backend> refers to some directory-centric storage mechanism, such as the local filesystem of GCS.

<backend>/
        {session_id}/
                scenarios/
                        {scenario_id}_{wallpaper or fill}_{name:sanitized}.tif  # with internal overviews
                        LATER: {scenario_id}_{thumbail}.png
                model_outputs/
                        {job_id}_{model_name}/
                                workspace/
                                        <logfile>
                                        <model specific outputs>
                                <actual output file with overviews, derived from model outputs>

NOTES:
input data will be kept in a private bucket (see: principle of least privilege)


TODO: do we save the whole stack of outputs or just a few of them?
TODO: push docker logs to GCP's logs when running on prod

[phargogh]

@dcdenu4 I just realized that scenarios are not attached to sessions, so I propose updating the file structure to be this:

<backend>/
        scenarios/
                {scenario_id}/
                        {scenario_id}_{wallpaper or fill}_{name:sanitized}.tif  # with internal overviews
                        LATER: {scenario_id}_{thumbail}.png
        model_outputs/
                {session_id}/
                        {job_id}_{model_name}/
                                workspace/
                                        <logfile>
                                        <model specific outputs>
                                <actual output file with overviews, derived from model outputs>

NOTES:
input data will be kept in a private bucket (see: principle of least privilege)


TODO: do we save the whole stack of outputs or just a few of them?
TODO: push docker logs to GCP's logs when running on prod

Does that sound OK to you?

[davemfish]

I'm not sure if there's any more work to be done here. If there is, it can be probably be done alongside #25 .