Submittiing a zip file to MSU

[hohonuuli]

Once a zip file of images has been preprocessed, we need a method to submit the zip file to MSU (e.g. SFTP) in an automated fashion.

Once MSU receives the zip file, it must be immediately unpacked at the expected location on the server.

Once unpacked the images must be publicly accessible

[hohonuuli]

Important, naming collisions are going to be an issue ... #139 (comment)

[hohonuuli]

The zip file will have content like:

/FNYYMM
|- FNYYMM_image1.png
|- FNYYMM_image2.png
|- FNYYMM_imageFoo.png
|- FNYYMM_blahblahblah.jpg
|- FNYYMM_namesDontMatterHere.jpg
|- FNYYMM_etc.png
`- somefile.csv

[hohonuuli]

I've added starter code for a service that FathomNet can upload image packages to be processed and managed. https://github.com/fathomnet/bifrost

[hohonuuli]

Updated naming for the data directory is FNYYMM(whatever I want)

[hohonuuli]

I've set up a staging area for zip file at https://fathomnet.org/static/staging/ which maps to atlas://FathomNet/web/staging for this email exchange:

David Moffitt:

Thanks for the server info. I was thinking we could poll the server daily unless it needs to be done more or less frequently. I could also set up automated email notifications for when a file is unpacked. How does that sound to you?

My responses:

I’ve posted an updated sequence diagram based on MSU polling for data. The new diagram is here. We can work with your proposal, although I think being able to push to your servers would be waaaay less work for us. Being able to a. push a scp a zip to your server and then b. run an ssh command to unpack it, would eliminate a ton of other support code we need for this polling model.

I’ve set up a staging area on FathomnNet for MUS to test the proposed polling workflow. See https://fathomnet.org/static/staging/ . I can put new zips of images there.

Let me know if you need anything else to start moving forward. For the moment you can send the emails to both [...] ). (Although that will change in the future)