Allow uploading local files as inputs

[akx]

It would be handy to allow uploading local files as inputs.

This will require some deduplication magic, i.e. we'll check if there's already a Datum in the project with the same checksum as the file the user is about to upload, and skip uploading if one exists.

[ruksi]

Alternative is to start implementing vh data subcommand which allows easy uploading of files to data stores and returns the datum URL for usage.

Maybe also vh store subcommand section for data stores... maybe.

Of course, automatically checking for duplicates with vh exec run train --dataset=/path/to/file.zip would be better, but the above manual uploading might be a nicer low hanging fruit.

Came up during customer onboarding last week.

[neomatrix369]

Alternative is to start implementing vh data subcommand which allows easy uploading of files to data stores and returns the datum URL for usage.

Maybe also vh store subcommand section for data stores... maybe.

Of course, automatically checking for duplicates with vh exec run train --dataset=/path/to/file.zip would be better, but the above manual uploading might be a nicer low hanging fruit.

Came up during customer onboarding last week.

I raised the same on the slack channel discussions.

Any chance of this option being available soon? Or do I still have to use the UI to upload my files.

I could make it part of my git - but then my git repo will be so big. Its sort of an antipattern.

[ruksi]

We haven't fleshed this out yet as there are other talks about improving data management big time later this year. In the meantime, we'll work on this if we have spare cycles, but, to clarify, nothing has been started yet.

[neomatrix369]

@ruksi is this something we could do with the help of the API and the client tool available? If you drop hints I could pick this up and make it work, as I find this useful and something I'd like to use myself when running experiments.