Bulk upload - technical details

[PaulKiddle]

Goal:

Allow a user to easilly upload and publish a collection of scans
and metadata related to the same publication.

Specifics:

User should be able to submit the following:

• Publication metadata:
  • Title, year, authors, journal-volume-page, abstract, url
• Publication PDF
• Multiple STL files
• CSV containing metadata related to STL files
  • filename
  • scientific name
  • alternate name
  • location
  • ID
  • URL to other media
  • Description
  • Geologic age
  • Ontogenetic age
  • Elements

Technical aspects:

• Metadata parsing
  • CSV file must contain a header row
  • filename column must contain filename of one of the stl files in the same form
  • required fields (which?) must be present
  • if any conditions are not met, form submission is blocked and an appropriate error displayed
  • csv may contain extra columns, which are ignored
• Data upload
  • Scan files should use chunked upload to avoid firewall issues
• Scan creation
• Association with publication
  • submitted by same form
• CTM generation
  • add a task to the task queue for each scan
• Thumbnail generation
  • This is currently done on the front-end, since it uses canvas/webGL
  • Use a python library to generate these?
  • Can it be done on backend w/ headless browser?
  • If not, can we automate it in the FE?

User interface:

Since a batch of scans are related to a publication,
it makes sense to add the batch upload form to the
existing publication form.

The file upload area has the following responsibilities:

• file select: allow user to select files, display selected files
• record list & errors: display the records found in the csv and associated files, and any errors
• upload progress: show how many files have been uploaded & overall progress
• ctm generation progress: show how many files have had their ctm generated (can be concurrent with upload)
• preview image generation progress: show how many files have had their preview image generated

Next steps

• Investigate possibility of preview/thumbnail generation on the server
  • Check if Anjali is ok with thumbnail mismatch
• UI mockup of upload form

[alycejenni]

pyrender/trimesh for server-generated thumbnails?

[PaulKiddle]

That's a good idea.
We should check Anjali doesn't mind the thumbnails looking a little different but TBH I can't see her having a problem - plus might be a good opportunity to change the model viewer component to something that's actually maintained.

[alycejenni]

I can probably make them look more like the current ones, I was just having fun with lighting 😄

[PaulKiddle]

Your ones do look a lot nicer - good to know it's flexible though