Fix file upload by adding an npm package that uses rust and aws sdk for rust to upload.

[RileySeaburg]

Summary

Simplified file upload and added npm package to upload the files as a rust process.

Preview

Link to Preview

Solution

Updated Upload method to call a rust function

Simplified gulp script to no longer have separate process and upload functions.

Click here for original rust source code

Multiple Image Upload

• Original Image
• 400_400.png

:~/gsa/digitalgov.gov$ npx gulp upload
[17:12:21] Using gulpfile ~/gsa/digitalgov.gov/gulpfile.js
[17:12:21] Starting 'upload'...
[17:12:21] Starting 'fileTidy'...
Converting image content/uploads/_inbox/pexels-diego-marcel-927914237-25568547.jpg to PNG
Moving file from content/uploads/_inbox/pexels-diego-marcel-927914237-25568547.png to content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547.png 
Converting image content/uploads/_inbox/pexels-tim-yisa-1040794498-20452012.jpg to PNG
Moving file from content/uploads/_inbox/pexels-tim-yisa-1040794498-20452012.png to content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012.png       
Moving file from content/uploads/_inbox/test-uploadts.xlsx to content/uploads/_working-files/to-process/test-uploadts.xlsx
[17:12:22] Finished 'fileTidy' after 519 ms
[17:12:22] Starting 'cleanInbox'...
[17:12:22] Finished 'cleanInbox' after 13 ms
[17:12:22] Starting 'writeDataFile'...
file is written
file is written
[17:12:22] Finished 'writeDataFile' after 329 ms
[17:12:22] Starting 'uploadTask'...
Starting file upload process...
Found 2 valid files in inbox.
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547.png" to S3 key: pexels-diego-marcel-927914237-25568547.png
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-diego-marcel-927914237-25568547.png
Uploaded original file to S3: pexels-diego-marcel-927914237-25568547.png
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547_w400_400.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547_w400_400.png" to S3 key: pexels-diego-marcel-927914237-25568547_w400_400.png
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-diego-marcel-927914237-25568547_w400_400.png
Uploaded resized file to S3: pexels-diego-marcel-927914237-25568547_w400_400.png      
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547_w1200_1200.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547_w1200_1200.png" to S3 key: pexels-diego-marcel-927914237-25568547_w1200_1200.png
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-diego-marcel-927914237-25568547_w1200_1200.png
Uploaded resized file to S3: pexels-diego-marcel-927914237-25568547_w1200_1200.png    
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547_w200_200.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547_w200_200.png" to S3 key: pexels-diego-marcel-927914237-25568547_w200_200.png
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-diego-marcel-927914237-25568547_w200_200.png
Uploaded resized file to S3: pexels-diego-marcel-927914237-25568547_w200_200.png      
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547_w800_800.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547_w800_800.png" to S3 key: pexels-diego-marcel-927914237-25568547_w800_800.png
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-diego-marcel-927914237-25568547_w800_800.png
Uploaded resized file to S3: pexels-diego-marcel-927914237-25568547_w800_800.png      
Successfully processed and uploaded: "content/uploads/_working-images/to-process/pexels-diego-marcel-927914237-25568547.png"
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012.png" to S3 key: pexels-tim-yisa-1040794498-20452012.png
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-tim-yisa-1040794498-20452012.png
Uploaded original file to S3: pexels-tim-yisa-1040794498-20452012.png
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012_w400_400.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012_w400_400.png" to S3 key: pexels-tim-yisa-1040794498-20452012_w400_400.png   
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-tim-yisa-1040794498-20452012_w400_400.png
Uploaded resized file to S3: pexels-tim-yisa-1040794498-20452012_w400_400.png
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012_w1200_1200.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012_w1200_1200.png" to S3 key: pexels-tim-yisa-1040794498-20452012_w1200_1200.png
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-tim-yisa-1040794498-20452012_w1200_1200.png
Uploaded resized file to S3: pexels-tim-yisa-1040794498-20452012_w1200_1200.png       
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012_w200_200.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012_w200_200.png" to S3 key: pexels-tim-yisa-1040794498-20452012_w200_200.png   
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-tim-yisa-1040794498-20452012_w200_200.png
Uploaded resized file to S3: pexels-tim-yisa-1040794498-20452012_w200_200.png
Attempting to upload file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012_w800_800.png"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012_w800_800.png" to S3 key: pexels-tim-yisa-1040794498-20452012_w800_800.png   
Upload completed. File should be accessible at: https://s3.amazonaws.com/digitalgov/pexels-tim-yisa-1040794498-20452012_w800_800.png
Uploaded resized file to S3: pexels-tim-yisa-1040794498-20452012_w800_800.png
Successfully processed and uploaded: "content/uploads/_working-images/to-process/pexels-tim-yisa-1040794498-20452012.png"
Warning: 2 valid files remain in the inbox after processing.
Upload process completed successfully.
Result: Successfully processed and uploaded 2 out of 2 valid files.
[17:12:57] Finished 'uploadTask' after 35 s
[17:12:57] Finished 'upload' after 36 s
file is written

File Upload

Link to uploaded excel.file

riley@Virtual-Desktop-1:~/gsa/digitalgov.gov$ npx gulp upload
[17:18:58] Using gulpfile ~/gsa/digitalgov.gov/gulpfile.js
[17:18:58] Starting 'upload'...
[17:18:58] Starting 'fileTidy'...
[17:18:58] Finished 'fileTidy' after 2.03 ms
[17:18:58] Starting 'cleanInbox'...
[17:18:58] Finished 'cleanInbox' after 14 ms
[17:18:58] Starting 'writeDataFile'...
[17:18:58] Finished 'writeDataFile' after 21 ms
[17:18:58] Starting 'uploadTask'...
Starting file upload process...
Found 0 valid files in inbox "content/uploads/_working-images/to-process".
Found 1 valid files in inbox "content/uploads/_working-files/to-process/".
Uploading non-image file to S3: static/test-uploadts.xlsx
Attempting to upload file: "content/uploads/_working-files/to-process/test-uploadts.xlsx"
Loading AWS config...
Using credentials from .env file
Creating S3 client...
Uploading file: "content/uploads/_working-files/to-process/test-uploadts.xlsx" to S3 key: static/test-uploadts.xlsx
Upload completed. File should be accessible at: [https://s3.amazonaws.com/digitalgov/static/test-uploadts.xlsx](https://s3.amazonaws.com/digitalgov/static/test-uploadts.xlsx)
Successfully processed and uploaded: "content/uploads/_working-files/to-process/test-uploadts.xlsx"
Upload process completed successfully.
Result: Successfully processed and uploaded 1 out of 1 valid files.
[17:18:58] Finished 'uploadTask' after 407 ms
[17:18:58] Finished 'upload' after 451 ms
file is written

How To Test

Note

Ensure you are using wsl or mac, or ubuntu lts.

1. Add as many files or images to the upload folder.
2. Run npx gulp upload
3. Click on the links generated by the cli, or the link in the yaml file.

Dependency updates

Dependency name	Previous version	New version
[@evolvingriley/file-uploa]	--	[0.1.3]
["@aws-sdk/client-s3"]	--	[ "^3.632.0"] \

[github-actions]

🔍 Preview in Federalist

[nick-mon1]

What I Tested

media types

type	result
png	pass
jpg	pass
pdf	pass
pptx	pass
doc	pass

• image variants are generated
• image variants display on aws as expected

upload types

case	result
single image	pass
multiple images	pass
single file	pass
multiple files	pass
multiple images and files	pass

Issue

Looks like the new upload script is appending the size twice for each image upload: test-upload-3-9-24-2024_w400_400.png. It should read: test-upload-3-9-24-2024_w400.png.

Suggestion

Can we remove sizes 200 & 400 and just use 800 & 1200. We can then merge #7908 changes to fit with these updates.

Idea

@RileySeaburg Should we consider updating the upload code to generate webp only going forward?

[nick-mon1]

Testing

I tested file and image uploading.

• image name is test-upload-3-9-24-2024_w400.png as expected
• creates 800,1200 variants and original only

Thanks @RileySeaburg.

[RileySeaburg]

Looks like the commits from CirceCI are not signed.

Since @hursey013 isn't on digital.gov anymore we can probably update the commit scripts to use one of ours.