Dataset download zip email #129

klpoland · 2025-06-19T17:06:34Z

SFDS-186

Added celery worker and mailhog to docker compose
refactored celery configs
Use smtp email backend on local
Added get_dataset_files and download_file helper functions to avoid duplicate code, easier maintenance
Added celery task for zipping and emailing files form dataset as well as some test tasks
Added tests for celery, file download, command to run celery tests with easy to read logs
Added download button, JS to run DatasetDownloadView
make test, make test-integration, and make pre-commit pass locally

gateway/sds_gateway/api_methods/helpers/download_file.py

gateway/sds_gateway/api_methods/tasks.py

gateway/sds_gateway/users/views.py

gateway/sds_gateway/api_methods/management/commands/test_celery.py

lucaspar · 2025-06-19T18:02:35Z

gateway/sds_gateway/api_methods/tasks.py

+            to=[user_email],
+        )
+
+        email.attach(zip_filename, zip_buffer.getvalue(), "application/zip")


let's provide a temporary download link instead, this can be GBs of data.

Yeah, that's what I do on REDI-NET for several things. So maybe the process is actually save the zip file on the server so they can download it? what does temporary mean? As in, the link only exists for a period of time and disappears when they use it? should the zip file be cleaned out once it's downloaded or remain on the server?

the link only exists for a period of time and disappears when they use it?

we can allow multiple uses (e.g. download failure), but let's make it expire in e.g. 24h and clean up the zip.

there's also a case to be made about not recreating the zip if it already exists for a given dataset, but this could be a future PR. For example, multiple "download" clicks while the link is still valid.

gateway/config/settings/base.py

lucaspar · 2025-06-19T18:10:04Z

gateway/sds_gateway/templates/users/dataset_list.html

+            document.addEventListener('DOMContentLoaded', function() {
+                console.log('DOM loaded, looking for download buttons...');
+
+                // Handle download button clicks
+                document.querySelectorAll('.download-dataset-btn').forEach(function(button) {
+                    console.log('Found download button:', button);
+                    button.addEventListener('click', function() {
+                        const datasetUuid = this.getAttribute('data-dataset-uuid');
+                        const datasetName = this.getAttribute('data-dataset-name');
+
+                        console.log('Download button clicked for dataset:', datasetUuid, datasetName);
+
+                        // Disable button and show loading state
+                        const originalText = this.innerHTML;
+                        this.innerHTML = '<i class="bi bi-hourglass-split"></i> Processing...';
+                        this.disabled = true;
+
+                        // Make API request
+                        console.log('Making request to:', `/users/dataset-download/${datasetUuid}/`);
+                        fetch(`/users/dataset-download/${datasetUuid}/`, {
+                                method: 'POST',
+                                headers: {
+                                    'X-CSRFToken': document.querySelector('[name=csrfmiddlewaretoken]').value,
+                                    'Content-Type': 'application/json',
+                                },
+                            })
+                            .then(response => {
+                                console.log('Response status:', response.status);
+                                if (response.ok) {
+                                    return response.json().then(data => {
+                                        console.log('Success response:', data);
+                                        // Show success message
+                                        showAlert('success', `Download request sent for "${datasetName}". You will receive an email with the files shortly.`);
+                                        this.innerHTML = '<i class="bi bi-check-circle"></i> Sent!';
+                                        this.classList.remove('btn-success');
+                                        this.classList.add('btn-secondary');
+                                    });
+                                } else {
+                                    return response.json().then(data => {
+                                        console.log('Error response:', data);
+                                        // Show error message
+                                        const errorMessage = data.detail || data.message || 'Failed to send download request';
+                                        showAlert('danger', `Error: ${errorMessage}`);
+                                        this.innerHTML = originalText;
+                                        this.disabled = false;
+                                    });
+                                }
+                            })
+                            .catch(error => {
+                                console.error('Network error:', error);
+                                showAlert('danger', 'Network error. Please try again.');
+                                this.innerHTML = originalText;
+                                this.disabled = false;
+                            });
+                    });
+                });
+
+                // Function to show alerts
+                function showAlert(type, message) {
+                    const alertDiv = document.createElement('div');
+                    alertDiv.className = `alert alert-${type} alert-dismissible fade show`;
+                    alertDiv.innerHTML = `
+                    ${message}
+                    <button type="button" class="btn-close" data-bs-dismiss="alert" aria-label="Close"></button>
+                `;
+
+                    // Insert at the top of the content area
+                    const contentArea = document.querySelector('.main-content-area');
+                    contentArea.insertBefore(alertDiv, contentArea.firstChild);
+
+                    // Auto-dismiss after 5 seconds
+                    setTimeout(() => {
+                        if (alertDiv.parentNode) {
+                            alertDiv.remove();
+                        }
+                    }, 5000);
+                }
+            });
+        </script>


remove most (all?) console.log

I think there's a usability improvement here, since downloads are async.

We should probably communicate these things to the user before starting the task, not verbatim:

Expect an email with the download link sent to <email-address>;

Ask for confirmation to proceed with download.

klpoland · 2025-06-20T12:10:03Z

Added an extra confirmation screen
Send user to a download page link (check login and ownership of dataset to make sure correct person is downloading)
Added celery flower to compose
Added temporary zip file model to track expiry

I'm doing soft deletion of the DB object but delete the actual files, this is a management command, maybe run this at some regular interval, probably not super often. I'm open to suggestions on how to handle the zip files when they expire.

…o re-direct

lucaspar · 2025-06-20T16:06:39Z

maybe run this at some regular interval, probably not super often. I'm open to suggestions on how to handle the zip files when they expire.

maybe running a celery's periodic task once or twice a day?

klpoland added 4 commits June 19, 2025 08:36

dataset download initial pass

f78aa7f

set up celery, mailhog, tests

e97aa0f

troubleshooting, bugfixing, tests

1befc29

troubleshooting, test refactoring

ff98438

klpoland requested a review from lucaspar June 19, 2025 17:06

klpoland self-assigned this Jun 19, 2025

klpoland added feature New feature or request gateway Gateway component styling Special focus on styling of front-end components javascript Pull requests that update javascript code labels Jun 19, 2025

klpoland added 2 commits June 19, 2025 14:06

edited email config with prod settings

744b887

make example file more general

79e50ee

lucaspar requested changes Jun 19, 2025

View reviewed changes

klpoland added 2 commits June 19, 2025 17:58

refactoring code to send email with link to download instead of raw zip

8f47219

inspection testing, linting

c3c5979

klpoland added 5 commits June 20, 2025 08:34

change temp zip file duration

d0e15f3

better assertions for testing zip file clean up

94debe6

simplified view and download template with processing message and aut…

6a61a31

…o re-direct

migration

637eb6b

skip verify, linter breaks template tags

fca98cd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Dataset download zip email #129

Dataset download zip email #129

Uh oh!

klpoland commented Jun 19, 2025 •

edited by lucaspar

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lucaspar Jun 19, 2025

Uh oh!

klpoland Jun 19, 2025

Uh oh!

lucaspar Jun 19, 2025

Uh oh!

lucaspar Jun 19, 2025

Uh oh!

klpoland Jun 20, 2025

Uh oh!

Uh oh!

lucaspar Jun 19, 2025

Uh oh!

lucaspar Jun 19, 2025

Uh oh!

klpoland Jun 20, 2025

Uh oh!

klpoland commented Jun 20, 2025 •

edited

Loading

Uh oh!

lucaspar commented Jun 20, 2025 •

edited

Loading

Uh oh!

Uh oh!

Dataset download zip email #129

Are you sure you want to change the base?

Dataset download zip email #129

Uh oh!

Conversation

klpoland commented Jun 19, 2025 • edited by lucaspar Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

klpoland commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lucaspar commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

klpoland commented Jun 19, 2025 •

edited by lucaspar

Loading

klpoland commented Jun 20, 2025 •

edited

Loading

lucaspar commented Jun 20, 2025 •

edited

Loading