Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch from downloadjs to multi-download #174

Merged
merged 15 commits into from
Oct 28, 2024
Merged
1 change: 0 additions & 1 deletion .github/FUNDING.yml

This file was deleted.

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,8 @@ Blaze app can now be accessed at port `8080` :tada:
## Privacy and Analytics
- Blaze server does not track or record the files that are being shared both by WebSockets and WebTorrent.
- Any user related data like nickname, room names are always stored on device, and are only shared with the server when the user joins a room for file sharing.
- Blaze client uses Google Analytics to record the following:
- [Basic visit data](https://developers.google.com/analytics/devguides/collection/analyticsjs#what_data_does_the_google_analytics_tag_capture) as recorded by [Google Analytics](https://support.google.com/analytics/answer/6004245?ref_topic=2919631)
- Blaze client uses Google Analytics 4 to record the following:
- Part of [Basic visit data](https://support.google.com/analytics/answer/9234069?hl=en) - page views, scrolls and outbound clicks, rest are disabled.
- If Blaze PWA is installed on the device, and whether files are shared using share targets.

## Contributing
Expand Down
6 changes: 0 additions & 6 deletions client/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion client/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
"classnames": "^2.3.2",
"copy-to-clipboard": "^3.3.3",
"date-fns": "^2.29.3",
"downloadjs": "^1.4.7",
"nanoid": "^3.3.4",
"preact": "^10.3.2",
"preact-feather": "^4.2.1",
Expand Down
18 changes: 9 additions & 9 deletions client/src/routes/App/FileTransfer/FileTransfer.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import { h, createRef } from 'preact';
import download from 'downloadjs';
import { route } from 'preact-router';
import { PureComponent, forwardRef, memo } from 'preact/compat';
import { ArrowLeft, CheckCircle, Home, Plus, Image, Film, Box, Music, File, Zap, Share2, Send } from 'preact-feather';
Expand All @@ -17,6 +16,7 @@ import Visualizer from '../../../utils/visualizer';
import formatSize from '../../../utils/formatSize';
import pluralize from '../../../utils/pluralize';
import urls from '../../../utils/urls';
import { download, multiDownload } from '../../../utils/download';
import constants from '../../../../../common/constants';
import roomsDispatch from '../../../reducers/rooms';

Expand Down Expand Up @@ -292,17 +292,17 @@ class FileTransfer extends PureComponent {
isSelectorEnabled: false,
});
},
onDone: (file, meta) => {
if (file !== undefined) {
if (Array.isArray(file)) {
file.forEach(file => {
file.getBlob((err, blob) => download(blob, file.name));
});
onDone: (files) => {
if (files !== undefined) {
if (Array.isArray(files)) {
multiDownload(
// make regular File from webtorrent File https://github.com/webtorrent/webtorrent/blob/v1.9.7/lib/file.js#L13
files.map(file => new File([file.getBlob()], file.name, {type: file.type}))
Copy link
Author

@ddelange ddelange Aug 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, it looks like File.getBlob was removed in webtorrent v2.

this seems related to them switching to local storage in v2 as opposed to keeping complete torrents in memory in v1 ref webtorrent/webtorrent#86 (comment)

edit: they now have File.blob to load the whole torrent into memory and (more interestingly) File.streamURL (requires client.createServer to be ran beforehand).

Maybe blaze can even get rid of download.js altogether using the local storage functionality of webtorrent v2 🤔

they use FileSystemDirectoryHandle ref webtorrent/webtorrent#2208 this PR was superceded again

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blaze currently uses Webtorrent v1, and we'll need to upgrade to v2. From what I can recall since the last time I had checked, there were a bunch of API changes associated with v2 that we'd also have to make in Blaze which I wasn't fond of picking up then. Would you be interested in picking this up?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I'm also afraid it's a can of worms... the local storage feature might make it worthwhile though. I guess a 10GB file would now fill up 10GB of my swap right?

Copy link
Author

@ddelange ddelange Aug 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

judging by the File API docs, I think using streamURL you can initiate a download before the torrent has finished downloading, and so download.js could theoretically be called upon as soon as the torrent is initiated.

most importantly, that way the file doesn't need to go fully into memory in a Blob in order to send to downloads.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

file.getBlob and file.blob are functionally identical, it was simply renamed to match the W3C file API, rather than create our own structure, but getBlob also loaded the entire file into memory

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah just saw your cross comment, so file.streamURL is a viable option to get around the 'whole file in memory' pattern in the current downloading setup, with minimal overhead from client.createServer. that's great to hear and probably the way to move forward there!

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the local storage feature might make it worthwhile though

@ddelange isn't the local storage limited to like 10MBs? I wonder how it would manage files of much higher sizes in such a limited space

Copy link
Author

@ddelange ddelange Aug 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in v2 the torrents directly download to /tmp/webtorrent by default using file system access.

path can be set on a per-torrent basis by the receiver before the torrent is added (folder selection is google chrome only if I understood the comments correctly)

Copy link

@ThaUnknown ThaUnknown Aug 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in v2 the torrents directly download to /tmp/webtorrent by default using file system access.

for the sake of simplicity lets say sure, but if you want specifics, it actually writes to individual chunks via FSA's origin private file system, which functionally is no different from the likes of localStorage or IDB, except its a LOT faster, and allows for MUCH, MUCH, MUCH bigger file sizes, in extreme cases up to terabytes

on chrome that's a bit different, because IF YOU CHOOSE TO specify a rootDir, it can emulate the file structure in a specified folder as already mentioned, on top of saving chunks as cache to OPFS

that's the big difference for webtorrent v2, the storage and interfaces it provides:

  • createServer now provides insanely powerful, 0 overhead methods for rendering media like video, text, images using browser's own tools, instead of implementing custom tooling for it, which also allows other libraries to consume what webtorrent creates as its exposed via plain URL
  • storage isn't now memory, so we're no longer at the poor sub 2GB file limit, the most I could get webtorrent to store now is ~2TB after which chrome stopped enjoying handling so much data and started erroring

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for pointing to rootDir!

api.md points to storeOpts.rootDir but I think that should be opts.rootDir right?

based on the comments in api.md, I'm now not sure what's the difference between opts.path and opts.rootDir. should I ignore opts.path and only work with opts.rootDir?

);
}
else {
download(file, meta.name, meta.type);
download(files);
}
}
this.resetState();
},
ddelange marked this conversation as resolved.
Show resolved Hide resolved
});
Expand Down
6 changes: 2 additions & 4 deletions client/src/routes/App/layouts/AppLanding/AppLanding.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { h } from 'preact';
import { memo } from 'preact/compat';
import { ArrowDownCircle, Gift, Grid, PieChart, Settings } from 'preact-feather';
import { ArrowDownCircle, Gift, Grid, Settings } from 'preact-feather';
import { Link } from 'preact-router/match';
import { useContext } from 'preact/hooks';
import { PWAInstall } from '../../contexts/PWAInstall';
Expand Down Expand Up @@ -36,9 +36,7 @@ function AppLanding({ children, title, subtitle }) {
deferredPrompt.prompt();
const { outcome } = await deferredPrompt.userChoice;

window.ga('send', 'event', {
eventCategory: 'pwa-install',
eventAction: 'promo-shown',
window.gtag('event', 'pwa-install-prompt', {
outcome,
});
};
Expand Down
10 changes: 5 additions & 5 deletions client/src/template.html
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,13 @@
<meta name="msapplication-TileColor" content="#0D1322" />
<meta name="msapplication-TileImage" content="/assets/images/mstile-144x144.png" />

<script async src="https://www.googletagmanager.com/gtag/js?id=G-K0HSN5M77L"></script>
<script>
window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)};ga.l=+new Date;
ga('create', 'UA-82138003-3', 'auto');
ga('set', 'transport', 'beacon');
ga('send', 'pageview');
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-K0HSN5M77L');
</script>
<script async src='https://www.google-analytics.com/analytics.js'></script>


<% preact.headEnd %>
Expand Down
41 changes: 41 additions & 0 deletions client/src/utils/download.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// Adapted from https://github.com/sindresorhus/multi-download/blob/v4.0.0/index.js
// to take File as input https://developer.mozilla.org/en-US/docs/Web/API/File

/**
* Creates a promise that resolves after the specified number of milliseconds
*/
const delay = milliseconds => new Promise(resolve => {
setTimeout(resolve, milliseconds);
});

/**
* Downloads a single file
* @param file - An instance of the File type representing the file to download
*/
const download = async (file) => {
const a = document.createElement('a');
const url = URL.createObjectURL(file);
a.download = file.name;
a.href = url;
a.style.display = 'none';
document.body.append(a);
a.click();
await delay(100); // for Chrome
a.remove();
URL.revokeObjectURL(url);
};

/**
* Initiates multiple file downloads with a constant delay between each one
* @param files - An array of instances of the File type representing the files to download
*/
const multiDownload = async (files) => {
if (!files) {
throw new Error('`files` required');
};

for (const file of files) {
await delay(1000);
download(file);
blenderskool marked this conversation as resolved.
Show resolved Hide resolved
}
};
4 changes: 2 additions & 2 deletions client/src/utils/fileShare.js
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ class FileShare {
this.socket.listen(constants.FILE_INIT, (data) => {
if (data.end) {
if (fileParts.length) {
onDone(new Blob(fileParts), metaData.meta[0]);
onDone(new File(fileParts, metaData.meta[0].name, {type: metaData.meta[0].type}));
fileParts = [];
size = 0;
statProg = 0.25;
Expand Down Expand Up @@ -227,4 +227,4 @@ class FileShare {

}

export default FileShare;
export default FileShare;
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ services:
context: .
dockerfile: ./server/Dockerfile
environment:
TRUST_PROXY: true
TRUST_PROXY: 1
ports:
- 3030:3030

Expand Down