Switch from Base64 to Z85

[edde746]

Less common but neither is human readable, the following benchmarks are of the current implementation in comparison to the one in this pull request:

Size Comparison

Input Size	Base64 Size	Z85 Size	Base64 Overhead	Z85 Overhead
100 B	136 B	125 B	36.00%	25.00%
1.00 KB	1.34 KB	1.25 KB	33.59%	25.00%
256.00 KB	341.34 KB	320.00 KB	33.33%	25.00%
1.00 MB	1.33 MB	1.25 MB	33.33%	25.00%

Encoding Performance

Input Size	Base64	Z85	Ratio (Z85/Base64)
100 B	0.002ms	0.001ms	0.79x
1.00 KB	0.019ms	0.016ms	0.82x
256.00 KB	7.121ms	6.222ms	0.87x
1.00 MB	86.522ms	52.788ms	0.61x

Decoding Performance

Input Size	Base64	Z85	Ratio (Z85/Base64)
100 B	0.006ms	0.002ms	0.37x
1.00 KB	0.028ms	0.019ms	0.67x
256.00 KB	10.564ms	7.609ms	0.72x
1.00 MB	62.924ms	30.164ms	0.48x

[ZYinMD]

Hi, sorry for appearing out of nowhere. I happen to have subscribed to all PRs of this repo because I'm personally interested in this repo, so I saw this PR. I also happen to have researched arrayBuffer <-> Base64 conversion before, so I have one thing to comment.

So back in 2023, because of a project I was doing, I benchmarked all browser implementations of arrayBuffer <-> Base64, and found that base64→arrayBuffer is fastest when done in js, but arrayBuffer→base64 is 10x faster when done with the FileReader API. Here's the code, I've been using it in production since 2023:

/** convert arrayBuffer to base64 using the fileReader API, 
 * much more performant than btoa or js implementation
 * @param {ArrayBuffer} buffer
 * @returns {Promise<string>}
 */
export async function arrayBufferToBase64(buffer) {
  const file = new File([buffer], "temp.bin", {
    type: "application/octet-stream",
  });
  const fileReader = new FileReader();
  const dataUrl = await new Promise((resolve) => {
    fileReader.onload = () => resolve(fileReader.result);
    fileReader.readAsDataURL(file);
  }); // something like `data:application/octet-stream;base64,AQpy3Es9Ja0Abb=`
  return dataUrl.split("base64,")[1];
}

Would someone like to benchmark it?

(one thing to note: my benchmark was done with larger data (10MB-100MB) because that's what my project involved)

[edde746]

@ZYinMD FileReader is not available in Node, it'd also require the function to be async so it'd be entirely unsuitable in this case.

Here is a benchmark with the more "native" (but deprecated) atob/btoa:

Encoding

Input Size	Base64	Z85	btoa
100 B	0.002ms	0.001ms	0.004ms
1000 B	0.018ms	0.007ms	0.022ms
9.77 KB	0.139ms	0.074ms	0.201ms
97.66 KB	1.502ms	1.193ms	2.022ms

Decoding

Input Size	Base64	Z85	atob
100 B	0.006ms	0.003ms	0.003ms
1000 B	0.030ms	0.013ms	0.010ms
9.77 KB	0.308ms	0.116ms	0.077ms
97.66 KB	3.226ms	1.028ms	0.769ms

[ZYinMD]

I also tested btoa and bota back then, and despised them - btoa would throw error on input larger than ~1MB, and atob is 10x slower than js implementation. Not sure why they did ok in your benchmark, maybe because the data wasn't large, but I remember they sucked at 10MB-100MB input.

Anyway, I thought on the server side some c++ library could be imported as dependency, and on the browser side async shouldn't be a problem. But what do I know?