What about Tiddler compression as a feature like encryption? (LZMA.js) #7920
Replies: 4 comments 4 replies
-
|
I think you should make it clearer, what you try to achieve. It's true we can reduce size, but we also have to have a look about the "backside of the coin". --> Is it worth it? - What is the "cost"? - Where should compression be applied? -- Is the file human readable without a browser? -- and so on. For example every browser understands compressed content, if it comes from the server.
Click to see the screenshot
As seen in the screenshot. GitHub, which hosts tiddlywiki.com, does compress the single file wiki serverside, which makes sense. It's reduction of more than 60%. That's exactly the value you found out. So the server did compress the file and the browser did uncompress it in a completely transparent way. IMO that makes sense and does not need any new code. Let's say we want to save some space on a mobile device. On my phone I do have
So in relation all my "valuable documents and wikis" 560MB are about 5% of the size of my images 11GB alone. Most of the images are not worth keeping. So the question is: Does it make sense, to compress a 7MB single-page-wiki to save disk space? My answer is no. There is nothing to win. Removing all my images that are "duplicates" would probably make more sense. On my main PC, which holds most of my production wikis, they need about: 100MB text in different setups.
Downloads folder
Would it make sense to remove them. -> May be TLDR; If harddisk space is a real concern, every modern OS has the possibility to compress document folders in a transparent way on the fly. just my thoughts |
Beta Was this translation helpful? Give feedback.
-
|
My Apologies I thought I was pretty clear. But i see the confusion. LZMA.js is a javascript implementation of LZMA ENCODIMG/compression algorythm(s), the idea is it being used as a Function similiar to tiddlywiki's "encrypt" wiki function where on load lzma.js is uses to decomress tiddlers, on save it compresses. LZMA.js would do this internally in the tiddlywiki file itself compressing only user tiddlers, not system or otherwise excluded things like itself or $:/tags/RawMarkup (etc). It would be uselful for someone like myself who uses tools like dropbox & drop sync. if i want uploads exceeding 10Mbs or space exceeding 2GB I'D have to pay mothly. AlsovServerside hosting requires a server. If i want to use tiddloid i cannot have compression, as i wont have anything but tiddlywiki and its saver, i have a custom cms.html TW5 file that alone is 60mb. I understand that dropping 50% off of 8Mb is not exactly going to save the world here, but i can say that as a concept atleast i'd use it. When you think about it uglyfi TW5 is seeking a similar task to minimize the file even further. I just thought it might help control large wiki's is all, or unique space constraints. |
Beta Was this translation helpful? Give feedback.
-
I would ask. Why is your wiki so big? If you included images or PDFs, compression will not do anything there, since this content is already compressed and compressing it again will not be the real solution. |
Beta Was this translation helpful? Give feedback.
-
|
it deploys Wiki's with other plugins. as a result, it stores virtually every plugin/theme/palette I would ever use to work with my WebDAV farm |
Beta Was this translation helpful? Give feedback.
-
|
Hi @shadowforce62 |
Beta Was this translation helpful? Give feedback.
-
|
ftr |
Beta Was this translation helpful? Give feedback.
-
|
.+ https://enkimute.github.io/shorty.js/ https://github.com/enkimute/shorty.js / https://github.com/kungfooman/shorty.js
fwiw reminded me of watching changes to tw metadata / internal state tiddlers in git commits idk if those or any other tiddlers might be considered a suitable candidate for this sort of compression method at least afaik they are not required / desired by default to be user facing / editable and seem *(to me) to create alot of noise between saved versions |
Beta Was this translation helpful? Give feedback.
-
|
I found this discussion in the context of making my charting plugins smaller (or any plugin incorporating a JS library). My plugins all incorporate a variety of JS libraries that are many 10's or 100's of KBs of minimised code. Compressing it would be even better. I was going to suggest adding inflate-only support to the |
Beta Was this translation helpful? Give feedback.
-
This was originally posted on TalkTiddlyWiki, also adding it here.It appears this idea hasn't been extensively explored before. The closest resemblance I've encountered are UglyfiTW5 or JSZip. However, the concept I'm contemplating leans towards a more literal approach. Picture this:
Consider TiddlyWiki: it compresses Tiddler Data and fields but not the core itself. Similar to how sjcl.js handles encryption, the proposition involves decrypting and/or decompressing TiddlyWiki Tiddlers (excluding System/Excluded ones) upon loading. In my trials, LZMA typically reduces space usage by approximately 60%. Imagine combining this with something like UglyfiTW5 for a wiki the size of the main page.
Using some rough calculations based on the main page of TiddlyWiki, the numbers came out as follows:
This estimation involved determining the empty size of TiddlyWiki, exporting non-system/shadow/plugin Tiddlers (user-only) as CSV, using the online LZMA.js demo to compress only text fields (excluding titles or other field values), and then reintegrating all 'Uncompressed' columns back into TiddlyWiki. The resulting compression from the Compressed Text Fields led to a reduction from 7.30MB to 3.66MB (about 50% smaller), excluding the 'Lossy' compression of UglyfiTW5.
Notably, LZMA being Lossless ensures retrieval of anything stored upon loading. This concept bears potential usefulness for various reasons. According to LZMA.js, Text Field Size was reduced by approximately 67% in 11.5 seconds. Moreover, LZMA.js claims compatibility as far back as IE6, hinting at likely compatibility across all browsers/mobile & HTA Hacks/APPS.
Additionally, LZMA.js allows users to choose their preferred compression level. My tests showed it could compress around 2.2Mb of data in roughly 3.6 seconds and decompress the same data in about 0.5 seconds using LZMA Level 1 with about 59% Compression. Running tests at Level 9 showed double decompression/compression times for approximately 8% more efficient storage. While 8% might seem modest, it's impressive given TW5's existing footprint.
Overall, the concept, when combined with uglify, appears to reduce the TW5 Home Page by approximately 50% without sacrificing functionality or data. At the very least, a plugin akin to the single Tiddler encryption plugin could be developed to compress individual Tiddlers.
My grasp of JavaScript is limited, and when it intertwines with TiddlyWiki, it becomes even more bewildering for me. An attempt I made in a sandbox to incorporate LZMA, failed unfortunately and my attempt to jerry-rig it was unsuccessful due to my lack of expertise in this area.
Thoughts?
TL:DR,
Links:
Demo Used to Test Text-Field Compression
A Demo of LZ-string vs LZMA.js
OP: https://talk.tiddlywiki.org/t/tiddler-compression-using-lz-lzma/8837Beta Was this translation helpful? Give feedback.
All reactions