Does the local (and remote) db contain the entire vault?

[CarstenNeubert]

Hello! I'm trying to understand the plugin a little bit better. One thing that isn't entirely clear to me yet: does the local db really contain the entire vault (+ all stored revisions for conflict resolution) or just the local changes that haven't yet been replicated to the remote couchdb?

It's mostly text files, but together will all the revisions (up to 100x per doc?) a full copy of the vault might still get pretty large over time.

[vrtmrz]

Thank you for asking me! The short answer is yes. However, there are many tricks.

• Every document is split into chunks and metadata, and on editing, only the metadata and the difference are transferred.
  This realises Live on MB order text files. You can see the structure by Dump information of this doc in the command palette.
• Revisions are stored in each client only if it needs.
  Changes in other vaults are not transferred and stored if we do not have to merge them.
• Chunks are shared globally inside the vault if the content is the same.

As you mentioned, one day it will be full of chunks that are no longer referenced. However, it will not be depending on the total sizes of the vault times 100. Our progress is going to consume database storage on a daily basis.
I hope this explanation has been helpful; please let me know if you have any further questions.

P.S. Half of them are written in the site.

[CarstenNeubert]

Great! Thank you for the explanation and the link. Does the plugin output the current size of the databases (local and remote) somewhere, so I can keep an eye on it?

Btw, unrelated to this question, but have you looked at Oracle Cloud for free hosting? I saw your fly.io post. Their free tier is extremely generous. Up to 4 machines with (in total) 4 virtual cpus, 24gb ram, 200gb ssd, 10TB egress. Signup without credit card for the free tier but idle machines may be turned off after a week unless you upgrade to (still free) pay-as-you-go.

[hilsonp]

Does that mean that as soon as I add a character to a line, a new chunk is created and the old one remain most probably unreferenced, most probably forever (unless I take a manual action) ?

I guess, I will go disable livesync 😨 and make it sync on open/save.

In any case, wouldn't it be a good idea to purge automatically unused trunk after a defined time ?

[vrtmrz]

@CarstenNeubert

Does the plugin output the current size of the databases (local and remote) somewhere, so I can keep an eye on it?

Unfortunately, we are not able to know the local database size (the total indexedDB usage of Obsidian could be known in devTools).
The remote size can be known in Fauxton(The administrating tool of CouchDB) or API. However, not everything will be transferred while Read chunks online has been enabled.

Oracle Cloud for free hosting

I think it is the best free hosting. I use OCI and have finally registered my card, and have never paid for it yet.
I would recommend it most to anyone who can set up a server with ssh.

[vrtmrz]

@hilsonp

Does that mean that as soon as I add a character to a line, a new chunk is created and the old one remain most probably unreferenced, most probably forever (unless I take a manual action) ?

In case you have enabled LiveSync, you are right. To prevent it, we can enable Batch database update.
If it has been enabled, changes will not be applied to the database immediately.

In any case, wouldn't it be a good idea to purge automatically unused trunk after a defined time ?

Choosing unreferenced chunks was more difficult than one might think. Due to sharing the chunks, we cannot make sure the chunk is not referenced. (even if the current device and the remote database are not using the chunk, another device possibly references it).
The easiest way to remove old revisions and unreferenced chunks in the local database is to enable Read chunks online and run Fetch again. Doing this will transfer all metadata again. Following that, only the currently referenced chunks will be transferred.
For remote databases, Rebuild everything is eventually an effective way.

I have written so far, but I may have had a flash of inspiration. It is not a lightweight method but may work.
(Lock the remote and collect garbage and delete, and replicate the database to a temporary database only with active things inside the remote server and write it back. After that, unlock and fetch.)