Replication between CouchDB nodes gets stuck

[in03]

Hey all.

I've been loving LiveSync. I use it every day for work for a steadily growing knowledge base. I've had it running on a Raspberry Pi 4 at home, but most of my heavy Obsidian usage is at work.

Unfortunately, the latency isn't great and my home upload speed (17mb/s) leaves something to be desired when uploading attachments.

I thought I'd give fly.io a crack. I figured fly.io will have more bandwidth and lower latency, so I can point LiveSync to that instead and have it replicate to my Pi as a backup.

Originally I followed the guide pinned at the top of discussions, but I ran into some weird authentication time-out issues – "You are not the server admin" – and I would keep having to log in. It also appeared to be inaccessible to LiveSync. Replication was very slow and eventually entered an error state.

Following that, I tried the colab notebook in the same discussion that uses Fly Builder to customise the ini file and that fixed most of the issues. Unfortunately, it still didn't fix my replication issues. I left it for a while in the hopes that it was pushing through some large attachments.

After a few hours, I checked the progress. Still stuck at 150MB...

My database is only 250MB, but I got a nasty surprise from fly.io telling me I've used 30GB! Only 11c, thankfully. Still, a lot more data than I was expecting.

I was hoping to keep this free while I trial it, but I had to throw in the minimum $25 prepaid credit to cover my 11c fee.

Any thoughts on my replication issues? I'm guessing the high bandwidth usage is caused by the replication repeatedly attempting to push through some problem files? Would that be logged somewhere? Apologies! I'm not well clued-in with CouchDB or fly.io just yet.

Thanks again @vrtmrz for your awesome contribution to the Obsidian community!

[in03]

Alright, I figured it out.
As it so often is, it was a combination of hasty user-errors.
I'll leave this here for others if that's okay?

---

First up, I ran the "cleanup databases" function under the "Maintain Database" settings in Livesync Settings. I think I got lucky and cleaned out all the chunks that hadn't been replicated yet, so after cleaning, there was nothing else to replicate and it was marked as "complete".

As for my replication freezing up issue...

I missed a couple of emails from fly (this is no longer my instance name btw, so no need to censor):

Hello! Your “long-fog-1341” application hosted on Fly.io crashed because it ran out of memory. Specifically, the instance e286711be59086. Adding more RAM to your application might help!

[...]

Here’s the raw log message:

Out of memory: Killed process 268 (beam.smp) total-vm:1843060kB, anon-rss:160600kB, file-rss:0kB, shmem-rss:26464kB, UID:5984 pgtables:604kB oom_score_adj:0

[...]

So that explains the replication stopping!

As for my crazy high usage... I misread the usage. I had only used about 800MB over my 3GB limit.
Of course 30GB for only 11c should've caught my attention. I probably attempted replicating from scratch enough times to go over the 3GB limit.

And finally the solution I came to!
Don't use the fly deployment guide pinned at the top of the discussions... Maybe it can be updated?
Use the Google colab guide linked in the README. Specifically the one that configures swap to be enabled:
https://gist.github.com/vrtmrz/37c3efd7842e49947aaaa7f665e5020a

[vrtmrz]

I am really sorry for the delay. And I am very sad about your unfortunate accident.

I can understand the confusion too. Replication issues are difficult to isolate. Thank you for your insights.
If I have allowed to add some notes, Replication statistics could possibly be helpful for us; that is found in the following.

1. Replication statics: http://your_instance/_scheduler/docs
2. couch_replicator of node statics: http://your_instance/_node/_local/_stats

Don't use the fly deployment guide pinned at the top of the discussions... Maybe it can be updated?

This is exactly my fault. I will fix it immediately!

[in03]

Hey @vrtmrz Don't be sorry! You've done amazing work. I needed to read more carefully and think for myself better is all. I've been running replication continuously successfully now. In fact, I have some devices on my Pi instance at home and some devices on the fly instance and I can't see any noticeable difference in LiveSync speed across the replicated databases. It's amazing! I don't really need it to be like this. I just wanted to have a backup going in case the fly volume goes down and I lose data. I have been experiencing some weird, conflicted files though. Any thoughts? They seem to be plugin configuration conflicts?  Thank you for your hints about replication!