Add remote borg backup support

[timdiels]

This adds support for backing up directly to a remote borg repo. (I tried reaching out earlier)

Why?

I'd like to backup/restore via the UI, resist ransomware and store the backups remotely without storing a local (compressed) copy of the nextcloud data.

Alternatives (https://github.com/nextcloud/all-in-one?tab=readme-ov-file#are-remote-borg-backups-supported):

• Mount a network file system: Ransomware can encrypt the backups on the network storage.
• rclone: Ransomware and stores a local copy of the borg repo
• borg backup the local repo to borgbase: Stores a local compressed copy of the nextcloud data as at least 1 borg backup
• create your own solution: backups cannot be managed through the UI and others will have to reinvent my work

How to use it

Instead of entering a local repo path (which remains supported), you can choose to instead enter a remote borg repo url. (These screenshots are from a disaster recovery but it's analogous, just ignore the passphrase):

The first you try to initialise the repo, the backup container will create an ssh key and foolishly try to init the borg repo resulting in error because you have to authorise the ssh key it generated first. So, e.g. I copy paste the public key shown here to my borgbase repo:

Then I can try again as instructed and it should work. The following backup info is shown:

Changes

• Support for storing backups directly at a remote borg repo (Replace commands to not assume the repo is local, e.g. use BORG_REPO env var, ...)
• Exclude files from backups if we don't plan on restoring them anyway, such as the audit log
• Restore with borg extract because, at least with a remote repo, borg mount is very slow (20 seconds vs 24 minutes restore for pretty much an empty nextcloud install).

FYI Oddly the original code never restores host-mounts.

Testing

I hacked the code til I could run local deployments of nextcloud-aio without any builds and manually tested:

• Entering invalid stuff into forms I changed
• Remote borg repo init
• Creating and restoring a backup. I created some files in the master container volume; some in an ignored directory to make sure those are not deleted, some in a directory that is part of the backup, after the backup made some changes and made sure that once restored, the ignored files were left alone, new (non-ignored) files were deleted and changed files restored. I can't really open the user UI with my local test setup.
• Checking a backup
• Setting up a new nextcloud from backup

TODO

• I did at one point test local backups, but I should redo that.
• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great
• Update the readme, but I first want to hear your thoughts before putting more time in it

[szaimen]

Hi, first of all thank you for your contribution!

I've thought a bit about this the last week and came to the conclusion that I do not want to maintain this since it requires an additional server for me for testing (with a not so easy to reproduce setup) and has the potential of many users needing help getting this to work (and thus only a limited usecase). So I would still only test local backup also in the future.

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

FYI Oddly the original code never restores host-mounts.

Yes, this is expected and documented.

TODO

• I did at one point test local backups, but I should redo that.

Sounds good!

• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Yeah, we could create a dev instance from this if we proceed...

• Update the readme, but I first want to hear your thoughts before putting more time in it

Yes

[szaimen]

I would personally like to keep rsync at least for local archives since as you found out borg extract does not allow for deleting existing files and rsync is much faster at least for local archives... but we could split the logic and use rsync for local archives and borg extract for remote archives...

[timdiels]

Sure, I'll bring back the borg mount and rsync for local repos.

[timdiels]

Done

[timdiels]

Hi,

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

• I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Note to self:

• Update readme
• Local repo should use borg mount and rsync
• Test local and remote on dev build

[szaimen]

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

Cool, then we can go ahead with this PR. I've invited you to the repo for easier collaboration :)

• I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

Great :)

• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Usually it is indeed always via push to github, wait for build and deploy. So probably it is easier for you to keep your debug branch 👍

[gregjohnsonsaltaire]

Very much appreciate the PR guys!
I'm happy to help test informally ... I have 3 nextcloud-aio sites of 20GB, 40GB & 350GB doing borg backups to Hetzner Storageboxes that I'm prepared to convert to the PR ... Regards

[asavageiv]

Should this be "Likely borg init will error on a partially initialized repo"?

[timdiels]

Probably most borg commands would fail but it's referring to the borg info on line 120. If we ever end up with a partially initialized repo (sounds very rare), I think manual intervention is required anyway? The important thing is I believe borg info will fail on such a corrupt repo and will end up at line 124 instead of trying to initialize it again. The config file was used as a sentinel to detect whether the repo has been initialized already (see the deleted line 107), which why it is removed here and why it's probably no longer necessary to do this.

[timdiels]

I tried out local backups on my machine and made all the requested changes I believe. Could we get a a dev build going?

[timdiels]

Created a backup on my server, I'll enable autobackup/update and try out restoring after some changes later (might be next weekend):

Duration: 1 hours 2 minutes 27.77 seconds
Number of files: 34348
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                9.93 GB              8.89 GB              8.65 GB

(the duration is pretty much optimal considering my 19 Mbps ISP upload speed)

[timdiels]

I rolled back to an old snapshot because apparently the dev build was older than the latest released nextcloud, so it wouldn't start anymore because e.g. elastic can't roll back. I'll try again on the weekend with a tmp vm instead.

[msullivan-lc]

Appreciate all the work you are doing on this @timdiels. Very much looking forward to this addition!

[devnoname120]

@szaimen Is something missing before this pull request can be merged? I see it keeps being pushed back to the next release. I'm not sure of the significance of this as I'm not very familiar with the release process that Nextcloud AIO follows. Thanks!

[szaimen]

Mostly I am waiting for the results of the tests that @timdiels was doing...

Afterwards this needs conflict resolution, review and testing...

[timdiels]

Is there a way to get it to rebuild the develop-4804 image? My last change is not included in the image.

[timdiels]

Retested remote and local backups with the dev image and it all seems to work. This time I added a local caddy so I could actually test with adding a user, some files, some notes through the UI.

[timdiels]

Up to date again, I mostly only saw a switch from <br> to <p> in the merge conflict.

There's the one change in the backupscript that I haven't been able to test because I need a rebuild of the dev image for that. Other than that it's a go for me.

[msullivan-lc]

Hi @szaimen, my turn for a gentle ask on the status of this one? @timdiels seems to have brought things up to date and is ready to test. Really looking forward to this feature so I can roll out Nextcloud for my organization. Easy backup to a remote location is the only thing holding me up. I've tried several of the other documented methods but found them to either be unreliable or too complicated.

Please let us know.

Thanks!

[szaimen]

Hi @timdiels, sorry for my late reply. I was on vacation.

So my plan would be to include this for the next major release of AIO which will likely be released after Nc30.0.1 or 30.0.2 is out. Until then we have a bit of time to finish this.

I for now rebased your branch and pushed my attempts to #5206. If possible, I would like to continue working on that branch. Please review my rebase, possibly some details where going wrong as I did resolve the conflicts manually.

I also already built containers for that branch, using the develop-5206 tag.

---

However if you would like to continue working on this branch though, then I can say that yesterday evening, I updated the containers using the existing develop-4804 tag.

[szaimen]

Hi @timdiels can you please decide on which branch we should continue to work on?

[timdiels]

Hi @szaimen This branch just needs one more manual check on the newly built image but other than that it's ready to be merged as far as I'm concerned so it would seem less work to stick with this branch, no? Or you mean that during the next steps more work may pop up that needs to be done before it can be released?

[timdiels]

On the other hand, if we then would need to repeat the work to merge it into the next major release, then I would prefer using the new branch.

[timdiels]

I manually tested my last change, it fixes the problem. Looks good to merge for me.