Skip to content

Commit

Permalink
docs: update the repository filesystem docs
Browse files Browse the repository at this point in the history
In the end, it will all depend on the borgstore backend that will be used,
so we better point to the borgstore project for details.
  • Loading branch information
ThomasWaldmann committed Aug 22, 2024
1 parent 61354fb commit ef1eb69
Showing 1 changed file with 34 additions and 27 deletions.
61 changes: 34 additions & 27 deletions docs/usage/general/file-systems.rst.inc
Original file line number Diff line number Diff line change
@@ -1,30 +1,37 @@
File systems
~~~~~~~~~~~~

We strongly recommend against using Borg (or any other database-like
software) on non-journaling file systems like FAT, since it is not
possible to assume any consistency in case of power failures (or a
sudden disconnect of an external drive or similar failures).

While Borg uses a data store that is resilient against these failures
when used on journaling file systems, it is not possible to guarantee
this with some hardware -- independent of the software used. We don't
know a list of affected hardware.

If you are suspicious whether your Borg repository is still consistent
and readable after one of the failures mentioned above occurred, run
``borg check --verify-data`` to make sure it is consistent.

.. rubric:: Requirements for Borg repository file systems

- Long file names
- At least three directory levels with short names
- Typically, file sizes up to a few hundred MB.
Large repositories may require large files (>2 GB).
- Up to 1000 files per directory.
- rename(2) / MoveFile(Ex) should work as specified, i.e. on the same file system
it should be a move (not a copy) operation, and in case of a directory
it should fail if the destination exists and is not an empty directory,
since this is used for locking.
- Also hardlinks are used for more safe and secure file updating (e.g. of the repo
config file), but the code tries to work also if hardlinks are not supported.
We recommend using a reliable, scalable journaling filesystem for the
repository, e.g. zfs, btrfs, ext4, apfs.

Borg now uses the ``borgstore`` package to implement the key/value store it
uses for the repository.

It currently uses the ``file:`` Store (posixfs backend) either with a local
directory or via ssh and a remote ``borg serve`` agent using borgstore on the
remote side.

This means that it will store each chunk into a separate filesystem file
(for more details, see the ``borgstore`` project).

This has some pros and cons (compared to legacy borg 1.x's segment files):
Pros:
- Simplicity and better maintainability of the borg code.
- Sometimes faster, less I/O, better scalability: e.g. borg compact can just
remove unused chunks by deleting a single file and does not need to read
and re-write segment files to free space.
- In future, easier to adapt to other kinds of storage:
borgstore's backends are quite simple to implement.
A ``sftp:`` backend already exists, cloud storage might be easy to add.
- Parallel repository access with less locking is easier to implement.

Cons:

- The repository filesystem will have to deal with a big amount of files (there
are provisions in borgstore against having too many files in a single directory
by using a nested directory structure).
- Bigger fs space usage overhead (will depend on allocation block size - modern
filesystems like zfs are rather clever here using a variable block size).
- Sometimes slower, due to less sequential / more random access operations.

0 comments on commit ef1eb69

Please sign in to comment.