Confusion about SFTP-URLs

[mirko]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

QUESTION

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

2.0.0b11.dev3+gc278a8e8

Operating system (distribution) and version.

Debian testing/Trixie

Full borg commandline that lead to the problem (leave away excludes and passwords)

export BORG_REPO='sftp://HOST:/storage/backup/borg2' ; borg repo-create -e keyfile-blake2-chacha20-poly1305

Describe the problem you're observing.

It's not entirely clear from the docs how to initialize a repo when using the SFTP backend - and if (needed) at all.

https://borgbackup.readthedocs.io/en/master/usage/general.html#repository-urls lists repository-URLS - incl. sftp:// ones, also stating it might make sense to store the URL in the env var BORG_REPO.

Env-var BORG_REPO is also used by repo-create, however fails when URL indicates sftp-backend.

It appears confusing/unclear how to create and use a repo via the SFTP backend, especially since the Quick-Start (https://borgbackup.readthedocs.io/en/master/quickstart.html#a-step-by-step-example) states repo-create as first step - no mention of e.g. "don't use with SFTP-backend".

Log

Local Exception

Error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/mirko/sftp:/HOST:/storage/backup/borg2'

If reporting bugs, please include the following:

Traceback (most recent call last):
  File "/opt/pipx/venvs/borgbackup/lib/python3.11/site-packages/borg/archiver/__init__.py", line 622, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/opt/pipx/venvs/borgbackup/lib/python3.11/site-packages/borg/archiver/__init__.py", line 516, in run
    rc = func(args)
         ^^^^^^^^^^
  File "/opt/pipx/venvs/borgbackup/lib/python3.11/site-packages/borg/archiver/_common.py", line 151, in wrapper
    with repository:
  File "/opt/pipx/venvs/borgbackup/lib/python3.11/site-packages/borg/repository.py", line 132, in __enter__
    self.create()
  File "/opt/pipx/venvs/borgbackup/lib/python3.11/site-packages/borg/repository.py", line 146, in create
    self.store.create()
  File "/opt/pipx/venvs/borgbackup/lib/python3.11/site-packages/borgstore/store.py", line 51, in create
    self.backend.create()
  File "/opt/pipx/venvs/borgbackup/lib/python3.11/site-packages/borgstore/backends/posixfs.py", line 40, in create
    self.base_path.mkdir()
  File "/usr/lib/python3.11/pathlib.py", line 1117, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/mirko/sftp:/HOST:/storage/backup/borg2'

Platform: Linux thar 6.10.9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.10.9-1 (2024-09-08) x86_64
Linux: Unknown Linux  
Borg: 2.0.0b11.dev3+gc278a8e8  Python: CPython 3.11.9 msgpack: 1.1.0 fuse: None [pyfuse3,llfuse]
PID: 60885  CWD: /home/mirko
sys.argv: ['/usr/local/bin/borg', 'repo-create', '-e', 'keyfile-blake2-chacha20-poly1305']
SSH_ORIGINAL_COMMAND: None

[mirko]

Issue appears to be much simpler than that:

$ borg -r sftp://HOST:/storage/backup/borg2 create TEST data/
Repository proto='file', user=None, host=None, port=None, path='sftp:/HOST:/storage/backup/borg2' does not exist.

SFTP is not recognised as its own backend, but URL is parsed as proto file / file-path.

[mirko]

Oh, apparently there's 2 things:

• unlike "used to" SFTP-URLs, the ones for borg are not allowed to have a ':' except when specifying a port
• SFTP-URLs need to have a user secified (USER@)
  So what works is: sftp://USER@HOST:22/storage/backup/borg2 as URL

[mirko]

I see the mandatory user was already found out (#8373 (comment)), addressed (borgbackup/borgstore#26) and merged.

However borg/master still references version v0.0.1 of borgstore (not that v0.0.2 would make a difference here, as it got merged after v0.0.2 was released)

[ThomasWaldmann]

borg2 b11 and current master branch requires borgstore ~= 0.0.1, so it will automatically use latest 0.0.x from pypi.

[ThomasWaldmann]

Also, I think having a bare : without a port number in a sftp (or ssh) URL is a invalid URL.

The : is the separator between hostname and port and it is only used IF a port number is given (which is not required if you want to used the default port 22).

Side note (not relevant here, just related and curious): scp-style (user@host:path) have the : as host/path separator and one can NOT give a port number, but one can use relative paths NOT starting with a /.

[ThomasWaldmann]

In the traceback, there seems to be another problem: using / instead of // to introduce the host part:

FileNotFoundError: [Errno 2] No such file or directory: '/home/mirko/sftp:/HOST:/storage/backup/borg2'

So, in the end, because you didn't start with sftp://, it rather looked like a local path for borg's matching. Local paths on POSIX OSes are allowed to contain :.

[ThomasWaldmann]

TODO: update borg sftp url docs (add more examples / remarks) after borgstore 0.0.3 is out.

[mirko]

While you're right and I was wrong about most parts (e.g. ':' only used in SCP URLs for indicating absolute paths):

So, in the end, because you didn't start with sftp://, it rather looked like a local path

I did start ALL my URLs with sftp://. But missing username (while fixed meanwhile in master/HEAD) or putting a ':' in it make it a "file"-URL. I guess that's because the URL scheme fails the regexp here https://github.com/borgbackup/borgstore/blob/master/src/borgstore/backends/sftp.py#L21 maybe?

[ThomasWaldmann]

Read the "FileNotFoundError" line again: there is only a single slash, not sftp://.

Hmm, maybe that could be also caused by some path normalisation? So you gave double slash, but as it thought it is a file path, it normalised it to only contain single slash separators.

[mirko]

Read the "FileNotFoundError" line again: there is only a single slash, not sftp://.

I mean, just try it yourself :)

$ export BORG_REPO='sftp://HOST:/storage/backup/borg2' ; borg repo-create -e keyfile-blake2-chacha20-poly1305
Local Exception

Error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/mirko/sftp:/HOST:/storage/backup/borg2'

[ThomasWaldmann]

borgstore 0.0.3 released!

https://github.com/borgbackup/borgstore/releases/tag/0.0.3