Simplify the repository a lot:

No repository transactions, no log-like appending, no append-only, no segments,
just using a key/value store for the individual chunks.

No locking yet.

Also:

mypy: ignore missing import
there are no library stubs for borgstore yet, so mypy errors without that option.

pyproject.toml: install borgstore directly from github
There is no pypi release yet.

use pip install -e . rather than python setup.py develop
The latter is deprecated and had issues installing the "borgstore from github" dependency.

It uses xxh64 hashes of the meta and data parts to verify their validity.
On a server with borg, this can be done server-side without the borg key.

The new RepoObj header has meta_size, data_size, meta_hash and data_hash.

borg transfer is primarily a general purpose archive transfer function
from borg2 to related borg2 repos.

but for upgrades from borg 1.x, we also need to support:
- rcreate with a borg 1.x "other repo"
- transfer with a borg 1.x "other repo"

Features:
- exclusive and non-exclusive locks
- acquire timeout
- lock auto-expiry (after 30mins of inactivity), lock refresh
- use tz-aware datetimes (in utc timezone) in locks

Also:
- document lock acquisition rules in the src
- increased default BORG_LOCK_WAIT to 10s
- better document with-lock test

Stale locks are ignored and automatically deleted.
Default: stale == 30 Minutes old.

lock.refresh() can be called frequently to avoid that an acquired lock becomes stale.
It does not do much if the last real refresh was recently.
After stale/2 time it checks and refreshes the locks in the store.

Update the repository3 code to call refresh frequently:
- get/put/list/scan
- inside check loop

repository:
- api/rpc support for get/put manifest
- api/rpc support to access the store

All chunks are separate objects in borgstore.

It also outputs some statistics and warns about missing/reappeared chunks.

This is now done in borg compact, so borg check does not need to care.

…later.

much faster and easier now, similar to what borg delete --force --force used to do.

considering that speed, no need for checkpointing anymore.

--stats does not work that way, thus it was removed. borg compact now shows some stats.

Note: this is the default cache implementation in borg 1.x,
it worked well, but there were some issues:

- if the local chunks cache got out of sync with the repository,
  it needed an expensive rebuild from the infos in all archives.
- to optimize that, a local chunks.archive.d cache was used to
  speed that up, but at the price of quite significant space needs.

AdhocCacheWithFiles replaced this with a non-persistent chunks cache,
requesting all chunkids from the repository to initialize a simplified
non-persistent chunks index, that does not do real refcounting and also
initially does not have size information for pre-existing chunks.

We want to move away from precise refcounting, LocalCache needs to die.

Lots of low-level code written back then to optimize runtime of some
functions.

We'll solve this differently by doing less stats, esp. if it is expensive to compute.

Dummy returns all-zero stats from that call.

Problem was that these values can't be computed from the chunks cache
anymore. No correct refcounts, often no size information.

Also removed hashindex.ChunkIndex.summarize (previously used by the above mentioned
.stats() call) and .stats_against (unused) for same reason.

Tests were a bit tricky as there is validation on 2 layers now:
- repository3 does an xxh64 check, finds most corruptions already
- on the archives level, borg also does an even stronger cryptographic check

This was used for an implementation detail of the borg 1.x
repository code, dumping uncommitted objects. Not needed any more.

Also remove local repository method scan_low_level, it was only used by --ghost.

This was an implementation specific "in on-disk order" list method that made sense
with borg 1.x log-like segment files only.

But we now store objects separately, so there is no "in on-disk order" anymore.

this heavily depended on having a repository index where the flags get stored.

we don't have that with borgstore.

For big repos, this might take a while, so at least have messages on debug level.

In the end, it will all depend on the borgstore backend that will be used,
so we better point to the borgstore project for details.

borg1 needed this due to its transactional / rollback behaviour:
if there was uncommitted stuff in the repo, next repo opening automatically
rolled back to last commit. thus we needed checkpoint archives to reference
chunks and commit the repo.

borg2 does not do that anymore, unused chunks are only removed when the
user invokes borg compact.

thus, if a borg create gets interrupted, the user can just run borg create
again and it will find some chunks are already in the repo, making progress
even if borg create gets frequently interrupted.

didn't do anything anyway in this implementation.

borg doesn't do precise refcounting anymore, so this is pretty useless.

We don't have precise refcounts, thus we can't compute these.

No precise refcounting, can't compute that inexpensively.

no "on-disk order" anymore.

Repository -> LegacyRepository
RemoteRepository -> LegacyRemoteRepository
borg.repository -> borg.legacyrepository
borg.remote -> borg.legacyremote

Repository3 -> Repository
RemoteRepository3 -> RemoteRepository
borg.repository3 -> borg.repository
borg.remote3 -> borg.remote

borg.locking -> borg.fslocking
borg.locking3 -> borg.storelocking

…ring

Note: LegacyRepository still returns [id, ...] and so does RemoteRepository.list,
if the remote repo is a LegacyRepository.

also: use LIST_SCAN_LIMIT

- compression factor
- dedup factor
- repo size

All values are approx. values without considering overheads.

…exceptions

When the AdhocCache(WithFiles) queries chunk IDs from the repo to build the chunks
index, it won't know their refcount and thus all chunks in the index have their
refcount at the MAX_VALUE (representing "infinite") and that would never decrease
nor could that ever reach zero and get the chunk deleted from the repo.

Only completely new chunks first written in the current borg run have a valid
refcount.

In some exception handlers, borg tried to clean up chunks that won't be used
by an item by decref'ing them. That is either:
- pointless due to refcount being at MAX_VALUE
- inefficient, because the user might retry the backup and would need to
  transmit these chunks to the repo again.

We'll just rely on borg compact ONLY to clean up any unused/orphan chunks.

.init_chunks has just built self.chunks using repository.list(), so don't
call that again, but just iterate over self.chunks.

also some other changes, making the code much simpler.

We don't care about unused or superseded repo objects any more here,
borg compact will deal with them.

That's the job of borg compact and not needed inside borg check.
check only needs to know if a chunk is present in the repo.

reuse_chunk is the complement of add_chunk for already existing chunks.

It doesn't do refcounting anymore.

.seen_chunk does not return the refcount anymore, but just whether the chunk exists.

If we add a new chunk, it immediately sets its refcount to MAX_VALUE, so
there is no difference anymore between previously existing chunks and new
chunks added. This makes the stats even more useless, but we have less complexity.

LocalCache used this to assemble a new overall chunks index from multiple
chunks.archive.d's single-archive chunks indexes.

…ackup#8347

otherwise the lock might become stale and could get
killed by any other borg process.

note: ThreadRunner class written by PyCharm AI and
only needed small enhancements. nice.

…gbackup#6039

Archives was built with a dictionary-like api,
but in future we want to go away from a
read-modify-write archives list.

previously, borg always read all archives entries, modified the
list in memory, wrote back to the repository (similar as borg 1.x
did).

now borg works directly with archives/* in the borgstore.

old borg just didn't commit the transaction and
thus caused a transaction rollback if not in
repair mode.

we can't do that anymore, thus we must avoid
modifying the repo if not in repair mode.

not for check and compact, these need an exclusive lock.

to try parallel repo access on same machine, same user,
one needs to use a non-locking cache implementation:

export BORG_CACHE_IMPL=adhoc

this is slow due the missing files cache in that implementation,
but unproblematic because no caches/indexes are persisted.

if the manifest file is missing, check generated *.1 *.2 ... archives although an entry for the correct name and id was already
present. BUG!

this is because if the manifest is lost, that does not imply
anymore that the complete archives directory is also lost, as it
did in borg 1.x.

Also improved log messages a bit.

borg delete and borg prune do a quick and dirty archive deletion,
just removing the archives directory entry for them.

--undelete-archives can still find the archive metadata objects
by completely scanning the repository and re-create missing
archives directory entries.

but only until borg compact would remove all unused data.

if only the manifest is missing or corrupted, do not run that
scan, it is not required for the manifest anymore.