Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

files cache improvements #8389

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

ThomasWaldmann
Copy link
Member

@ThomasWaldmann ThomasWaldmann commented Sep 17, 2024

Fixes #8385.

@codecov-commenter
Copy link

codecov-commenter commented Sep 18, 2024

Codecov Report

Attention: Patch coverage is 69.89247% with 28 lines in your changes missing coverage. Please review.

Project coverage is 81.43%. Comparing base (f1a39a0) to head (381c33a).
Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
src/borg/cache.py 70.51% 18 Missing and 5 partials ⚠️
src/borg/archive.py 55.55% 2 Missing and 2 partials ⚠️
src/borg/archiver/list_cmd.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8389      +/-   ##
==========================================
- Coverage   81.54%   81.43%   -0.11%     
==========================================
  Files          70       70              
  Lines       12717    12712       -5     
  Branches     2297     2306       +9     
==========================================
- Hits        10370    10352      -18     
- Misses       1694     1704      +10     
- Partials      653      656       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ThomasWaldmann ThomasWaldmann force-pushed the files-cache-from-archive branch 3 times, most recently from 8456761 to 219a47a Compare September 19, 2024 19:39
@ThomasWaldmann ThomasWaldmann marked this pull request as ready for review September 19, 2024 19:46
@ThomasWaldmann ThomasWaldmann changed the title replace files cache by reading previous archive files cache improvements Sep 19, 2024
@ThomasWaldmann ThomasWaldmann force-pushed the files-cache-from-archive branch 2 times, most recently from f1b3c5c to f452b97 Compare September 19, 2024 21:43
- changes to locally stored files cache:

  - store as files.<H(archive_name)>
  - user can manually control suffix via env var
  - if local files cache is not found, build from previous archive.
- enable rebuilding the files cache via loading the previous
  archive's metadata from the repo (better than starting with
  empty files cache and needing to read/chunk/hash all files).
  previous archive == same archive name, latest timestamp in repo.
- remove AdHocCache (not needed any more, slow)
- remove BORG_CACHE_IMPL, we only have one
- remove cache lock (this was blocking parallel backups to same
  repo from same machine/user).

Cache entries now have ctime AND mtime.

Note: TTL and age still needed for discarding removed files.
      But due to the separate files caches per series, the TTL
      was lowered to 2 (from 20).
if we detect the conditions for this (rare) race,
abort reading the file and retry.

The caller (_process_any) will do up to MAX_RETRIES
before giving up. If it gives up, a warning is logged
and the file is not written to the archive and won't
be memorized in the files cache either.

Thus, the file will be read/chunked/hashed again at
the next borg create run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

improve files cache
2 participants