prune attempted 22hrs rewrite, errored on "No space left on device" (checks for prevention, pruning strategy with less repack, salvaging the repo)

[LeoniePhiline]

Hi!

Yesterday, for testing my rustic backup launcher script, I had created two snapshots shortly after each other.
Naturally, I wanted to delete the first of the two almost identical snapshots afterwards.

I first ran forget, then prune.

Here's the prune output:

~user/.cargo/bin/rustic  \
  --repo "/run/media/user/external-hdd/backup" \
  --log-level info prune    
using no config file, none of these exist: /root/.config/rustic/rustic.toml, /etc/rustic/rustic.toml, ./rustic.toml
enter repository password: [hidden]
[INFO] repository local:/run/media/user/external-hdd/backup: password is correct.
[INFO] using cache at /root/.cache/rustic/72fb76a3b31221e44dbce296144c1a6d0cbe559be40aa8cb56a88bdb80d4ddb5
[00:00:01] reading index...               ████████████████████████████████████████        228/228                                                                                                                                                                                        [00:00:00] reading snapshots...           ████████████████████████████████████████         21/21                                                                                                                                                                                         [00:01:12] finding used blobs...          ████████████████████████████████████████         21/21                                                                                                                                                                                         [00:00:04] getting packs from repository...                                                                                                                                                                                                                                              to repack:     107759 packs,    4926667 blobs,    1.7 TiB
this removes:                      2623 blobs,    5.5 MiB
to delete:          1 packs,          1 blobs,      251 B
unindexed:          0 packs,         ?? blobs,        0 B
total prune:                       2624 blobs,    5.5 MiB
remaining:                      8991305 blobs,    1.8 TiB
unused size after prune:    1.0 GiB (0.05% of remaining size)

packs marked for deletion:          0,        0 B
 - complete deletion:               0,        0 B
 - keep marked:                     0,        0 B
 - recover:                         0,        0 B
error: couldn't write to buffer: `Os { code: 28, kind: StorageFull, message: "No space left on device" }`

(took 13h15m6s, rustic had estimated 22h ETA)

The backup drive is entirely filled up:

/dev/dm-3                2,7T    2,7T     0  100% /run/media/user/external-hdd

The repo is 1.8T of 2.7T in size. The rest of space filled is the duplication from the prune repacking.

---

1. Disproportionally large repack for tiny space saving

I believed prune would not do much work if only little data is to be removed. However, to remove 5.5 MiB, 1.7 TiB had to be repacked. I did not even have the idea for a second to attempt --dry-run.

Could rustic issue a warning with a Y/n prompt if the repackable data is so disproportionately large for the extremely little saved space?

---

2. Estimate in advance if free storage space is insufficient

Now my HDD is entirely filled up.

What can I do now? I need to bring my repository back in order and undo the half-done-then-crashed-prune.

I had naively expected pruning meant deleting data. Instead, my repository was duplicated.

Can this be prevented?
Can rustic predict how much data it will write on re-packing and refuse if storage space can be expected to be insufficient?
Can rustic perform an incremental repack, where not 1.7 TiB must be duplicated before the old version is removed? Or a selective repack, where most data is kept in place, and only the pack / blob containing the data to be pruned is rewritten?

---

Addendum: Using check:

using no config file, none of these exist: /root/.config/rustic/rustic.toml, /etc/rustic/rustic.toml, ./rustic.toml
enter repository password: [hidden]
[INFO] repository local:/run/media/user/external-hdd/backup: password is correct.
[INFO] using cache at /root/.cache/rustic/72fb76a3b31221e44dbce296144c1a6d0cbe559be40aa8cb56a88bdb80d4ddb5
[00:00:00] checking Snapshot in cache...  ████████████████████████████████████████  14.62 KiB/14.62 KiB  101.09 KiB/s (ETA 0s)
[00:00:02] checking Index in cache...     ███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  66.69 MiB/686.58 MiB 18.17 MiB/s  (ETA 18s)
[ERROR] Cached file Type: Index, Id: c14dee9c is not identical to backend!
[00:00:04] checking Index in cache...     ██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 116.80 MiB/686.58 MiB 21.85 MiB/s  (ETA 19s)
[ERROR] Cached file Type: Index, Id: 653c8931 is not identical to backend!
[00:00:05] checking Index in cache...     ██████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 174.31 MiB/686.58 MiB 24.34 MiB/s  (ETA 14s)
[ERROR] Cached file Type: Index, Id: 87db1b62 is not identical to backend!
[00:00:08] checking Index in cache...     ██████████████░░░░░░░░░░░░░░░░░░░░░░░░░░ 246.72 MiB/686.58 MiB 26.95 MiB/s  (ETA 14s)
[ERROR] Cached file Type: Index, Id: 208c788e is not identical to backend!
[00:00:10] checking Index in cache...     ████████████████░░░░░░░░░░░░░░░░░░░░░░░░ 284.11 MiB/686.58 MiB 27.77 MiB/s  (ETA 14s)
[ERROR] Cached file Type: Index, Id: a3452cf7 is not identical to backend!
[00:00:11] checking Index in cache...     ███████████████████░░░░░░░░░░░░░░░░░░░░░ 336.28 MiB/686.58 MiB 28.20 MiB/s  (ETA 11s)
[ERROR] Cached file Type: Index, Id: 7137f8ac is not identical to backend!
[00:00:14] checking Index in cache...     ██████████████████████████░░░░░░░░░░░░░░ 447.23 MiB/686.58 MiB 29.10 MiB/s  (ETA 7s)
[ERROR] Cached file Type: Index, Id: 8b3c35bc is not identical to backend!
[00:00:16] checking Index in cache...     ████████████████████████████░░░░░░░░░░░░ 489.21 MiB/686.58 MiB 29.26 MiB/s  (ETA 6s)
[ERROR] Cached file Type: Index, Id: 458a4de0 is not identical to backend!
[00:00:18] checking Index in cache...     ████████████████████████████████░░░░░░░░ 554.29 MiB/686.58 MiB 29.57 MiB/s  (ETA 4s)
[ERROR] Cached file Type: Index, Id: 94f0f58d is not identical to backend!
[00:00:19] checking Index in cache...     ██████████████████████████████████░░░░░░ 583.71 MiB/686.58 MiB 29.64 MiB/s  (ETA 3s)
[ERROR] Cached file Type: Index, Id: 741f4d90 is not identical to backend!
[00:00:19] checking Index in cache...     ██████████████████████████████████░░░░░░ 590.85 MiB/686.58 MiB 29.66 MiB/s  (ETA 3s)
[ERROR] Cached file Type: Index, Id: 3ad4df4c is not identical to backend!
[00:00:20] checking Index in cache...     ███████████████████████████████████░░░░░ 612.08 MiB/686.58 MiB 29.71 MiB/s  (ETA 2s)
[ERROR] Cached file Type: Index, Id: b3770f91 is not identical to backend!
[00:00:22] checking Index in cache...     ████████████████████████████████████████ 686.58 MiB/686.58 MiB 30.39 MiB/s  (ETA 0s)
[00:00:02] reading index...               ████████████████████████████████████████        409/409
[00:00:00] listing packs...               ⠁
[WARN] pack e017eb71 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 327adeca not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack e3ed8345 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 0f47fec1 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 9c037f3c not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 6a5fd620 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 180da54d not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack ec4a293e not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack b7f7be24 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 35eea1cb not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 5a8bde22 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack da6c2816 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack da943c07 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack e459c7ef not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 72d12934 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 403f04cb not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 7941102b not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 609b1193 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 1a9dcf62 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack cac8dc7b not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 71c7ca53 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack c63853cd not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 67f9aa51 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 0e252e74 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 22270b29 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack b08c2107 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 64144be3 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 88f28f65 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 88b7592e not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack d9549c6c not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 90a36aca not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 0b4f8bd3 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 6f83b24a not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[00:00:01] listing packs...
[00:00:00] cleaning up packs from cache...
[00:01:32] checking packs in cache...     ████████████████████████████████████████   3.45 GiB/3.45 GiB   38.16 MiB/s  (ETA 0s)
[00:00:00] reading snapshots...           ████████████████████████████████████████         21/21
[00:01:12] checking trees...              ████████████████████████████████████████         21/21                 

A run of check --read-data is still ongoing.

[aawsome]

Thanks a lot @LeoniePhiline for writing about your problems!

prune shouldn't repack so much data to free just 5.5 MiB - except it needs to repack the data in order to resize the packs. Could it be that this is a former restic repository you are now treating with rustic? restic uses much smaller pack sizes for this amount of data than rustic does - this would result in a one-time repack.
Anyway, can you post the output of rustic repoinfo? This should give an indication about current pack sizes...

[LeoniePhiline]

Could it be that this is a former restic repository you are now treating with rustic

I don't fully remember. I thought it was a pure rustic repo, but it might be that I am mistaken and had copied the restic repo over to this HDD to save some time. (I had imported from backintime into borg and restic before adding rustic as experimental third backup solution.)

Anyway, can you post the output of rustic repoinfo? This should give an indication about current pack sizes...

Here's repoinfo:

using no config file, none of these exist: /root/.config/rustic/rustic.toml, /etc/rustic/rustic.toml, ./rustic.toml
enter repository password: [hidden]
enter repository password: [hidden]
[INFO] repository local:/run/media/user/external-hdd/backup: password is correct.
[INFO] using cache at /root/.cache/rustic/72fb76a3b31221e44dbce296144c1a6d0cbe559be40aa8cb56a88bdb80d4ddb5
[00:00:02] scanning index...              ████████████████████████████████████████        409/409                                                                                    repository files

| File type |  Count | Total Size |
|-----------|--------|------------|
| Key       |      1 |      450 B |
| Snapshot  |     21 |   14.6 kiB |
| Index     |    409 |  686.6 MiB |
| Pack      | 121222 |    2.7 TiB |
| Total     | 121653 |    2.7 TiB |


| Blob type      |    Count | Total Size | Total Size in Packs |
|----------------|----------|------------|---------------------|
| Tree           |  5657878 |   20.4 GiB |             4.6 GiB |
| Data           |  7232637 |    3.1 TiB |             2.7 TiB |
| Tree to delete |    43871 |  218.9 MiB |            45.3 MiB |
| Data to delete |  1932106 |    1.0 TiB |             0.9 TiB |
| Total          | 14866492 |    4.1 TiB |             3.6 TiB |

| Blob type            | Pack Count | Minimum Size | Maximum Size |
|----------------------|------------|--------------|--------------|
| Tree packs           |        421 |        328 B |     18.5 MiB |
| Data packs           |     121283 |    671.6 kiB |     81.9 MiB |
| Tree packs to delete |         11 |      1.4 kiB |      6.2 MiB |
| Data packs to delete |      56859 |    966.4 kiB |     22.4 MiB |

[aawsome]

Ok, at least this shows that you have lots of packs which are too small - so prune was repacking so much because of too small packs. IIRC the ~22 MiB are the standard restic maximum pack size, so I would guess these to be restic packs. But anyway, it doesn't matter as you can handle the too small packs within prune as I already wrote.

[aawsome]

In general, if repacking because of pack sizes poses a problem, there are the following things, you can use to tweak:

• Use rustic config with --set-min-packsize-tolerate-percent (or --set-max-packsize-tolerate-percent if you already changed this) to change the tolerated pack sizes. This needs to be run just once for a repository. rustic config --set-min-packsize-tolerate-percent 0 usually prevents any pack-size indicated repacks.
• You can use rustic prune --no-resize to disable resizing packs for a specific prune run. If you want to have such settings permanent, please use rustic config as written above.
• If you want to resize packs, but don't want to resize all in one run, use prune --max-repack to limit the amount of data repacked within this prune run. E.g. prune --max-repack 10% or prune --max-repack 10GiB.

[LeoniePhiline]

• Use rustic config with --set-min-packsize-tolerate-percent (or --set-max-packsize-tolerate-percent if you already changed this) to change the tolerated pack sizes. This needs to be run just once for a repository. rustic config --set-min-packsize-tolerate-percent 0 usually prevents any pack-size indicated repacks.

Given the repoinfo I posted above, do you think it is necessary for me to change any of these values?

• If you want to resize packs, but don't want to resize all in one run, use prune --max-repack to limit the amount of data repacked within this prune run. E.g. prune --max-repack 10% or prune --max-repack 10GiB.

Thanks, that sounds very useful!

[aawsome]

I would suggest you to repack too small packs. Do it incrementally, if this works better for you!

[aawsome]

Moreover, sometimes the two-way pruning may result in unexpectedly not removing no-longer used packs at the end of the prune run. This is because prune can't know if a backup takes place in parallel which actually does need the packs. (The current behavior is extra-safe and does even keep packs which are repacked only in order to resize the packs - this can actually be improved in future...). If, however, you really know that you are running prune exclusively, you can use prune --instant-delete which does remove all repacked pack files at the end.

[aawsome]

And, of course, running prune --dry-run before doing something new, is always a good idea as it exactly shows what prune will do.

[LeoniePhiline]

Yeah, definitely. Pruning sounded so much like a delete-only operation that I had never even considered it. I'm sure rustic-rs/docs#24 will help! :)

[aawsome]

What can I do now? I need to bring my repository back in order and undo the half-done-then-crashed-prune.

I would do the following:

• ensure all automated backups to the repository are paused, so you can work exclusively on it.
• run prune --max-repack 0 --instant-delete --dry-run - it should show that only packs are removed.
• run without --dry-run if the dry-run shows what you expect.

[aawsome]

Ok - the first run actually did remove the unindexed packs. Now the problem is that rustic has to rebuild index files, and needs a litte space for them... (Could be optimized by rustic-rs/rustic_core#13, but I had never time to work on it... :-( )
All index files are ~600MiB. Do you think you could free about 1GiB or like that to let this step pass?

[LeoniePhiline]

Do you think you could free about 1GiB or like that to let this step pass?

Apart from the rustic repo, the only thing on this HDD is some 128 MiB backups of MBRs, and stuff like that. I'd delete that, but it's not large enough.

[aawsome]

Until the mentioned issue isn't tackled there is no official help, but I'll try to give you some advise on Discord..

[LeoniePhiline]

For the archive:

The solution was to drop the index during instant-delete prune with zero allowed repack, just to then recreate the index at the end of the prune operation.

In other words:

rustic prune --max-repack 0 --instant-delete --early-delete-index

(With --early-delete-index being available in rustic-core since rustic-rs/rustic_core#63.)

After the initial few Gigabytes had been cleared, repacking in larger and larger chunks, by repeating with increasing --max-repack from 10G to 100G and more (without --early-delete-index).

[simonsan]

Added an issue in the docs repo: rustic-rs/docs#24