Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Documentation enhancement] btrfs + dedupe tools can get you into a bad state if you have LOTS of data #1230

Open
Motophan opened this issue Aug 19, 2023 · 1 comment

Comments

@Motophan
Copy link

Motophan commented Aug 19, 2023

https://www.reddit.com/r/btrfs/comments/15vbcpi/parent_transid_verify_failed_but_o_usebackuproot/

Just a little sentence somewhere that notes something to the effect of

"When using hardlinks some COW filesystems like btrfs can get into a state with no free metadata left even though the filesystem has free space. This is caused by too many transactions happening in too short of a time period. Before hardlinking a large amount of data, consult your filesystem's documentation regarding this. With btrfs a simple command of for i in {0..90}; do btrfs balance start -dusage=$i /btrfs ; done will usually free up enough metadata space to process the large volume of transactions."

I did ~ 100k hardlinks at once and it blew up my filesystem (due to no free metadata space).
After this command it free'ed up enough space to get everything working again.

idk if this should even be in the documentation but basically btrfs will accept as many transactions as you throw at it and then process them in the background. Usually this is fine because you send it a sane amount of things to do at once. However if you send it a insane amount mergerfs will treat it like any filesystem (as it should) and receive the "ok done" syscall even though its processing the ramifications in the background. (this is operating correctly and as designed, not a bug). Instead of ratelimiting the transactions on mergerfs (since thats not really the purpose of mergerfs) a doc update to make users aware should be fine. Its not really mergerfs's problem or concern when a filesystem tells it everything is good when it is not good.

@trapexit
Copy link
Owner

I struggle with stuff like this. On the one hand this ultimately isn't a mergerfs concern in the sense it isn't unique to it. But the reality is a lot of users get themselves into *nix filesystem complexities through mergerfs. It is hard not to put 'filesystem 101' style comments in the docs due to that.

Right now the docs are too verbose. At least in their current form. My plan is to build a "readthedocs" style site in the not to distant future. When I have that kind of setup I think these kinds of filesystem specific notices / gotchas will be easier to fit in.

To that end... I'll keep this ticket open but I think will wait to add it till I do that documentation migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants