General Protection Fault while looking up a file #16608
Comments
step_into() fs/namei.c#L1874 -> pick_link() fs/namei.c#L1775 -> may_follow_link() fs/namei.c#L1125 -> audit_inode() include/linux/audit.h#L375 |
|
OK, this really is not the first time we're seeing the issue, but it was harder to correlate after a myriad of config changes, like turning off selinux in the kernel config. This was with OpenZFS @ 8f2f6cd: step_into ends a bit earlier b/c there's selinux in the mix |
|
|
Interestingly, while this happens, arc_evict and arc_prune are both active at 70-100% cpu each. This is a list of processes in D state with their stacks after a few processes died with ^ similar oopses: |
Other than openzfs#16612, this is the only place I see where trouble similar to openzfs#16608 could arise. I'm not sure all the supported kernel versions will be okay with these modifications, I'm posting this as a draft to let the bots chew on it to find out. Signed-off-by: Pavel Snajdr <[email protected]>
Other than openzfs#16612, this is the only place I see where trouble similar to openzfs#16608 could arise. I'm not sure all the supported kernel versions will be okay with these modifications, I'm posting this as a draft to let the bots chew on it to find out. Signed-off-by: Pavel Snajdr <[email protected]>
|
OK now that we have those two ideas invalidated, now I'm thinking what's this @behlendorf if you have a nanosecond, could you take a look if I'm missing something, is that |
|
@snajpa I only took a quick look but the |
|
oh, right, I see |
|
Turns out it's because the OS is allowed to freely deallocate the znode even if there's a SA hold and maybe other conditions which I didn't yet check for. This patch seems to mitigate: diff --git a/module/os/linux/zfs/zfs_znode_os.c b/module/os/linux/zfs/zfs_znode_os.c
index a153d0850..87ff4facb 100644
--- a/module/os/linux/zfs/zfs_znode_os.c
+++ b/module/os/linux/zfs/zfs_znode_os.c
@@ -186,7 +186,7 @@ zfs_znode_init(void)
znode_cache = kmem_cache_create("zfs_znode_cache",
sizeof (znode_t), 0, zfs_znode_cache_constructor,
zfs_znode_cache_destructor, NULL, NULL, NULL,
- KMC_SLAB | KMC_RECLAIMABLE);
+ KMC_SLAB);
ASSERT(znode_hold_cache == NULL);
znode_hold_cache = kmem_cache_create("zfs_znode_hold_cache",ping @amotin it seems that maybe it's not that harmless marking as evictable, but perhaps it's also enabling some extra unexpected eviction, at least for znodes? I'll investigate more tomorrow, but now I have a confirmation that the template build completes in 4 hrs less than it took for the bug to really be hit with those traces :) => way faster + bug free build now |
@snajpa I don't believe KMC_RECLAIMABLE presence should do more than moving the allocation into a different memory region with more active reuse, and your change just hides it back deeper, which is not a solution for anything. |
|
So I think maybe I got it... now with the I'll also update the two previous PR with current state. Might get some sleep first though. Rough 2 weeks :D |
|
Also thank you @amotin you're absolutely right; it just took longer for problems to arrive, about 5000 more painful seconds of waiting for the first thing to go wrong with a first stack trace :D |
afa7b34 didn't account for when tmpfile gets linked, the inode was still I_NEW even after it was linked and should thus be hashed in Fixes openzfs#16608 Signed-off-by: Pavel Snajdr <[email protected]>
|
I love this :D squash one bug, uncover another one :-D you know what when I'm done with this I'll just send one PR and I'll close the two currently WIP, it's shaping up to be several small commits, seems like less reviewer overhead |
|
We really need to figure out low memory test scenarios... |
|
Current status: vpsfreecz/zfs@0ba322e...9190abd |
|
Current progress: vpsfreecz#1 -> the General Protection Fault is fixed by vpsfreecz@298053a (WIP, needs a configure test for older kernels) Still a few issues to figure out + not sure I'm gonna need to submit all the patches. As I said, a work in progress :) |
We have seen what appears to be this same thing in the wild: |
|
@allanjude this should be solved by #16788, I can't reproduce it when it's applied |
System information
Describe the problem you're observing
Turns out the problems logged in #16594 were always early after the machine has booted, but that wasn't the problem causing the crashes, this seems to be it. After we applied fix for the Bad Page State bug (b052035), this is what we're left with, at least it's without noise now :D
Something about this
step_intorings a bell though, I think I've seen it this year already, grepping through the log archive now.Describe how to reproduce the problem
Include any warning/errors/backtraces from the system logs
The text was updated successfully, but these errors were encountered: