kernel panics on i386/amd64 9.2 when writing to mounted filesystem on zfs

[beyondcreed]

UPDATE: same happens on amd64 system when using pefs from pkgng, below is i386 example

uname -a
FreeBSD node3 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Fri Sep 27 03:52:52 UTC 2013 [email protected]:/usr/obj/usr/src/sys/GENERIC i386

pkg info pefs*

pefs-kmod-2013.10.13

Dec 22 14:15:18 node3 kernel: Fatal trap 12: page fault while in kernel mode
Dec 22 14:15:18 node3 kernel: cpuid = 0; apic id = 00
Dec 22 14:15:18 node3 kernel: fault virtual address = 0x4
Dec 22 14:15:18 node3 kernel: fault code = supervisor read, page not present
Dec 22 14:15:18 node3 kernel: instruction pointer = 0x20:0xc052a0e7
Dec 22 14:15:18 node3 kernel: stack pointer = 0x28:0xeffe2a08
Dec 22 14:15:18 node3 kernel: frame pointer = 0x28:0xeffe2a18
Dec 22 14:15:18 node3 kernel: code segment = base 0x0, limit 0xfffff, type 0x1b
Dec 22 14:15:18 node3 kernel: = DPL 0, pres 1, def32 1, gran 1
Dec 22 14:15:18 node3 kernel: processor eflags = interrupt enabled, resume, IOPL = 0
Dec 22 14:15:18 node3 kernel: current process = 1533 (vi)
Dec 22 14:15:18 node3 kernel: trap number = 12
Dec 22 14:15:18 node3 kernel: panic: page fault
Dec 22 14:15:18 node3 kernel: cpuid = 0

commands executed in following order ( note the use of camellia xts instead of aes xts)

UPDATE: same thing happens when default algorithm is in use ( AES128 XTS mode )

commands taken from https://wiki.freebsd.org/PEFS

mkdir ~/private ~/private.enc
pefs addchain -a camellia128 -f -Z ~/private.enc
pefs mount ~/private.enc ~/private
pefs addkey -c ~/private
cd ~/private
vi hello
trying to save file
kernel panics, system reboots

root on zfs setup as per: https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE

UPDATE:

pefs works on amd64 when using example from man pefs

       % mkdir ~/Private
       % pefs addchain -fZ ~/Private
       Enter parent key passphrase:
       Reenter parent key passphrase:
       % pefs mount ~/Private ~/Private
       % pefs addkey -c ~/Private
       Enter passphrase:
       ...
       % pefs unmount ~/Private

the only difference here is same directory is used in mount, could wiki have an error?

[beyondcreed]

closing this, it seems possibly wiki has error

[glk]

Can't reproduce on CURRENT, didn't install 9.2 vm to verify it yet.

What do you mean by "wiki error"?

Error log doesn't contain backtrace, backtrace would be really helpful.

Could you also check if problem persists if building pefs from ports. I don't know which kernel packages are build against. Kernel ABI is supposed to remain stable but could have been accidentally broken.

[beyondcreed]

by "wiki error" i meant this

https://wiki.freebsd.org/PEFS

pefs mount ~/private.enc ~/private

here private.enc and private are being used, in pefs man page example it is like this

pefs mount ~/Private ~/Private

If I use what wiki suggests, kernel panics, however the example in man page works fine

sorry didn't have backtrace, I can supply if this is still legitimate problem, however it is working fine for me now that I am using example from man page

yes the same problem persists even from ports, I tried building on amd64, but again this is only when I do "pefs mount ~/private.enc ~/private"

thanks.

On Tuesday 24 December 2013 20:01:50 Gleb Kurtsou wrote:

Can't reproduce on CURRENT, didn't install 9.2 vm to verify it yet.

What do you mean by "wiki error"?

Error log doesn't contain backtrace, backtrace would be really helpful.

Could you also check if problem persists if building pefs from ports. I don't know which kernel packages are build against. Kernel ABI is supposed to remain stable but could have been accidentally broken.

---

Reply to this email directly or view it on GitHub:
#12 (comment)

Alex Pavlovic - CEO
Taskforce-1 Industries Ltd.
http://www.taskforce-1.com
+1.888.630.5733 Ext. 500
Google+: [email protected]
Skype: alex.pavlovic.tf1

[glk]

I was able to reproduce it with 9.2 packages.

I believe the problem to be KABI breakage between 9.0 and 9.2 when VOP_ADD_WRITECOUNT was introduced.
Although it's not clear why "pefs mount /a /a" doesn't panic instantaneously..

The following VOP methods are initialized correctly:

  vop_add_writecount = 0xc7ab7360 <pefs_aes_encrypt>, 
  vop_spare4 = 0xc7ab7340 <pefs_aes_decrypt>, 
  vop_spare5 = 0xc7ab7310 <pefs_aes_keysetup>

Backtrace:

root@fbsd-92-i386:/var/crash # kgdb /boot/kernel/kernel.symbols vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x4
fault code      = supervisor read, page not present
instruction pointer = 0x20:0xc052a0e7
stack pointer           = 0x28:0xe7fd9a08
frame pointer           = 0x28:0xe7fd9a18
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, def32 1, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process     = 816 (vi)
trap number     = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xc0b1810f at kdb_backtrace+0x4f
#1 0xc0adf38f at panic+0x16f
#2 0xc0f4c563 at trap_fatal+0x323
#3 0xc0f4c668 at trap_pfault+0xf8
#4 0xc0f4da29 at trap+0x599
#5 0xc0f365cc at calltrap+0x6
#6 0xc0f6ef6a at VOP_ADD_WRITECOUNT_APV+0xaa
#7 0xc0b8e72c at vn_open_cred+0x57c
#8 0xc0b8e87b at vn_open+0x3b
#9 0xc0b89ccc at kern_openat+0x1ec
#10 0xc0b8a0e5 at kern_open+0x35
#11 0xc0b8a120 at sys_open+0x30
#12 0xc0f4ce53 at syscall+0x443
#13 0xc0f36661 at Xint0x80_syscall+0x21
Uptime: 3m3s
Physical memory: 1003 MB
Dumping 204 MB: 189 173 157 141 125 109 93 77 61 45 29 13

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/modules/pefs.ko...done.
Loaded symbols for /boot/modules/pefs.ko
Reading symbols from /boot/kernel/crypto.ko...Reading symbols from /boot/kernel/crypto.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/crypto.ko
#0  doadump (textdump=1) at pcpu.h:249
249 pcpu.h: No such file or directory.
    in pcpu.h
(kgdb) bt
#0  doadump (textdump=1) at pcpu.h:249
#1  0xc0adf0d5 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:449
#2  0xc0adf3d2 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:637
#3  0xc0f4c563 in trap_fatal (frame=0xe7fd99c8, eva=4)
    at /usr/src/sys/i386/i386/trap.c:1044
#4  0xc0f4c668 in trap_pfault (frame=0xe7fd99c8, usermode=0, eva=4)
    at /usr/src/sys/i386/i386/trap.c:896
#5  0xc0f4da29 in trap (frame=0xe7fd99c8) at /usr/src/sys/i386/i386/trap.c:555
#6  0xc0f365cc in calltrap () at /usr/src/sys/i386/i386/exception.s:170
#7  0xc052a0e7 in rijndael_encrypt (ctx=0x0, src=0x0, 
    dst=0xcd080700 "P\a\bН \a\bНа8jЗ4ь>Е")
    at /usr/src/sys/crypto/rijndael/rijndael-api.c:58
#8  0xc0f6ef6a in VOP_ADD_WRITECOUNT_APV (vop=0xc7abd580, a=0xe7fd9ae8)
    at vnode_if.c:4236
#9  0xc0b8e72c in vn_open_cred (ndp=0xe7fd9b5c, flagp=0xe7fd9c24, cmode=420, 
    vn_open_flags=0, cred=0xc6865800, fp=0xc59e94d0) at vnode_if.h:1791
#10 0xc0b8e87b in vn_open (ndp=0xe7fd9b5c, flagp=0xe7fd9c24, cmode=420, 
    fp=0xc59e94d0) at /usr/src/sys/kern/vfs_vnops.c:113
#11 0xc0b89ccc in kern_openat (td=0xc5bf02f0, fd=-100, 
    path=0x284cf090 <Address 0x284cf090 out of bounds>, pathseg=UIO_USERSPACE, 
    flags=1538, mode=438) at /usr/src/sys/kern/vfs_syscalls.c:1132
#12 0xc0b8a0e5 in kern_open (td=0xc5bf02f0, 
---Type <return> to continue, or q <return> to quit---q
path=0x284cf090 <Address 0x284cf090 Quit
) at /usr/src/sys/kern/vfs_syscalls.c:1083
#13 0xc0b8a120 in sys_open (td=0xc5bf02f0, uap=0xe7fd9ccc)
    at /usr/src/sys/kern/vfs_syscalls.c:1059
#14 0xc0f4ce53 in syscall (frame=0xe7fd9d08) at subr_syscall.c:135
#15 0xc0f36661 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:270
#16 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) fr 8
#8  0xc0f6ef6a in VOP_ADD_WRITECOUNT_APV (vop=0xc7abd580, a=0xe7fd9ae8)
    at vnode_if.c:4236
4236    vnode_if.c: No such file or directory.
    in vnode_if.c
(kgdb) p *vop
$1 = {vop_default = 0xc1261120, vop_bypass = 0, 
  vop_islocked = 0xc0b6a170 <vop_stdislocked>, 
  vop_lookup = 0xc0b67170 <vfs_cache_lookup>, 
  vop_cachedlookup = 0xc7ab6390 <pefs_lookup>, 
  vop_create = 0xc7ab55c0 <pefs_create>, vop_whiteout = 0, 
  vop_mknod = 0xc7ab4ed0 <pefs_mknod>, vop_open = 0xc7ab4250 <pefs_open>, 
  vop_close = 0xc7ab1390 <pefs_close>, vop_access = 0xc7ab3fb0 <pefs_access>, 
  vop_accessx = 0xc7ab3f40 <pefs_accessx>, 
  vop_getattr = 0xc7ab2610 <pefs_getattr>, 
  vop_setattr = 0xc7ab4020 <pefs_setattr>, vop_markatime = 0, 
  vop_read = 0xc7ab24f0 <pefs_read>, vop_write = 0xc7ab34e0 <pefs_write>, 
  vop_ioctl = 0xc7ab4380 <pefs_ioctl>, vop_poll = 0, vop_kqfilter = 0, 
  vop_revoke = 0, vop_fsync = 0xc0b6a460 <vop_stdfsync>, 
  vop_remove = 0xc7ab5450 <pefs_remove>, vop_link = 0xc7ab5350 <pefs_link>, 
  vop_rename = 0xc7ab5a20 <pefs_rename>, vop_mkdir = 0xc7ab58b0 <pefs_mkdir>, 
  vop_rmdir = 0xc7ab5730 <pefs_rmdir>, 
  vop_symlink = 0xc7ab5040 <pefs_symlink>, 
  vop_readdir = 0xc7ab3720 <pefs_readdir>, 
  vop_readlink = 0xc7ab1cd0 <pefs_readlink>, 
  vop_inactive = 0xc7ab1560 <pefs_inactive>, 
  vop_reclaim = 0xc7ab3050 <pefs_reclaim>, vop_lock1 = 0xc7ab2bb0 <pefs_lock>, 
  vop_unlock = 0xc7ab0f20 <pefs_unlock>, 
  vop_bmap = 0xc0b69a90 <vop_eopnotsupp>, 
  vop_strategy = 0xc0b6a030 <vop_panic>, 
---Type <return> to continue, or q <return> to quit---
  vop_getwritemount = 0xc7ab13f0 <pefs_getwritemount>, 
  vop_print = 0xc7ab1520 <pefs_print>, 
  vop_pathconf = 0xc7ab1290 <pefs_pathconf>, vop_advlock = 0, 
  vop_advlockasync = 0, vop_advlockpurge = 0, vop_reallocblks = 0, 
  vop_getpages = 0xc0b6a000 <vop_stdgetpages>, 
  vop_putpages = 0xc0b69fa0 <vop_stdputpages>, 
  vop_getacl = 0xc7ab13c0 <pefs_getacl>, vop_setacl = 0, vop_aclcheck = 0, 
  vop_closeextattr = 0, vop_getextattr = 0, vop_listextattr = 0, 
  vop_openextattr = 0, vop_deleteextattr = 0, vop_setextattr = 0, 
  vop_setlabel = 0, vop_vptofh = 0xc7ab1350 <pefs_vptofh>, 
  vop_vptocnp = 0xc0b6b130 <vop_stdvptocnp>, vop_allocate = 0, vop_advise = 0, 
  vop_unp_bind = 0, vop_unp_connect = 0, vop_unp_detach = 0, vop_is_text = 0, 
  vop_set_text = 0, vop_unset_text = 0, vop_get_writecount = 0, 
  vop_add_writecount = 0xc7ab7360 <pefs_aes_encrypt>, 
  vop_spare4 = 0xc7ab7340 <pefs_aes_decrypt>, 
  vop_spare5 = 0xc7ab7310 <pefs_aes_keysetup>}
(kgdb) q

[beyondcreed]

Just another update, on i386 mounting via same directory and trying to write as per below panics as well

   % mkdir ~/Private
   % pefs addchain -fZ ~/Private
   Enter parent key passphrase:
   Reenter parent key passphrase:
   % pefs mount ~/Private ~/Private
   % pefs addkey -c ~/Private
   Enter passphrase:
   % echo 'bye' > ~/Private/myfile

[beyondcreed]

Just builtworld with STABLE 10 on i386 as per below, mounting and writing works fine using man page example. Built pefs from ports afterwards.

       % mkdir ~/Private
       % pefs addchain -fZ ~/Private
       Enter parent key passphrase:
       Reenter parent key passphrase:
       % pefs mount ~/Private ~/Private
       % pefs addkey -c ~/Private
       % echo 'hello' > ~/Private/world

FreeBSD node3.grid1 10.0-PRERELEASE FreeBSD 10.0-PRERELEASE #1 r259961: Fri Dec 27 21:55:13 PST 2013 [email protected]:/usr/obj/usr/src/sys/NODE3-GRID1 i386

[glk]

I've send follow up to the same issue discussed in Nov on stable@. No response yet.

http://marc.info/?l=freebsd-stable&m=138795016121536&w=2

[droe]

FWIW, I can confirm this on two 9.2-RELEASE-p3 systems, amd64, GENERIC kernel installed by freebsd-update, on UFS, with pefs installed through pkgng (pefs-kmod-2013.10.13), crashes kernel on first write:

% mkdir ~/Private
% pefs mount ~/Private ~/Private
% pefs addkey -c ~/Private
% echo test > ~/Private/test # <-- crashes the kernel

kgdb backtrace (via panicmail):

#0  doadump (textdump=<value optimized out>) at pcpu.h:234             
#1  0xffffffff8090d486 in kern_reboot (howto=260)                             
    at /usr/src/sys/kern/kern_shutdown.c:449                                  
#2  0xffffffff8090d987 in panic (fmt=0x1 <Address 0x1 out of bounds>)         
    at /usr/src/sys/kern/kern_shutdown.c:637                                  
#3  0xffffffff80b79eca in vm_fault_hold (map=0xfffffe0002000000,              
    vaddr=18446743523957809152, fault_type=1 '\001', fault_flags=0,           
    m_hold=0x0) at /usr/src/sys/vm/vm_fault.c:285                             
#4  0xffffffff80b7a663 in vm_fault (map=0xfffffe0002000000,                   
    vaddr=18446743523957809152, fault_type=<value optimized out>,             
    fault_flags=0) at /usr/src/sys/vm/vm_fault.c:229                          
#5  0xffffffff80cf264f in trap_pfault (frame=0xffffff80003e1700, usermode=0)  
    at /usr/src/sys/amd64/amd64/trap.c:772                                  
#6  0xffffffff80cf2a24 in trap (frame=0xffffff80003e1700)                     
    at /usr/src/sys/amd64/amd64/trap.c:463                                    
#7  0xffffffff80cdbd53 in calltrap ()                                         
    at /usr/src/sys/amd64/amd64/exception.S:232                               
#8  0xffffffff80343628 in rijndaelEncrypt (rk=0xffffff80003e1fe8,             
    Nr=<value optimized out>, pt=<value optimized out>,                       
    ct=0x4 <Address 0x4 out of bounds>)                                       
    at /usr/src/sys/crypto/rijndael/rijndael-alg-fst.c:966                    
#9  0xffffffff80d968d8 in VOP_ADD_WRITECOUNT_APV (vop=0xffffffff81851ea0,     
    a=0xffffff80003e1920) at vnode_if.c:4236                                  
#10 0xffffffff809bad48 in vn_open_cred (ndp=0xffffff80003e19d0,               
    flagp=0xffffff80003e19cc, cmode=420, vn_open_flags=<value optimized out>, 
    cred=0xfffffe00028b9e00, fp=0xfffffe0002abc7d0) at vnode_if.h:1791        
#11 0xffffffff809b676a in kern_openat (td=0xfffffe00028c9490, fd=-100,        
    path=0x80141f9c0 <Address 0x80141f9c0 out of bounds>,                     
    pathseg=UIO_USERSPACE, flags=1538, mode=<value optimized out>)            
    at /usr/src/sys/kern/vfs_syscalls.c:1132                                  
#12 0xffffffff80cf18ba in amd64_syscall (td=0xfffffe00028c9490, traced=0)     
    at subr_syscall.c:135                                                     
#13 0xffffffff80cdc037 in Xfast_syscall ()                                    
    at /usr/src/sys/amd64/amd64/exception.S:391                               
#14 0x0000000800de0d3c in ?? ()