Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase default LKL memory to 64M? #130

Open
letmaik opened this issue May 1, 2020 · 3 comments
Open

Increase default LKL memory to 64M? #130

letmaik opened this issue May 1, 2020 · 3 comments
Assignees
@mikbras
Labels
area: disk Related to mounted disk images/encryption enhancement p1 Medium priority
Milestone
Milestone 1

Comments

@letmaik
Copy link
Contributor

letmaik commented May 1, 2020

cryptsetup allocates/uses LKL memory (and not enclave mmap memory). The default size for LKL visible memory is 32MB. Using dm-crypt+dm-integrity I stumbled over the following. Increasing the size to 64M fixed it. Should 64M become a new default? It may be nontrivial for users to diagnose this themselves.

[   SGX-LKL  ] SGX-LKL (OE) Git version fc08365-dirty LKL version 5.3.0 [DEBUG build (-O0)] [HARDWARE DEBUG]
[   SGX-LKL  ] nproc=4 ETHREADS=4 CMDLINE="mem=32M" GETTIME_VDSO=0
[   SGX-LKL  ] HW TLS support: conf->fsgsbase=1
[   SGX-LKL  ] Registering disk 0 (path='.../disk1.img', mnt='/', [RW   ])
[   SGX-LKL  ] Registering disk 1 (path='.../disk2.img', mnt='/data/input', [RW   ])
[   SGX-LKL  ] Registering disk 2 (path='.../disk3.img', mnt='/data/output', [RW   ])
[   SGX-LKL  ] get_signed_libsgxlkl_path... result=/opt/sgx-lkl/lib/libsgxlkl.so.signed
[   SGX-LKL  ] oe_create_enclave... 
[   SGX-LKL  ] sgxlkl_enclave_init(ethread_id=0)
[[  SGX-LKL ]] sgxlkl_enclave_init(): enter
[[  SGX-LKL ]] sgxlkl_enclave_show_attribute(): enclave base=0x7fbc00000000 size=8.042 GB
[[  SGX-LKL ]] sgxlkl_enclave_show_attribute(): enclave heap base=0x7fbc00b3f000 size=8.000 GB end=0x7fbe00b3f000
[[  SGX-LKL ]] sgxlkl_enclave_init(): calling _dlstart_c()
[[  SGX-LKL ]] _register_enclave_signal_handlers(): Registering OE exception handler...
[[  SGX-LKL ]] lkl_start_init(): kernel command line: 'mem=32M console=hvc0'
[[  SGX-LKL ]] lkl_start_init(): lkl_start_kernel() called
[    0.000000] Linux version 5.3.0+ (@1784b8ead1a7) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #1 Thu Apr 23 10:42:08 UTC 2020
[    0.000000] memblock address range: 0x7fbdfe7bc000 - 0x7fbe007bb000
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 8079
[    0.000000] Kernel command line: mem=32M console=hvc0 virtio_mmio.device=256@0x1000000:1 virtio_mmio.device=273@0x2000000:2
[    0.000000] Dentry cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.000000] Inode-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory available: 32172k/32764k RAM
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 4096
[    0.000000] lkl: irqs initialized
[    0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000015] lkl: time and timers initialized (irq3)
[    0.000701] printk: console [hvc0] enabled
[    0.000743] pid_max: default: 4096 minimum: 301
[    0.000825] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.000891] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.003983] devtmpfs: initialized
[    0.007684] random: get_random_bytes called from 0x00007fbc0065ab07 with crng_init=0
[    0.007809] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.007889] xor: automatically using best checksumming function   8regs     
[    0.008125] NET: Registered protocol family 16
[    0.015031] clocksource: Switched to clocksource lkl
[    0.015450] NET: Registered protocol family 2
[    0.016292] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.016352] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.016408] TCP bind hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.016455] TCP: Hash tables configured (established 512 bind 512)
[    0.016553] UDP hash table entries: 128 (order: 0, 4096 bytes, linear)
[    0.016603] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes, linear)
[    0.016694] NET: Registered protocol family 1
[    0.016799] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x10000ff, IRQ 1.
[    0.016875] virtio-mmio: Registering device virtio-mmio.1 at 0x2000000-0x2000110, IRQ 2.
[    0.017166] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[    0.018449] NET: Registered protocol family 38
[    0.018513] io scheduler mq-deadline registered
[    0.018552] io scheduler kyber registered
[    0.124788] software IO TLB: mapped [mem 0x7fc054565000-0x7fc058565000] (64MB)
[    0.126937] random: fast init done
[    0.137450] loop: module loaded
[    0.137949] device-mapper: ioctl: 4.40.0-ioctl (2019-01-18) initialised: [email protected]
[    0.146540] wireguard: WireGuard 0.0.20191219 loaded. See www.wireguard.com for information.
[    0.146603] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
[    0.146921] NET: Registered protocol family 10
[    0.147761] Segment Routing with IPv6
[    0.147819] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.148451] This architecture does not have kernel memory protection.
[    0.148499] Run /init as init process
[[  SGX-LKL ]] lkl_start_init(): lkl_start_kernel() finished
[[  SGX-LKL ]] lkl_start_init(): creating LKL termination thread
[[  SGX-LKL ]] create_lkl_termination_thread(): enter
[[  SGX-LKL ]] lkl_start_init(): calling lkl_mount_virtial()
[[  SGX-LKL ]] lkl_termination_thread(): enter
[[  SGX-LKL ]] lkl_start_init(): calling init_random()
[[  SGX-LKL ]] init_random(): Adding entropy to entropy pool
[[  SGX-LKL ]] lkl_termination_thread(): Performed LKL syscall to get host task allocated (pid=36)
[    0.149748] random: crng init done
[[  SGX-LKL ]] wg0 has public key BPY74ddJHpWW42M5wWmlFIuOGl95auTNPZJUTXvXKXs=
[[  SGX-LKL ]] aas_release_resources(): aas_release_resources: deallocate all resources
[    0.153640] virtio_blk virtio2: [vda] 2017648 512-byte logical blocks (1.03 GB/985 MiB)
[    0.154317] virtio_blk virtio3: [vdb] 9026720 512-byte logical blocks (4.62 GB/4.30 GiB)
[    0.154735] virtio_blk virtio4: [vdc] 6291456 512-byte logical blocks (3.22 GB/3.00 GiB)
[[  SGX-LKL ]] lkl_mount_disk(): lkl_mount_disk(dev="/dev/vda", mnt="/mnt/vda", ro=0)
[[  SGX-LKL ]] lkl_mount_disk(): Activating crypto disk
[    5.835475] EXT4-fs (dm-1): mounted filesystem without journal. Opts: 
[[  SGX-LKL ]] lkl_mount_disk(): lkl_mount_disk(dev="/dev/vdb", mnt="/data/input", ro=0)
[[  SGX-LKL ]] lkl_mount_disk(): Activating crypto disk
[   11.471980] host3 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[   11.472047] Call Trace:
[   11.472070] 0000000045b62742:  [<7fbc000d9123>] 0x00007fbc000d9123
[   11.472115] 000000002184529f:  [<7fbc00086876>] 0x00007fbc00086876
[   11.472170] 000000008c1b9c1e:  [<7fbc003ee471>] 0x00007fbc003ee471
[   11.472221] 00000000a05b6691:  [<7fbc000d989b>] 0x00007fbc000d989b
[   11.472263] 00000000bc9f1796:  [<7fbc000d9489>] 0x00007fbc000d9489
[   11.472321] 00000000c9321b8e:  [<7fbc000d9c58>] 0x00007fbc000d9c58
[   11.472367] 00000000b535166a:  [<7fbc000f2f6b>] 0x00007fbc000f2f6b
[   11.472413] 0000000074feb5d2:  [<7fbc000fabf1>] 0x00007fbc000fabf1
[   11.472459] 00000000c66dbb31:  [<7fbc002818be>] 0x00007fbc002818be
[   11.472504] 00000000981a4c0b:  [<7fbc002819b6>] 0x00007fbc002819b6
[   11.472551] 00000000dc7b3603:  [<7fbc0020df9e>] 0x00007fbc0020df9e
[   11.472597] 0000000032178354:  [<7fbc0027138f>] 0x00007fbc0027138f
[   11.472644] 0000000054c3665b:  [<7fbc0027f735>] 0x00007fbc0027f735
[   11.472690] 
[   11.472712] Mem-Info:
[   11.472736] active_anon:0 inactive_anon:0 isolated_anon:0
[   11.472736]  active_file:1 inactive_file:1 isolated_file:0
[   11.472736]  unevictable:1 dirty:0 writeback:0 unstable:0
[   11.472736]  slab_reclaimable:256 slab_unreclaimable:1410
[   11.472736]  mapped:0 shmem:0 pagetables:0 bounce:0
[   11.472736]  free:179 free_pcp:0 free_cma:0
[   11.473930] Node 0 active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[   11.474064] Normal free:716kB min:716kB low:892kB high:1068kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:4kB writepending:0kB present:32764kB managed:32172kB mlocked:0kB kernel_stack:168kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[   11.474375] lowmem_reserve[]: 0 0
[   11.474405] Normal: 1*4kB (U) 1*8kB (M) 2*16kB (UM) 1*32kB (U) 0*64kB 1*128kB (M) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 716kB
[   11.474489] 3 total pagecache pages
[   11.474519] 8191 pages RAM
[   11.474540] 0 pages HighMem/MovableOnly
[   11.474570] 148 pages reserved
[   11.474600] Unreclaimable slab info:
[   11.474787] Name                      Used          Total
[   11.474824] bio-4                     34KB        108KB
[   11.474861] bio-3                      8KB         15KB
[   11.474899] bio-2                     11KB         26KB
[   11.474937] RAWv6                     15KB         15KB
[   11.474984] UDPv6                     29KB         47KB
[   11.475024] bio-1                      3KB          3KB
[   11.475097] RAW                        7KB          7KB
[   11.475282] UDP                        7KB          7KB
[   11.475321] request_queue             31KB         31KB
[   11.475360] biovec-max               672KB        672KB
[   11.475398] biovec-64                  8KB         32KB
[   11.475436] bio_integrity_payload         31KB         43KB
[   11.475474] skbuff_head_cache          3KB          3KB
[   11.475512] file_lock_cache            3KB          3KB
[   11.475549] file_lock_ctx              3KB          3KB
[   11.475739] proc_dir_entry          1559KB       1559KB
[   11.475777] pde_opener                 3KB          3KB
[   11.475811] seq_file                   3KB          3KB
[   11.475850] kernfs_node_cache       1087KB       1087KB
[   11.475888] mnt_cache                  7KB          7KB
[   11.475926] names_cache               32KB         32KB
[   11.475964] nsproxy                    6KB          7KB
[   11.476002] vm_region                  3KB          3KB
[   11.476176] vm_area_struct             3KB          3KB
[   11.476212] mm_struct                  7KB          7KB
[   11.476251] fs_cache                   6KB          8KB
[   11.476289] files_cache                7KB          7KB
[   11.476327] signal_cache              39KB         39KB
[   11.476366] sighand_cache             61KB         61KB
[   11.476405] task_struct               61KB         63KB
[   11.476594] cred_jar                   8KB          8KB
[   11.476634] pid                        7KB          7KB
[   11.476676] pool_workqueue           138KB        160KB
[   11.476714] kmalloc-8k               120KB        128KB
[   11.476762] kmalloc-4k               128KB        160KB
[   11.476801] kmalloc-2k               256KB        256KB
[   11.476839] kmalloc-1k               230KB        232KB
[   11.476877] kmalloc-512              368KB        368KB
[   11.477065] kmalloc-256              180KB        180KB
[   11.477104] kmalloc-192               31KB         31KB
[   11.477143] kmalloc-128               18KB         36KB
[   11.477180] kmalloc-96                19KB         19KB
[   11.477220] kmalloc-64                15KB         16KB
[   11.477258] kmalloc-32                 9KB         16KB
[   11.477296] kmalloc-16                 4KB          4KB
[   11.477334] kmalloc-8                 20KB         20KB
[   11.477529] kmem_cache_node            8KB          8KB
[   11.477564] kmem_cache                23KB         23KB
[   11.477606] Tasks state (memory values in pages):
[   11.477642] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[   11.477706] [     36]     0    36      162        0        0        0             0 host0
[   11.477760] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=host0,pid=36,uid=0
[   11.477825] Out of memory: Killed process 36 (host0) total-vm:648kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[   11.478049] oom killer 36 (host0) has mm pinned by 1 (init)
[   11.478281] host3 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[   11.478342] Call Trace:
[   11.478511] 00000000bdf6c896:  [<7fbc000d9123>] 0x00007fbc000d9123
[   11.478559] 00000000bc9f1796:  [<7fbc00086876>] 0x00007fbc00086876
[   11.478606] 00000000c9321b8e:  [<7fbc000d9f31>] 0x00007fbc000d9f31
[   11.478652] 00000000b535166a:  [<7fbc000f2f6b>] 0x00007fbc000f2f6b
[   11.478699] 0000000074feb5d2:  [<7fbc000fabf1>] 0x00007fbc000fabf1
[   11.478745] 00000000c66dbb31:  [<7fbc002818be>] 0x00007fbc002818be
[   11.478793] 00000000981a4c0b:  [<7fbc002819b6>] 0x00007fbc002819b6
[   11.478989] 00000000dc7b3603:  [<7fbc0020df9e>] 0x00007fbc0020df9e
[   11.479035] 0000000032178354:  [<7fbc0027138f>] 0x00007fbc0027138f
[   11.479081] 0000000054c3665b:  [<7fbc0027f735>] 0x00007fbc0027f735
[   11.479128] 
[   11.479150] Mem-Info:
[   11.479173] active_anon:0 inactive_anon:0 isolated_anon:0
[   11.479173]  active_file:1 inactive_file:1 isolated_file:0
[   11.479173]  unevictable:1 dirty:0 writeback:0 unstable:0
[   11.479173]  slab_reclaimable:256 slab_unreclaimable:1410
[   11.479173]  mapped:0 shmem:0 pagetables:0 bounce:0
[   11.479173]  free:179 free_pcp:0 free_cma:0
[   11.479520] Node 0 active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[   11.479655] Normal free:716kB min:716kB low:892kB high:1068kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:4kB writepending:0kB present:32764kB managed:32172kB mlocked:0kB kernel_stack:168kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[   11.479971] lowmem_reserve[]: 0 0
[   11.480001] Normal: 1*4kB (U) 1*8kB (M) 2*16kB (UM) 1*32kB (U) 0*64kB 1*128kB (M) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 716kB
[   11.480079] 3 total pagecache pages
[   11.480109] 8191 pages RAM
[   11.480131] 0 pages HighMem/MovableOnly
[   11.480161] 148 pages reserved
[   11.480340] Unreclaimable slab info:
[   11.480371] Name                      Used          Total
[   11.480410] bio-4                     34KB        108KB
[   11.480450] bio-3                      8KB         15KB
[   11.480487] bio-2                     11KB         26KB
[   11.480525] RAWv6                     15KB         15KB
[   11.480563] UDPv6                     29KB         47KB
[   11.480602] bio-1                      3KB          3KB
[   11.480639] RAW                        7KB          7KB
[   11.480841] UDP                        7KB          7KB
[   11.480881] request_queue             31KB         31KB
[   11.480919] biovec-max               672KB        672KB
[   11.480957] biovec-64                  8KB         32KB
[   11.480995] bio_integrity_payload         31KB         43KB
[   11.481033] skbuff_head_cache          3KB          3KB
[   11.481073] file_lock_cache            3KB          3KB
[   11.481266] file_lock_ctx              3KB          3KB
[   11.481303] proc_dir_entry          1559KB       1559KB
[   11.481341] pde_opener                 3KB          3KB
[   11.481379] seq_file                   3KB          3KB
[   11.481417] kernfs_node_cache       1087KB       1087KB
[   11.481455] mnt_cache                  7KB          7KB
[   11.481492] names_cache               32KB         32KB
[   11.481530] nsproxy                    6KB          7KB
[   11.481568] vm_region                  3KB          3KB
[   11.481765] vm_area_struct             3KB          3KB
[   11.481802] mm_struct                  7KB          7KB
[   11.481840] fs_cache                   6KB          8KB
[   11.481879] files_cache                7KB          7KB
[   11.481917] signal_cache              39KB         39KB
[   11.481954] sighand_cache             61KB         61KB
[   11.481991] task_struct               61KB         63KB
[   11.482029] cred_jar                   8KB          8KB
[   11.482223] pid                        7KB          7KB
[   11.482261] pool_workqueue           138KB        160KB
[   11.482298] kmalloc-8k               120KB        128KB
[   11.482336] kmalloc-4k               128KB        160KB
[   11.482373] kmalloc-2k               256KB        256KB
[   11.482411] kmalloc-1k               230KB        232KB
[   11.482448] kmalloc-512              368KB        368KB
[   11.482486] kmalloc-256              180KB        180KB
[   11.482525] kmalloc-192               31KB         31KB
[   11.482711] kmalloc-128               18KB         36KB
[   11.482749] kmalloc-96                19KB         19KB
[   11.482787] kmalloc-64                15KB         16KB
[   11.482824] kmalloc-32                 9KB         16KB
[   11.482862] kmalloc-16                 4KB          4KB
[   11.482900] kmalloc-8                 20KB         20KB
[   11.482938] kmem_cache_node            8KB          8KB
[   11.482976] kmem_cache                23KB         23KB
[   11.483165] Tasks state (memory values in pages):
[   11.483203] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[   11.483266] [     36]     0    36      162        0        0        0             0 host0
[   11.483319] Out of memory and no killable processes...
[   11.483358] Kernel panic - not syncing: System is deadlocked on memory
@letmaik letmaik added the area: samples Samples label May 1, 2020
@prp prp added area: disk Related to mounted disk images/encryption enhancement p1 Medium priority and removed area: samples Samples labels May 1, 2020
@prp prp added this to the Milestone 1 milestone May 1, 2020
@prp
Copy link
Member

prp commented May 1, 2020

I think that we should decide this based on the requirements of the cryptsetup replacement. @mikbras?

@letmaik
Copy link
Contributor Author

letmaik commented May 1, 2020

Isn't this independent of cryptsetup? The kernel-level dm-* modules would still be there.

@davidchisnall davidchisnall added the needs-triage Bug does not yet have a priority assigned label Jul 28, 2020
@SeanTAllen SeanTAllen removed the needs-triage Bug does not yet have a priority assigned label Jul 29, 2020
@mikbras
Copy link
Collaborator

mikbras commented Aug 18, 2020

Isn't this independent of cryptsetup? The kernel-level dm-* modules would still be there.

That is correct. The kernel formats integrity volumes and depletes memory when doing so (even with vicsetup).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mikbras mikbras

Labels
area: disk Related to mounted disk images/encryption enhancement p1 Medium priority
Projects
None yet
Milestone
Milestone 1
Development

No branches or pull requests
5 participants
@SeanTAllen @davidchisnall @letmaik @prp @mikbras