Skip to content

Add support for Arm64 including all changes in PR #1302 (@swine, @surajjs95, @t-msn, and others) #1439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

puranjaymohan
Copy link
Contributor

@puranjaymohan puranjaymohan commented Feb 7, 2025

This is a rebase of #1302 with some more changes and some already merged patches removed.
It is supposed to work with the sframe based reliable stack unwinder posted upstream:

[PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

NOTE:

  • The sframe sections are skipped for now in the livepatch module because the above series doesn't support modules yet. I am trying to make that work both from the kernel side and also in Kpatch and will add a patch later.

Building and testing a livepatch on a 6.12 arm64 kernel for the following basic patch:

[ec2-user@ip-172-31-32-86 ~]$ cat CVE-1234-5678.patch
From 9c372fa658da9ac55e3acb7c3e4f3035f9241c1d Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <[email protected]>
Date: Tue, 14 Jan 2025 09:40:34 +0000
Subject: [PATCH] fs: proc: cmdline: livepatch test

This patch will be turned into a livepatch.

Signed-off-by: Puranjay Mohan <[email protected]>
---
 fs/proc/cmdline.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/fs/proc/cmdline.c b/fs/proc/cmdline.c
index a6f76121955f..90bef8411dbf 100644
--- a/fs/proc/cmdline.c
+++ b/fs/proc/cmdline.c
@@ -7,8 +7,7 @@

 static int cmdline_proc_show(struct seq_file *m, void *v)
 {
-       seq_puts(m, saved_command_line);
-       seq_putc(m, '\n');
+       seq_printf(m, "%s\n", "this has been live patched");
        return 0;
 }

--
2.47.1
[ec2-user@ip-172-31-32-86 ~]$ uname -r
6.12.11-6.47.al2023pjy.aarch64

[ec2-user@ip-172-31-32-86 ~]$ ./kpatch/kpatch-build/kpatch-build -v /usr/lib/debug/lib/modules/6.12.11-6.47.al2023pjy.aarch64/vmlinux -r kernel-6.12.11-6.47.al2023pjy.src.rpm CVE-1234-5678.patch
Using cache at /home/ec2-user/.kpatch/src
Testing patch file(s)
Reading special section data
Building original source
Building patched source
Extracting new and modified ELF sections
cmdline.o: changed function: cmdline_proc_show
Patched objects: vmlinux
Building patch module: livepatch-CVE-1234-5678.ko
SUCCESS
[ec2-user@ip-172-31-32-86 ~]$

Load and Unload test

[root@ip-172-31-32-86 ec2-user]# kpatch load livepatch-CVE-1234-5678.ko
loading patch module: livepatch-CVE-1234-5678.ko
waiting (up to 15 seconds) for patch transition to complete...
transition complete (2 seconds)
[root@ip-172-31-32-86 ec2-user]# dmesg
[69888.542598] livepatch: enabling patch 'livepatch_CVE_1234_5678'
[69888.544388] livepatch: 'livepatch_CVE_1234_5678': starting patching transition
[69890.128882] livepatch: 'livepatch_CVE_1234_5678': patching complete
[root@ip-172-31-32-86 ec2-user]#

[root@ip-172-31-32-86 ec2-user]# cat /proc/cmdline
this has been live patched

[root@ip-172-31-32-86 ec2-user]# kpatch unload livepatch-CVE-1234-5678.ko
disabling patch module: livepatch_CVE_1234_5678
waiting (up to 15 seconds) for patch transition to complete...
transition complete (1 seconds)
unloading patch module: livepatch_CVE_1234_5678
[root@ip-172-31-32-86 ec2-user]# dmesg
[69888.542598] livepatch: enabling patch 'livepatch_CVE_1234_5678'
[69888.544388] livepatch: 'livepatch_CVE_1234_5678': starting patching transition
[69890.128882] livepatch: 'livepatch_CVE_1234_5678': patching complete
[69906.212505] livepatch: 'livepatch_CVE_1234_5678': starting unpatching transition
[69907.128716] livepatch: 'livepatch_CVE_1234_5678': unpatching complete

[root@ip-172-31-32-86 ec2-user]# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt1)/boot/vmlinuz-6.12.11-6.47.al2023pjy.aarch64 root=UUID=da8e1b79-5f9e-4471-89e7-958620dff70a ro console=tty0 console=ttyS0,115200n8 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0 selinux=1 security=selinux quiet numa_cma=1:64M

Unit tests pass with the objects at: https://github.com/mihails-strasuns/kpatch-unit-test-objs/tree/arm64

CC=gcc10 ARCHES=aarch64 make -j64 unit
make -C kpatch-build
make[1]: Entering directory '/local/home/pjy/code/Kpatch/kpatch-build'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/local/home/pjy/code/Kpatch/kpatch-build'
make -C test/unit
make[1]: Entering directory '/local/home/pjy/code/Kpatch/test/unit'
make -C objs/aarch64
make[2]: Entering directory '/local/home/pjy/code/Kpatch/test/unit/objs/aarch64'
BUILD bug-table-section
BUILD cmdline-string
BUILD data-new
BUILD data-read-mostly
BUILD fixup-section
BUILD gcc-isra
BUILD gcc-static-local-var-2
BUILD gcc-static-local-var-3
BUILD gcc-static-local-var-4
BUILD gcc-static-local-var-5
BUILD gcc-static-local-var-6
BUILD mapping
BUILD meminfo-init-FAIL
BUILD meminfo-string
BUILD new-function
BUILD parainstructions-section
BUILD smp-locks-section
BUILD special-static
BUILD syscall
BUILD tracepoints-section
TEST bug-table-section
TEST cmdline-string
TEST data-new
TEST fixup-section
TEST gcc-isra
TEST gcc-static-local-var-2
TEST gcc-static-local-var-3
TEST gcc-static-local-var-4
TEST gcc-static-local-var-5
TEST gcc-static-local-var-6
TEST meminfo-init
TEST meminfo-string
TEST new-function
TEST parainstructions-section
TEST smp-locks-section
BUILD ASSERT_RTNL-detect
TEST special-static
TEST mapping
TEST tracepoints-section
TEST syscall
TEST data-read-mostly
TEST ASSERT_RTNL-detect
make[2]: Leaving directory '/local/home/pjy/code/Kpatch/test/unit/objs/aarch64'
make[1]: Leaving directory '/local/home/pjy/code/Kpatch/test/unit'
Integration tests
[root@ip-172-31-32-86 kpatch]# KPATCH_BUILD_OPTS=-R make -j64 integration-slow
make -C kpatch-build
make -C kpatch
make -C kmod
make[1]: Entering directory '/home/ec2-user/kpatch/kpatch-build'
make[1]: Entering directory '/home/ec2-user/kpatch/kmod'
make[1]: Entering directory '/home/ec2-user/kpatch/kpatch'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/ec2-user/kpatch/kpatch'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/ec2-user/kpatch/kmod'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/ec2-user/kpatch/kpatch-build'
make -C test/integration slow
make[1]: Entering directory '/home/ec2-user/kpatch/test/integration'
rm -f *.ko *.log COMBINED.patch
./kpatch-test --kpatch-build-opts="-R" -d "amzn"-"2023"
build: data-new
build: gcc-static-local-var-6
build: macro-callbacks
build: module
build: new-function
build: new-globals
build: shadow-newpid
build: special-static
build: symvers-disagreement-FAIL
build: syscall
build: warn-detect-FAIL
combine: skipping amzn-2023/symvers-disagreement-FAIL.patch
combine: skipping amzn-2023/warn-detect-FAIL.patch
build: combined module
load test: data-new
load test: gcc-static-local-var-6 (no test prog)
load test: macro-callbacks (no test prog)
load test: module
load test: new-function (no test prog)
load test: new-globals (no test prog)
load test: shadow-newpid
load test: special-static (no test prog)
load test: syscall
load test: combined module
custom test: multiple
SUCCESS
make[1]: Leaving directory '/home/ec2-user/kpatch/test/integration'
[root@ip-172-31-32-86 kpatch]# echo $?
0

puranjaymohan and others added 7 commits February 7, 2025 10:33
The "has_function_profiling" support field in the symbol struct is used to show
that a function symbol is able to be patched. This is necessary to check that
functions which need to be patched are able to be.

On arm64 this means the presence of 2 NOP instructions at function entry
which are patched by ftrace to call the ftrace handling code. These 2 NOPs
are inserted by the compiler and the location of them is recorded in a
section called "__patchable_function_entries". Check whether a symbol has a
corresponding entry in the "__patchable_function_entries" section and if so
mark it as "has_func_profiling".

Signed-off-by: Suraj Jitindar Singh <[email protected]>
[Modify to use __patchable_function_entries support added upstream]
Signed-off-by: Puranjay Mohan <[email protected]>
…rch64

Add aarch64 support to kpatch_create_ftrace_callsite_sections(). Check
for the 2 required NOP instructions on function entry, which may be
preceded by a BTI C instruction depending on whether the
function is a leaf function. This determines the offset of the
patch site.

Signed-off-by: Pete Swain <[email protected]>
Signed-off-by: Puranjay Mohan <[email protected]>
Add the final support required for aarch64 and enable building on that arch.

Signed-off-by: Suraj Jitindar Singh <[email protected]>
[kpatch_line_macro_change_only() fixes]
Signed-off-by: Puranjay Mohan <[email protected]>
On arm64 when CONFIG_DEBUG_BUGVERBOSE is enabled debugging information about
bug sections is stored in the ".rodata.str" section of the object file. If this
isn't included then linking fails as labels in this section are referenced
by relocations in the bug table.

Include this section to enable building patches which contain this bug debug
information.

Signed-off-by: Suraj Jitindar Singh <[email protected]>
It seems mapping symbols in aarch64 elf has section size of 0.
So, exclude it in section symbol replacing code just like
kpatch_correlate_symbols().
This fixes the data-read-mostly unit test on aarch64.

Signed-off-by: Misono Tomohiro <[email protected]>
Signed-off-by: Puranjay Mohan <[email protected]>
Copy from kernel source tree.

Signed-off-by: Misono Tomohiro <[email protected]>
@puranjaymohan puranjaymohan changed the title Add support Arm64 including all changes in PR #1302 (@swine, @surajjs95, @t-msn, and others) Add support for Arm64 including all changes in PR #1302 (@swine, @surajjs95, @t-msn, and others) Feb 7, 2025
dylanbhatch and others added 2 commits February 7, 2025 12:09
For arm64 this option uses -fpatchable-function-entry=M,2, so 2 NOPs
are placed before the function entry point (in order to store a pointer
to ftrace_ops). When calculating function padding, check for the
presence of the two NOPs, and adjust the padding size by 8 if they are
found.

This was merged in the upstream kernel in v6.8 with:

      baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS")

With this into the equation, the entry of a function can look like one
of:

1. Without DYNAMIC_FTRACE_WITH_CALL_OPS and CONFIG_ARM64_BTI_KERNEL
-------------------------------------------------------------------

Disassembly of section .text.cmdline_proc_show:

0000000000000008 <cmdline_proc_show>:
   8:   d503201f        nop
   c:   d503201f        nop

2. Without DYNAMIC_FTRACE_WITH_CALL_OPS and with CONFIG_ARM64_BTI_KERNEL
------------------------------------------------------------------------

Disassembly of section .text.cmdline_proc_show:

0000000000000008 <cmdline_proc_show>:
   0:   d503245f        bti     c
   4:   d503201f        nop
   8:   d503201f        nop

3. With DYNAMIC_FTRACE_WITH_CALL_OPS and without CONFIG_ARM64_BTI_KERNEL
------------------------------------------------------------------------

Disassembly of section .text.cmdline_proc_show:

0000000000000000 <cmdline_proc_show-0x8>:
   0:   d503201f        nop
   4:   d503201f        nop

0000000000000008 <cmdline_proc_show>:
   8:   d503201f        nop
   c:   d503201f        nop

4. With DYNAMIC_FTRACE_WITH_CALL_OPS and with CONFIG_ARM64_BTI_KERNEL
---------------------------------------------------------------------

Disassembly of section .text.cmdline_proc_show:

0000000000000000 <cmdline_proc_show-0x8>:
   0:   d503201f        nop
   4:   d503201f        nop

0000000000000008 <cmdline_proc_show>:
   8:   d503245f        bti     c
   c:   d503201f        nop
  10:   d503201f        nop

make create-diff-object aware of DYNAMIC_FTRACE_WITH_CALL_OPS and its
quirks.

Signed-off-by: Dylan Hatch <[email protected]>
Signed-off-by: Puranjay Mohan <[email protected]>
The sframe sections can't be normally diffed and need to be created
separately. skip them for now.

Signed-off-by: Puranjay Mohan <[email protected]>
kpatch_replace_sections_syms() substitutes the object/function symbols
for the section symbol in the relocation sections. But relocations in
.rela.__patchable_function_entries can point to an instruction ouside
the function. This is because with CALL_OPS enabled, two NOPs are added
before the function entry. __patchable_function_entries needs the
address of the first NOP above the function.

We anyway generate __patchable_function_entries again in
kpatch_create_ftrace_callsite_sections() so we can skip mangling these
relocations

Signed-off-by: Puranjay Mohan <[email protected]>
for (i = 0; (void *)insn < insn_end && *insn == 0xd503201f; i++, insn++)
;

if (i == 2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we believe the i should be either 0 or 2, would it be better if we can report an error here when the i is some other value?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the function_padding_size is decideced by the nop padding at the start of the function.
The number of nop adding to the function head is decided by the N of building option
-fpatchable-function-entry.

Or maybe this function padding size can be calculated by this building option of the target kernel
build Makefile? And we can support more specific situation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the arm64 kernel build, there could be:

  1. no nops at the start
  2. 2 nops at the start
  3. 2 nops before the start and two after the start.

There could also be a BTI instruction at the entry of the function before the two nops, if they exist.

@cw-tsato
Copy link

cw-tsato commented Apr 6, 2025

Hi, Puranjay

I am testing this pull request using two types of kernels: one [1] that uses the sframe unwinder for livepatch, and one [2] that does not.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/

The test environment is as follows:

  • Kernel 6.13
  • GCC 13.3.0
  • Integration test patches were rebased from linux-6.2.0 using the rebase-patches script. module.patch and symvers-disagreement-FAIL.patch failed to rebase and were manually fixed.

In both kernels, the "module-LOADED.test" fails during integration testing. The results of reproducing the issue on the command line are as follows:

# insmod ./test-module.ko
test_module: loading out-of-tree module taints kernel.
test_module: tainting kernel with TAINT_LIVEPATCH
livepatch: enabling patch 'test_module'
livepatch: 'test_module': starting patching transition
livepatch: 'test_module': patching complete
# modprobe xfs
Unable to handle kernel write to read-only memory at virtual address ffff8000792802b8
Mem abort info:
  ESR = 0x000000009600004f
  EC = 0x25: DABT (current EL), IL = 32 bits
  SET = 0, FnV = 0
  EA = 0, S1PTW = 0
  FSC = 0x0f: level 3 permission fault
Data abort info:
  ISV = 0, ISS = 0x0000004f, ISS2 = 0x00000000
  CM = 0, WnR = 1, TnD = 0, TagAccess = 0
  GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000040ec4000
[ffff8000792802b8] pgd=0000000000000000, p4d=1000000041467003, pud=1000000042846403, pmd=1000000042847403, pte=0040000042845783
Internal error: Oops: 000000009600004f [#1] SMP
Modules linked in: xfs(+) libcrc32c test_module(OK)
CPU: 0 UID: 0 PID: 114 Comm: modprobe Tainted: G           O  K    6.13.0 #1
Tainted: [O]=OOT_MODULE, [K]=LIVEPATCH
Hardware name: linux,dummy-virt (DT)
pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : apply_relocate_add+0x800/0x8a8
lr : apply_relocate_add+0x7fc/0x8a8
sp : ffff8000814ab950
x29: ffff8000814ab950 x28: ffff0000028cb3c0 x27: 0000000000000018
x26: ffff800079285908 x25: ffff0000028cb480 x24: 0000000000000000
x23: ffff800079285908 x22: 0000000000000018 x21: ffff8000792802a0
x20: 00000000001b6000 x19: ffff800079436590 x18: ffff8000814ab9e0
x17: ffff8000794cb000 x16: ffffffffffffffff x15: 0000000000000008
x14: 0000000000000001 x13: 0000000000000048 x12: 0000000000000003
x11: 000000000000ffff x10: fffffffff7ffd170 x9 : ffff80008002dd5c
x8 : ffff800079280000 x7 : 7000782400642401 x6 : 80f8e6f3dfe6f3df
x5 : ffff0000028ca2c0 x4 : ffff800079282600 x3 : 0000000000000000
x2 : 0000000040000000 x1 : 0000000090000002 x0 : 00000000d0000da2
Call trace:
 apply_relocate_add+0x800/0x8a8 (P)
 klp_write_section_relocs+0xe8/0x170
 klp_write_object_relocs+0x78/0xd0
 klp_init_object_loaded+0x2c/0x138
 klp_module_coming+0x98/0x2a8
 load_module+0x1bb4/0x1f30
 init_module_from_file+0x90/0xe0
 __arm64_sys_finit_module+0x214/0x418
 invoke_syscall+0x5c/0x130
 el0_svc_common.constprop.0+0x48/0xf0
 do_el0_svc+0x24/0x38
 el0_svc+0x30/0xd0
 el0t_64_sync_handler+0x10c/0x138
 el0t_64_sync+0x198/0x1a0
Code: cb080294 d34c8282 942597d8 9360fe83 (b8366aa0)
---[ end trace 0000000000000000 ]---
Segmentation fault
# cat /proc/modules
xfs 2326528 1 - Loading 0xffff800079291000
libcrc32c 12288 1 xfs, Live 0xffff800079287000
test_module 16384 1 - Live 0xffff800079280000 (OK)

It seems likely that the cause of this issue is the same as the one mentioned in the previous patch submission: https://lore.kernel.org/linux-arm-kernel/[email protected]/
After adapting that patch for 6.13 and applying it to the kernel, module-LOADED.test now passes.

It appears that module-LOADED.test is passing in #1439 (comment).
If a fix has been implemented in the kernel, could you please describe it?


Adding more information:

It seems likely that the cause of this issue is the same as the one mentioned in the previous patch submission: https://lore.kernel.org/linux-arm-kernel/[email protected]/
After adapting that patch for 6.13 and applying it to the kernel, module-LOADED.test now passes.

Applying the update patch for 6.13 available at https://lore.kernel.org/all/[email protected]/ now causes module-LOADED.test to PASS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants