Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support live migration for Hygon CSV1/2/3 guest, fix nesting #VC Exception #5

Merged
merged 1 commit into from
Oct 14, 2024

Conversation

wojiaohanliyang
Copy link

@wojiaohanliyang wojiaohanliyang commented Oct 14, 2024

The live migration for Hygon CSV1/2/3 guest depends on the KVM hypercall KVM_HC_MAP_GPA_RANGE, add
code to sync page enc/dec status to KVM. Then, KVM transfer these info to Qemu. During live migration, Qemu will check
whether request firmware APIs to assist migrate private pages of the guest.

In addition, OVMF will setup efi variable to express whether live migration is supported to guest's kernel. If live migration is
supported, the guest's kernel will invoke KVM_HC_MAP_GPA_RANGE to tell page enc/dec status changes when dynamic
change page enc/dec mappings.

The MMIO routine of VC handler will get memory encrypt status to validate MMIO address. MemEncryptSevGetEncryptionMask() will enable interrupt while interrupt must be disabled during VC. During DXE stage, VC routine as below:
CcExitHandleVc
-> MemEncryptSevGetAddressRangeState
-> MemEncryptSevGetEncryptionMask->PcdGet64(PcdPteMemoryEncryptionAddressOrMask)

[ hly: Fix the changelog of edk2 (2024.08-2deepin1). ]

0015-OvmfPkg-BaseMemEncryptLib-Detect-SEV-live-migration-.patch
- Use guest's cpuid to check live migration capability of this confidential guest.
0016-OvmfPkg-BaseMemEncryptLib-Hypercall-API-for-page-enc.patch
- Introduce interface to tell page enc/dec status to KVM.
0017-OvmfPkg-BaseMemEncryptLib-Invoke-page-encryption-sta.patch
- Invoke KVM_HC_MAP_GPA_RANGE when page enc/dec status are changed.
0018-OvmfPkg-VmgExitLib-Encryption-state-change-hypercall.patch
- Support KVM_HC_MAP_GPA_RANGE for CSV2/CSV3 guest.
0019-OvmfPkg-PlatformPei-Mark-SEC-GHCB-page-as-unencrypte.patch
- Tell the page enc/dec status of the SEC Ghcb page.
0020-OvmfPkg-AmdSevDxe-Add-support-for-SEV-live-migration.patch
- Setup efi variable SevLiveMigrationEnabled so that guest's kernel will check whether live migration is supported in the guest.
0021-OvmfPkg-BaseMemcryptSevLib-Correct-the-calculation-o.patch
- Fix the number of pages when tell the enc/dec status to KVM.
0022-OvmfPkg-BaseMemEncryptLib-Return-SUCCESS-if-not-supp.patch
- Prevent ASSERT() error if live migration for confidential guest is unsupported in Qemu.
0023-OvmfPkg-BaseMemEncryptLib-Save-memory-encrypt-status.patch
- Fix nesting #VC exception caused by MMIO access.

How to Test

Code requirements

Generated binaries

  • Linux Kernel: deb packages of the kernel, install these packages on both the host and guest, reboot the host with this new kernel.
  • Linux cmdline must contains: csv_mem_percentage=@xxx kvm-amd.sev=1 kvm-amd.sev_es=1. The @xxx is decimal number, for example, 50 means provide 50% of the total main memory for CSV3 guest at most.
  • The CPU must support CSV3 hardware feature.
  • Run command dmesg | grep 'CSV3 enabled', if we see CSV: CSV3 enabled (ASIDs ...) at the host side,it means we can launch and run CSV3 guest.
  • Qemu: assume the qemu bin is installed to /usr/bin/qemu-system-x86_64
  • Edk2: assuem the OVMF bin is installed to /usr/share/OVMF/OVMF_CODE_4M.fd

Live migrate CSV guest to target machine

Assume the ip of the source machine is HostA_IP, the ip of the target machine is HostB_IP.

  • Act on target machine: Get cert chain at target machine
$ sudo ./hag csv export_cert_chain
$cat hsk.cert hrk.cert > vendor_cert.bin
$cat pek.cert oca.cert cek.cert > plat_cert.bin
$cat pdh.cert > pdh.bin
$base64 -w 0 vendor_cert.bin > vendor_cert.base64
$base64 -w 0 plat_cert.bin > plat_cert.base64
$base64 -w 0 pdh.bin > pdh.base64
$sudo chmod 666 *.base64
$cp -a *.base64 /tmp
  • Act on source machine: Transfer target machine's cert chain to source machine
$ scp ${HostB_IP}:/tmp/*.base64 /tmp/
  • Act on target machine: Create CSV guest on target machine
$ /usr/bin/qemu-system-x86_64 \
-name csv-receive --enable-kvm -cpu host -m 6G -smp 40 \
-drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE_4M.fd,readonly=on \
-object sev-guest,id=sev0,policy=0x1,cbitpos=47,reduced-phys-bits=5 \
-machine memory-encryption=sev0 \
-device virtio-blk-pci,scsi=off,drive=drive0,disable-legacy=on,iommu_platform=on,bootindex=1 \
-drive file=csv_receive.qcow2,format=qcow2,if=none,id=drive0 \
-vnc 0.0.0.0:1 -qmp tcp:localhost:4445,server,nowait \
-incoming tcp:${HostB_IP}:6666 &
  • Act on source machine: Create CSV guest on source machine
$ /usr/bin/qemu-system-x86_64 \
-name csv-send --enable-kvm -cpu host -m 6G -smp 40 \
-drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE_4M.fd,readonly=on \
-object sev-guest,id=sev0,policy=0x1,cbitpos=47,reduced-phys-bits=5 \
-machine memory-encryption=sev0 \
-device virtio-blk-pci,scsi=off,drive=drive0,disable-legacy=on,iommu_platform=on,bootindex=1 \
-drive file=csv_send.qcow2,format=qcow2,if=none,id=drive0 \
-vnc 0.0.0.0:0 -qmp tcp:localhost:4444,server,nowait &
  • Act on source machine: Issue live migration on source machine
$ sudo socat - tcp:${HostA_IP}:4444
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 2, "major": 8}, "package": "Debian 1:8.2.0+ds-1deepin5"}, "capabilities": ["oob"]}}
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute":"migrate-set-parameters","arguments":{"sev-pdh":"/tmp/pdh.base64","sev-plat-cert":"/tmp/plat_cert.base64","sev-amd-cert":"/tmp/vendor_cert.base64"}}
{"return": {}}
{"execute":"migrate","arguments":{"uri":"${HostB_IP}:6666"}}
{"return": {}}
  • Wait a while, the guest on source machine will be stopped, and the guest on the target machine continues to run based on the state of guest on source machine.

The process to live migrate CSV2/CSV3 guest are the same as CSV guest. For CSV2 guest, the bit2 (start from bit 0) of the policy attr must be set; For CSV3 guest, the bit2 and bit6 (start from bit 0) of the policy attr must be set.

…ption

The live migration for Hygon CSV1/2/3 guest depends on the KVM
hypercall KVM_HC_MAP_GPA_RANGE, add code to sync page enc/dec
status to KVM.

The MMIO routine of VC handler will get memory encrypt status to
validate MMIO address. MemEncryptSevGetEncryptionMask() will enable
interrupt while interrupt must be disabled during VC. During DXE
stage, VC routine as below:
  CcExitHandleVc
    -> MemEncryptSevGetAddressRangeState
      -> MemEncryptSevGetEncryptionMask->PcdGet64(PcdPteMemoryEncryptionAddressOrMask)

[ hly: Fix the changelog of edk2 (2024.08-2deepin1). ]

Signed-off-by: hanliyang <[email protected]>
@deepin-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tsic404 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@deepin-ci-robot
Copy link
Contributor

Hi @wojiaohanliyang. Thanks for your PR.

I'm waiting for a deepin-community member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Copy link

TAG Bot

TAG: 2024.08-2deepin2
EXISTED: no
DISTRIBUTION: unstable

@Zeno-sole Zeno-sole merged commit e766181 into deepin-community:master Oct 14, 2024
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants