Skip to content

Make TCP interfaces compatible. #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

moharaka
Copy link

This is a proposed patch to get the TCP working. It simply redefines the "struct tx_add" to "extent_t":

typedef struct tx_add extent_t;

@xianliang66
Copy link
Contributor

It can surely silence the compiler complaint. But it may not work with multiple vCPU threads at one node. Do you boot GiantVM successfully?

@moharaka
Copy link
Author

No. Although I am not sure if I am booting the system correctly. I currently have only two nodes, with one VCPU per node.

  1. To run giantVM, I start qemu on the master
$ ../QEMU/build/x86_64-softmmu/qemu-system-x86_64 -smp 2 -m 2048 --enable-kvm -machine q35,kernel-irqchip=off -serial mon:stdio -nographic -net user -net nic,model=e1000 -kernel bzImage -initrd initrd.img-5.4.0-4-amd64 -drive if=none,file=debian86.img,format=raw,id=drive-sata0-0-0 -device ahci,id=ahci0 -device ide-drive,bus=ahci0.0,drive=drive-sata0-0-0,id=sata0-0-0 -local-cpu '1,start=0,iplist=140.77.13.166 140.77.13.116' -append 'root=/dev/sda rw earlyprintk console=ttyS0'
CPU Info
Total: 2
Local: 1 [0-0]
Remote: 1[ 1 ]
KVM API version[12], QEMU version[12]
CPU 1 is remote CPU, pause
start kvm dsm server, total memory size: 2147483648
run_on_cpu: not local CPU, ignore here. This could be a latent bug.
error: msi send vector in range 0-15
error: cannot find current apic
run_on_cpu: not local CPU, ignore here. This could be a latent bug.
run_on_cpu: not local CPU, ignore here. This could be a latent bug.
QEMU nums: 2, Total CPU nums: 2, CPU per QEMU: 1
connect_io_router...
connecting 140.77.13.116:40001
Connecting...
QEMU 0 listen on TCP socket 140.77.13.166:40000
  1. then on the remote machine:
$ ../QEMU/build/x86_64-softmmu/qemu-system-x86_64 -smp 2 -m 2048 --enable-kvm -serial mon:stdio -nographic -kernel bzImage -initrd initrd.img-5.4.0-4-amd64 -device ahci,id=ahci0 -drive if=none,file=debian86.img,format=raw,id=drive-sata0-0-0 -device ide-drive,bus=ahci0.0,drive=drive-sata0-0-0,id=sata0-0-0 -machine q35,kernel-irqchip=off -net user -net nic,model=e1000 -local-cpu '1,start=1,iplist=140.77.13.166 140.77.13.169' -append 'root=/dev/sda rw earlyprintk console=ttyS0'
CPU Info
Total: 2
Local: 1 [1-1]
Remote: 1[ 0 ]
KVM API version[12], QEMU version[12]
CPU 0 is remote CPU, pause
start kvm dsm server, total memory size: 2147483648
run_on_cpu: not local CPU, ignore here. This could be a latent bug.
error: msi send vector in range 0-15
error: cannot find current apic
run_on_cpu: not local CPU, ignore here. This could be a latent bug.
run_on_cpu: not local CPU, ignore here. This could be a latent bug.
QEMU nums: 2, Total CPU nums: 2, CPU per QEMU: 1
connect_io_router...
connecting 140.77.13.166:40000
Connecting...
QEMU 1 listen on TCP socket 140.77.13.169:40001
connecting io router done
connect_io_router done

As it can be seen, both machines connect. However, the Qemus do not make any further progress. Everything freezes.

Runing dmesg on another terminal, I get this output related to kvm. On the master machine:

$ dmesg | tail
...
[  242.223111] kvm_dsm_init: Enable kvm dsm mode, this kvm instance will be node-0
[  242.223114] kvm_dsm_init: kvm_dsm_init: kvm 0 use TCP connection

On the remote machine:

$ dmesg | tail
...
[  250.132399] kvm_dsm_init: Enable kvm dsm mode, this kvm instance will be node-1
[  250.132406] kvm_dsm_init: kvm_dsm_init: kvm 1 use TCP connection

@xianliang66
Copy link
Contributor

I don't know what's going wrong for your code. But you need to make sure the commands work well for vanilla QEMU-KVM (without -local-cpu). There's a variable kvm_dsm_debug_verbose in dsm.c which enables more log. And I can share you with my script:

#!/bin/sh

x86_64-softmmu/qemu-system-x86_64 --nographic -kernel \
$LINUX_BZIMAGE -hda $LINUX_IMAGE \
-append "root=/dev/sda1 console=ttyS0 clocksource=tsc" \
-cpu host -machine kernel-irqchip=off \
-smp 2 -m 4096 --enable-kvm -serial mon:stdio \
-numa node,cpus=0 -numa node,cpus=1 \
-local-cpu 1,start=1,iplist="$NODE0_IP $NODE1_IP" \
-monitor telnet:127.0.0.1:1235,server,nowait & > /dev/null
sleep 2
x86_64-softmmu/qemu-system-x86_64 --nographic -kernel \
$LINUX_BZIMAGE -hda $LINUX_IMAGE \
-append "root=/dev/sda1 console=ttyS0 clocksource=tsc" \
-cpu host -machine kernel-irqchip=off \
-smp 2 -m 4096 --enable-kvm -serial mon:stdio \
-numa node,cpus=0 -numa node,cpus=1 \
-local-cpu 1,start=0,iplist="$NODE0_IP $NODE1_IP" \
-monitor telnet:127.0.0.1:1236,server,nowait

@moharaka
Copy link
Author

Hi,
Are you running both QEMUs on the same machine?

@xianliang66
Copy link
Contributor

You can deploy GiantVM on the same machine with different port numbers (hard-coded) or different machines.

@moharaka
Copy link
Author

Can you show me how to hard-code different port numbers?
What are the values of $NODE0_IP and $NODE1_IP?

@moharaka
Copy link
Author

Okay. I think I figured it out. The ports are hard-coded in QEMU's code (ports 40000 and 40001). I have setup both ports to same ip address (127.0.0.1 == localhost) and run the two QEMUs. I am getting the following error:

io-router listen error, exit...

@xianliang66
Copy link
Contributor

What do you mean by 'setup both ports to the same ip addr'? The error message is generated after the failure of listen of Linux socket and it may be triggered by something like 'port has been used'. And two instances use 4 different ports totally for both QEMU and KVM connection.

@moharaka
Copy link
Author

Sorry for the bad phrasing... By that I mean I have set NODE0_IP and NODE1_IP to the same value:

NODE0_IP=127.0.0.1
NODE1_IP=127.0.0.1

Qemu code will then automatically assign diffrent ports for each qemu:
40000 for qemu 0
40001 for qemu 1

@xianliang66
Copy link
Contributor

I guess it's caused by busy ports. E.g., previous QEMU doesn't exit normally. netstat -na | grep 40000 could verify it. If not, then the return value of listen and errno can be checked.

@moharaka
Copy link
Author

moharaka commented Mar 18, 2020

Good news, I am able to boot GiantVM (with both Qemus on the same machine). The "io-router problem was due to a misconfiguration of my shell script.

However, now I am hitting a bug further in the execution. There are two kernel Oops produced by the kernel. The full log is as follow:

[   86.838019] kvm_dsm_init: Enable kvm dsm mode, this kvm instance will be node-1
[   86.838025] kvm_dsm_init: kvm_dsm_init: kvm 1 use TCP connection
[   98.875240] kvm_dsm_init: Enable kvm dsm mode, this kvm instance will be node-0
[   98.875242] kvm_dsm_init: kvm_dsm_init: kvm 0 use TCP connection
[   99.711351] kvm-dsm: node-1 established connection with node-0 [127.0.0.1:37710]
[   99.711352] kvm-dsm: node-0 accepted connection
[   99.711635] kvm-dsm: node-0 established connection with node-1 [127.0.0.1:37711]
[   99.711636] kvm-dsm: node-1 accepted connection
[  100.236388] kvm [1584]: vcpu0, guest rIP: 0xffffffff810b4396 unhandled rdmsr: 0x1c9
[  100.236391] kvm [1584]: vcpu0, guest rIP: 0xffffffff810b4396 unhandled rdmsr: 0x1a6
[  100.236392] kvm [1584]: vcpu0, guest rIP: 0xffffffff810b4396 unhandled rdmsr: 0x1a7
[  100.236394] kvm [1584]: vcpu0, guest rIP: 0xffffffff810b4396 unhandled rdmsr: 0x3f6
[  100.236395] kvm [1584]: vcpu0, guest rIP: 0xffffffff810b4396 unhandled rdmsr: 0x3f7
[  100.601616] kvm: zapping shadow pages for mmio generation wraparound
[  100.609131] kvm: zapping shadow pages for mmio generation wraparound
[  100.610300] kvm: zapping shadow pages for mmio generation wraparound
[  100.618629] kvm: zapping shadow pages for mmio generation wraparound
[  101.548280] kvm [1557]: vcpu1, guest rIP: 0xffffffff81f67bf6 unhandled rdmsr: 0x64e
[  101.548800] kvm [1557]: vcpu1, guest rIP: 0xffffffff81f67bf6 unhandled rdmsr: 0x34
[  107.980285] ------------[ cut here ]------------
[  107.980995] kernel BUG at arch/x86/kvm/dsm-util.c:214!
[  107.981706] invalid opcode: 0000 [#1] SMP
[  107.982423] Modules linked in: ccm arc4 iwlmvm joydev mac80211 mei_wdt hid_alps snd_soc_skl snd_soc_skl_ipc snd_hda_codec_hdmi snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_codec_realtek snd_hda_ext_core snd_hda_codec_generic dell_wmi snd_soc_sst_match dell_smbios snd_soc_core snd_compress snd_pcm_dmaengine ac97_bus dcdbas dell_smm_hwmon iwlwifi snd_hda_intel snd_hda_codec cfg80211 intel_rapl snd_hda_core x86_pkg_temp_thermal intel_powerclamp coretemp snd_hwdep snd_pcm snd_timer snd input_leds serio_raw soundcore idma64 virt_dma hci_uart uvcvideo mei_me btusb mei videobuf2_vmalloc btrtl videobuf2_memops videobuf2_v4l2 videobuf2_core btbcm videodev btqca btintel bluetooth media intel_pch_thermal intel_lpss_pci processor_thermal_device intel_soc_dts_iosf acpi_als kfifo_buf intel_hid int3400_thermal intel_lpss_acpi
[  107.985763]  acpi_pad intel_lpss dell_smo8800 int3403_thermal industrialio acpi_thermal_rel mac_hid int340x_thermal_zone sparse_keymap iscsi_tcp libiscsi_tcp autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear i2c_hid i915 crct10dif_pclmul drm_kms_helper crc32_pclmul syscopyarea sysfillrect sysimgblt ghash_clmulni_intel fb_sys_fops drm aesni_intel e1000e aes_x86_64 lrw glue_helper ablk_helper cryptd ahci libahci wmi hid pinctrl_sunrisepoint pinctrl_intel video fjes
[  107.990389] CPU: 1 PID: 1592 Comm: qemu-system-x86 Not tainted 4.9.76+ #5
[  107.991368] Hardware name: Dell Inc. Latitude 5280/08DMYJ, BIOS 1.9.3 03/08/2018
[  107.992346] task: ffff92fa7e1f9700 task.stack: ffffa2dc837e4000
[  107.993310] RIP: 0010:[<ffffffffb606794e>]  [<ffffffffb606794e>] dsm_decode_diff+0xbe/0xd0
[  107.994310] RSP: 0018:ffffa2dc837e7b50  EFLAGS: 00010286
[  107.995310] RAX: 00000000ffffffff RBX: ffff92fa66090000 RCX: 0000000000001000
[  107.996294] RDX: ffff92fa66096000 RSI: 0000000000000001 RDI: ffff92fa66090000
[  107.997297] RBP: ffffa2dc837e7b78 R08: ffffa2dc837e8000 R09: 0000000000001000
[  107.998292] R10: 0000000000001000 R11: 0000000000000030 R12: ffff92fa66096000
[  107.999305] R13: ffffa2dc85af31c0 R14: 0000000000078948 R15: 0000000000000001
[  108.000303] FS:  00007f92cfbff700(0000) GS:ffff92fab1480000(0000) knlGS:0000000000000000
[  108.001326] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  108.002316] CR2: 00007f5c3c3ca688 CR3: 00000007fdc84000 CR4: 0000000000362670
[  108.003333] Stack:
[  108.004354]  00000007f9348548 ffffa2dc85b09000 0000000000078948 ffffa2dc85af31c0
[  108.005379]  ffff92fa7dc98000 ffffa2dc837e7bf8 ffffffffb6069713 0000000100000000
[  108.006427]  000000017e1fa940 ffff92fa66090000 0000000000000000 0000000000000000
[  108.007480] Call Trace:
[  108.008508]  [<ffffffffb6069713>] ivy_kvm_dsm_page_fault+0x573/0x840
[  108.009572]  [<ffffffffb60645e1>] kvm_dsm_page_fault+0x71/0xd0
[  108.010633]  [<ffffffffb606639c>] kvm_dsm_memcpy+0x33c/0x5c0
[  108.011667]  [<ffffffffb6066c76>] kvm_vm_ioctl_dsm+0xc6/0x360
[  108.012721]  [<ffffffffb6035ef9>] kvm_arch_vm_ioctl+0x929/0xbf0
[  108.013767]  [<ffffffffb611b605>] ? update_load_avg+0x75/0x390
[  108.014806]  [<ffffffffb611b605>] ? update_load_avg+0x75/0x390
[  108.015827]  [<ffffffffb60219ea>] kvm_vm_ioctl+0x8a/0x7c0
[  108.016828]  [<ffffffffb60957c5>] ? __switch_to+0x2e5/0x700
[  108.017842]  [<ffffffffb62a2412>] do_vfs_ioctl+0x92/0x5b0
[  108.018830]  [<ffffffffb68f9e12>] ? __sys_recvmsg+0x62/0x80
[  108.019823]  [<ffffffffb62a29a9>] SyS_ioctl+0x79/0x90
[  108.020806]  [<ffffffffb6a3159e>] entry_SYSCALL_64_fastpath+0x1e/0xc9
[  108.021801] Code: 0f 00 00 48 29 fb 48 29 de 81 c3 00 10 00 00 c1 eb 03 89 d9 f3 48 a5 4c 89 e7 e8 3e f8 1f 00 5b 41 5c 41 5d 41 5e 41 5f 5d f3 c3 <0f> 0b 0f 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 
[  108.023980] RIP  [<ffffffffb606794e>] dsm_decode_diff+0xbe/0xd0
[  108.025003]  RSP <ffffa2dc837e7b50>
[  108.026039] ------------[ cut here ]------------
[  108.026053] ---[ end trace f86a59019fa4dfe3 ]---
[  108.027396] kernel BUG at arch/x86/kvm/ivy.c:184!
[  108.028007] invalid opcode: 0000 [#2] SMP
[  108.028614] Modules linked in: ccm arc4 iwlmvm joydev mac80211 mei_wdt hid_alps snd_soc_skl snd_soc_skl_ipc snd_hda_codec_hdmi snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_codec_realtek snd_hda_ext_core snd_hda_codec_generic dell_wmi snd_soc_sst_match dell_smbios snd_soc_core snd_compress snd_pcm_dmaengine ac97_bus dcdbas dell_smm_hwmon iwlwifi snd_hda_intel snd_hda_codec cfg80211 intel_rapl snd_hda_core x86_pkg_temp_thermal intel_powerclamp coretemp snd_hwdep snd_pcm snd_timer snd input_leds serio_raw soundcore idma64 virt_dma hci_uart uvcvideo mei_me btusb mei videobuf2_vmalloc btrtl videobuf2_memops videobuf2_v4l2 videobuf2_core btbcm videodev btqca btintel bluetooth media intel_pch_thermal intel_lpss_pci processor_thermal_device intel_soc_dts_iosf acpi_als kfifo_buf intel_hid int3400_thermal intel_lpss_acpi
[  108.031146]  acpi_pad intel_lpss dell_smo8800 int3403_thermal industrialio acpi_thermal_rel mac_hid int340x_thermal_zone sparse_keymap iscsi_tcp libiscsi_tcp autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear i2c_hid i915 crct10dif_pclmul drm_kms_helper crc32_pclmul syscopyarea sysfillrect sysimgblt ghash_clmulni_intel fb_sys_fops drm aesni_intel e1000e aes_x86_64 lrw glue_helper ablk_helper cryptd ahci libahci wmi hid pinctrl_sunrisepoint pinctrl_intel video fjes
[  108.034214] CPU: 3 PID: 1587 Comm: qemu-vcpu/0 Tainted: G      D         4.9.76+ #5
[  108.034829] Hardware name: Dell Inc. Latitude 5280/08DMYJ, BIOS 1.9.3 03/08/2018
[  108.035435] task: ffff92fa7e1f8000 task.stack: ffffa2dc86d88000
[  108.036046] RIP: 0010:[<ffffffffb606834d>]  [<ffffffffb606834d>] kvm_dsm_invalidate+0xdd/0x110
[  108.036658] RSP: 0018:ffffa2dc86d8b9e8  EFLAGS: 00010246
[  108.037260] RAX: 0000000000000103 RBX: ffff92fa7dc98000 RCX: ffffa2dcb7202000
[  108.037882] RDX: 0000000000000000 RSI: 0000000000000010 RDI: ffffa2dc86d8ba80
[  108.038484] RBP: ffffa2dc86d8ba50 R08: 0000000000000000 R09: 072e0127df01071f
[  108.039092] R10: ffffe9afdf980200 R11: 0000000000180000 R12: ffffa2dc85b09030
[  108.039693] R13: 0000000000000000 R14: 0000000000000002 R15: 00000007f93c9b5b
[  108.040290] FS:  00007f94ddf90700(0000) GS:ffff92fab1580000(0000) knlGS:0000000000000000
[  108.040902] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  108.041498] CR2: 000055bf2677c738 CR3: 00000007fdc84000 CR4: 0000000000362670
[  108.042108] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  108.042699] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  108.043284] Stack:
[  108.043872]  0000000000139f5b ffffa2dc86d8ba80 88ffffffb606814f 03117a012f01010e
[  108.044466]  08430001070139f5 0000000000000000 0000000000139f5b 4fc2040701030000
[  108.045072]  0f616c550404a7fe 0404f76002e10404 850402a40f498d98 c05ac60304a7f39b
[  108.045668] Call Trace:
[  108.046258]  [<ffffffffb60645e1>] ? kvm_dsm_page_fault+0x71/0xd0
[  108.046863]  [<ffffffffb6064ad6>] ? __kvm_dsm_acquire_page+0x186/0x1f0
[  108.047463]  [<ffffffffb606560c>] ? kvm_dsm_vcpu_acquire_page+0x4c/0x60
[  108.048064]  [<ffffffffb60488f7>] ? tdp_page_fault+0x177/0x350
[  108.048678]  [<ffffffffb603f3d4>] ? kvm_mmu_page_fault+0x64/0x110
[  108.049285]  [<ffffffffb606ed3f>] ? handle_ept_violation+0x8f/0x180
[  108.049892]  [<ffffffffb6075a2e>] ? vmx_handle_exit+0x1ae/0x13d0
[  108.050499]  [<ffffffffb6078fa0>] ? atomic_switch_perf_msrs+0x60/0x80
[  108.051116]  [<ffffffffb6079ab1>] ? vmx_vcpu_run+0x2d1/0x490
[  108.051722]  [<ffffffffb603a211>] ? kvm_arch_vcpu_ioctl_run+0x621/0x15f0
[  108.052332]  [<ffffffffb62a35a6>] ? pollwake+0x66/0x70
[  108.052924]  [<ffffffffb6114500>] ? wake_up_q+0x70/0x70
[  108.053488]  [<ffffffffb6034037>] ? kvm_arch_vcpu_load+0x57/0x270
[  108.054059]  [<ffffffffb601fc84>] ? kvm_vcpu_ioctl+0x2e4/0x560
[  108.054618]  [<ffffffffb628d268>] ? __vfs_write+0x28/0x140
[  108.055176]  [<ffffffffb62a2412>] ? do_vfs_ioctl+0x92/0x5b0
[  108.055746]  [<ffffffffb628e57a>] ? vfs_write+0x15a/0x1b0
[  108.056304]  [<ffffffffb62a29a9>] ? SyS_ioctl+0x79/0x90
[  108.056860]  [<ffffffffb6a3159e>] ? entry_SYSCALL_64_fastpath+0x1e/0xc9
[  108.057410] Code: c1 48 8d 0c c0 48 8d 04 48 49 8b 4c 24 28 48 8d 04 c1 8b 80 90 00 00 00 66 89 45 d2 74 a0 44 39 b3 1c 30 00 00 0f 87 75 ff ff ff <0f> 0b 31 c0 48 83 c4 40 5b 41 5c 41 5d 41 5e 41 5f 5d c3 4c 89 
[  108.058581] RIP  [<ffffffffb606834d>] kvm_dsm_invalidate+0xdd/0x110
[  108.059121]  RSP <ffffa2dc86d8b9e8>
[  108.059644] ---[ end trace f86a59019fa4dfe4 ]---

@xianliang66
Copy link
Contributor

It looks like the guest OS has executed quite a few code successfully. The assertion failure at arch/x86/kvm/ivy.c:184 indicates the incorrect copyset fetching from a server, which is probably caused by bugs of network transmission, according to my experiences. Specifically, the copyset is stored in tx_add which you have redefined. And, as you know, besides the normal vCPU threads, the I/O thread in QEMU which is used to simulate devices also invokes network operations. So maybe it's still necessary to port the TCP code to a multi-thread compatible version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants