adding a driver

[venoom27]

Is there anyway to install the broadcom-wl driver on the wifibox?

[pgj]

Yes, the broadcom-wl driver can be installed as part of the Wifibox/Alpine image. There are two caveats though: it is not part of the default packages because the driver and the firmware require the user to accept Broadcom's licensing terms, and due to lack of hardware, it has not yet been tested fully.

That is, you should install the net/wifibox-alpine port yourself by selecting the respective port options. Also, it is not guaranteed that the driver will work. But I am still looking for testers who could help me with troubleshooting the related issues or confirm that the driver works.

Please refer to the net/wifibox-alpine port for the details.

[venoom27]

Also when the wifibox boots up it post

• Could not find a wireless interface
• /var/run/wpa_supplicant: creating directory
• Starting WPA Supplicant ... [ ok ]
• Starting networking ... * lo ... [ ok ]
• eth0 ... [ ok ]
• wlan0 ...ip: SIOCGIFFLAGS: No such device
  udhcpc: SIOCGIFINDEX: No such device
  ifup: failed to change interface wlan0 state to 'up'
  [ !! ]

Is the device not getting passed on? My bhyve.conf is set as

passthru=2/0/0

and my

pciconf -lv pci0:2:0:0
ppt0@pci0:2:0:0: class=0x028000 rev=0x02 hdr=0x00 vendor=0x14e4 device=0x4331 subvendor=0x106b subdevice=0x00f5
vendor = 'Broadcom Inc. and subsidiaries'
device = 'BCM4331 802.11a/b/g/n'
class = network

[pgj]

If the pciconf(8) output shows ppt0 for the device in question, the pass-through should be working. I would recommend to explore the dmesg file on the guest, which may give more reasons for the unavailability of the device there. The No such device messages on the networking interface configuration indicate that the device driver could not be loaded or initialized.

Here is an example log (excerpt) for the successful initialization of iwlwifi:

[    1.173043] Run /sbin/init as init process
[    1.174056]   with arguments:
[    1.174057]     /sbin/init
[    1.174058]   with environment:
[    1.174058]     HOME=/
[    1.174059]     TERM=linux
[    1.174059]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    1.174060]     modules=squashfs
[    1.491797] Intel(R) Wireless WiFi driver for Linux
[    1.492495] iwlwifi 0000:00:06.0: can't derive routing for PCI INT A
[    1.492497] iwlwifi 0000:00:06.0: PCI INT A: not connected
[    1.627095] iwlwifi 0000:00:06.0: loaded firmware version 36.ca7b901d.0 8000C-36.ucode op_mode iwlmvm
[    1.658126] iwlwifi 0000:00:06.0: Detected Intel(R) Dual Band Wireless AC 8260, REV=0x204

According to the Arch Linux wiki, it would be worth to try the b43 driver first for BCM4331. There is a separate configuration option for the net/wifibox-alpine port about that. The b43 and the b43legacy drivers are also not packaged automatically but they are present as additional options. Anyhow, the dmesg output for wl could be useful too, please share it.

[venoom27]

I am dual booting with Arch and the broadcom-wl driver is what works with the device in question . https://wiki.debian.org/wl I will pull the dmesg latter when I am at home and post.

[venoom27]

/var/run/wifibox/appliance/log/dmesg reads:

``` [    0.772812] Run /sbin/init as init process
[    0.773786]   with arguments:
[    0.773787]     /sbin/init
[    0.773788]   with environment:
[    0.773788]     HOME=/
[    0.773789]     TERM=linux
[    0.773789]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.773790]     modules=squashfs
[    1.337361] wl: loading out-of-tree module taints kernel.
[    1.337368] wl: module license 'MIXED/Proprietary' taints kernel.
[    1.337369] Disabling lock debugging due to kernel taint
[    1.341319] wl 0000:00:06.0: can't derive routing for PCI INT A
[    1.341329] wl 0000:00:06.0: PCI INT A: not connected
[    1.485170] ERROR @wl_cfg80211_detach : 
[    1.485172] NULL ndev->ieee80211ptr, unable to deref wl

[pgj]

I have noticed that Arch Linux has newer patches for the broadcom-wl driver so I imported them. Please try the dev branch of freebsd-wifibox-port, which could be found here: https://github.com/pgj/freebsd-wifibox-port/tree/dev .

[venoom27]

I deinstalled the older version and downloaded the dev version at https://github.com/pgj/freebsd-wifibox-port/tree/dev and installed it. After rebooting /var/run/wifibox/appliance/log/dmesg reads:

[    0.773290] Run /sbin/init as init process
[    0.774210]   with arguments:
[    0.774211]     /sbin/init
[    0.774211]   with environment:
[    0.774212]     HOME=/
[    0.774213]     TERM=linux
[    0.774213]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.774214]     modules=squashfs
[    1.381761] wl: loading out-of-tree module taints kernel.
[    1.381768] wl: module license 'MIXED/Proprietary' taints kernel.
[    1.381769] Disabling lock debugging due to kernel taint
[    1.385691] wl 0000:00:06.0: can't derive routing for PCI INT A
[    1.385696] wl 0000:00:06.0: PCI INT A: not connected
[    1.533679] ERROR @wl_cfg80211_detach :
[    1.533681] NULL ndev->ieee80211ptr, unable to deref wl
[    1.533685] ------------[ cut here ]------------
[    1.533686] name 'brcm_monitor0'
[    1.533690] WARNING: CPU: 0 PID: 446 at fs/proc/generic.c:712 remove_proc_entry+0xfb/0x150
[    1.533696] Modules linked in: wl(PO+)
[    1.533699] CPU: 0 PID: 446 Comm: modprobe Tainted: P           O      5.15.104-0-lts #1-Alpine
[    1.533702] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[    1.533704] RIP: 0010:remove_proc_entry+0xfb/0x150
[    1.533706] Code: ff e9 5e ff ff ff 48 8b b0 98 00 00 00 48 c7 c7 78 21 31 b3 e8 71 7e 3e 00 0f 0b 48 c7 c7 b7 20 31 b3 48 89 de e8 60 7e 3e 00 <0f> 0b >
[    1.533708] RSP: 0018:ffffb3cf80103b30 EFLAGS: 00010286
[    1.533710] RAX: 0000000000000000 RBX: ffffb3cf80103b68 RCX: 00000000ffffdfff
[    1.533712] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000009ffb
[    1.533713] RBP: 0000000000000000 R08: 00000000ffffdfff R09: 00000000ffffdfff
[    1.533714] R10: ffffffffb34536c0 R11: ffffffffb34536c0 R12: ffffffffb353b080
[    1.533715] R13: ffff9b00c1953000 R14: ffff9b00c1952c00 R15: 0000000000004331
[    1.533717] FS:  00007f81c069cb48(0000) GS:ffffffffb344b000(0000) knlGS:0000000000000000
[    1.533718] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.533720] CR2: 00007f81bfce3ffc CR3: 000000000194d005 CR4: 00000000001706f0
[    1.533721] Call Trace:
[    1.533723]  <TASK>
[    1.533725]  wl_free+0xb9/0x280 [wl]
[    1.533796]  ? wl_pci_probe+0x293/0xf80 [wl]
[    1.533832]  wl_pci_probe+0x5ff/0xf80 [wl]
[    1.533865]  pci_device_probe+0x8d/0xf0
[    1.533869]  really_probe.part.0+0x94/0x270
[    1.533873]  __driver_probe_device+0x89/0x110
[    1.533876]  driver_probe_device+0x19/0xd0
[    1.533879]  __driver_attach+0x78/0x160
[    1.533882]  ? __device_attach_driver+0x100/0x100
[    1.533884]  bus_for_each_dev+0x6a/0xb0
[    1.533887]  bus_add_driver+0x114/0x1d0
[    1.533890]  driver_register+0x84/0xd0
[    1.533892]  ? 0xffffffffc08f3000
[    1.533894]  do_one_initcall+0x45/0x180
[    1.533896]  ? kmem_cache_alloc+0x2c/0x120
[    1.533898]  do_init_module+0x40/0x210
[    1.533902]  __do_sys_init_module+0x14f/0x180
[    1.533904]  do_syscall_64+0x34/0x80
[    1.533910]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[    1.533912] RIP: 0033:0x7f81c063b704
[    1.533914] Code: 0e 4c 8b 44 24 10 4d 8d 48 08 4c 89 4c 24 10 44 8b 4c 24 08 4d 8b 00 4c 01 c9 41 83 f9 2f 76 05 48 8b 4c 24 10 4c 8b 09 0f 05 <48> 89 >
[    1.533916] RSP: 002b:00007ffdee275920 EFLAGS: 00000212 ORIG_RAX: 00000000000000af
[    1.533918] RAX: ffffffffffffffda RBX: 00007f81c069cb7c RCX: 00007f81c063b704
[    1.533919] RDX: 0000564c85b62da6 RSI: 0000000000731d90 RDI: 00007f81bf5b1560
[    1.533920] RBP: 0000564c85b62da6 R08: 0000000000000004 R09: 0000000000731d90
[    1.533921] R10: 0000000000000000 R11: 0000000000000212 R12: 00007f81bf5b1560
[    1.533922] R13: 0000000000000000 R14: 0000000000000001 R15: 0000564c85b639e4
[    1.533924]  </TASK>
[    1.533925] ---[ end trace 2081ca3c84a0fcaa ]---

[pgj]

That is not so good... How much memory did you configure for the guest in bhyve.conf? Is this at the default limit of 128 MB? I have tried to build a version of the driver that includes more debug information but I could not make it work so easily.

[pgj]

Could you please get me some more information about the driver itself from Arch Linux where it works? What is the kernel version? What does modinfo wl say? How does the relevant output of dmesg look like in that case?

[venoom27]

Arch dmesg

[   20.616193] wlan0: Broadcom BCM4331 802.11 Hybrid Wireless Controller 6.30.223.271 (r587334)

[   20.634615] wl 0000:02:00.0 wlp2s0: renamed from wlan0

modinfo wl

filename:       /lib/modules/6.4.2-arch1-1/extramodules/wl.ko.xz
license:        MIXED/Proprietary
srcversion:     39B1E3189AEBF0E4D013DA2
alias:          pci:v*d*sv*sd*bc02sc80i*
depends:        cfg80211
retpoline:      Y
name:           wl
vermagic:       6.4.2-arch1-1 SMP preempt mod_unload 
parm:           passivemode:int
parm:           wl_txq_thresh:int
parm:           oneonly:int
parm:           piomode:int
parm:           instance_base:int
parm:           nompc:int
parm:           intf_name:string

uname -r

6.4.2-arch1-1

bhyve.conf is set to the default setting.

[pgj]

Great, thank you! Please give me some time to think about what to do next.

[pgj]

I have found a repository on GitHub that tries to collect all the patches for broadcom-wl. Now I created another version of the wifibox-alpine port to use the binary package built using that. Please use the 20230714 version from the dev branch and test it again.

[venoom27]

I downloaded and installed the 20230714 dev version.
dmesg

Run /sbin/init as init process
[    0.775547]   with arguments:
[    0.775549]     /sbin/init
[    0.775551]   with environment:
[    0.775551]     HOME=/
[    0.775552]     TERM=linux
[    0.775553]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.775554]     modules=squashfs
[    2.050333] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    2.082977] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[    2.089214] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

So it didn't load at all.

[pgj]

That is strange. Could you please do a wifibox console and check the actual status of the module? After login, you can use the standard Linux commands such as lsmod and modprobe for investigation. I have tried it myself and I was able to see that module was loaded to the memory.

[venoom27]

OK the issue was on my end on not loading the wl module. My bhyve.conf got switch back to the wrong pci device during a rollback. Here is the dmesg with the correct pci device in the bhyve.conf on the 714 branch.

[    0.769136] Run /sbin/init as init process
[    0.770004]   with arguments:
[    0.770005]     /sbin/init
[    0.770006]   with environment:
[    0.770006]     HOME=/
[    0.770007]     TERM=linux
[    0.770007]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.770008]     modules=squashfs
[    1.858719] wl: loading out-of-tree module taints kernel.
[    1.858725] wl: module license 'MIXED/Proprietary' taints kernel.
[    1.858727] Disabling lock debugging due to kernel taint
[    1.862656] wl 0000:00:06.0: can't derive routing for PCI INT A
[    1.862661] wl 0000:00:06.0: PCI INT A: not connected
[    2.006325] ERROR @wl_cfg80211_detach : 
[    2.006327] NULL ndev->ieee80211ptr, unable to deref wl
[    2.006331] ------------[ cut here ]------------
[    2.006332] name 'brcm_monitor0'
[    2.006336] WARNING: CPU: 0 PID: 446 at fs/proc/generic.c:712 remove_proc_entry+0xfb/0x150
[    2.006342] Modules linked in: wl(PO+)
[    2.006345] CPU: 0 PID: 446 Comm: modprobe Tainted: P           O      5.15.104-0-lts #1-Alpine
[    2.006348] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[    2.006349] RIP: 0010:remove_proc_entry+0xfb/0x150
[    2.006352] Code: ff e9 5e ff ff ff 48 8b b0 98 00 00 00 48 c7 c7 78 21 f1 b8 e8 71 7e 3e 00 0f 0b 48 c7 c7 b7 20 f1 b8 48 89 de e8 60 7e 3e 00 <0f> 0b e9 33 ff ff ff 48 89 df e8 76 fb 0b 00 48 8b 8d 98 00 00 00
[    2.006354] RSP: 0018:ffffb0a980103b30 EFLAGS: 00010286
[    2.006356] RAX: 0000000000000000 RBX: ffffb0a980103b68 RCX: 00000000ffffdfff
[    2.006357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000009ffb
[    2.006358] RBP: 0000000000000000 R08: 00000000ffffdfff R09: 00000000ffffdfff
[    2.006360] R10: ffffffffb90536c0 R11: ffffffffb90536c0 R12: ffffffffb913b080
[    2.006361] R13: ffff9100c194f000 R14: ffff9100c194ec00 R15: 0000000000004331
[    2.006362] FS:  00007f78407dfb48(0000) GS:ffffffffb904b000(0000) knlGS:0000000000000000
[    2.006364] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    2.006365] CR2: 00007f783fe26ffc CR3: 0000000001971005 CR4: 00000000001706f0
[    2.006367] Call Trace:
[    2.006369]  <TASK>
[    2.006370]  wl_free+0xb9/0x280 [wl]
[    2.006437]  ? wl_pci_probe+0x293/0xf80 [wl]
[    2.006472]  wl_pci_probe+0x5ff/0xf80 [wl]
[    2.006511]  pci_device_probe+0x8d/0xf0
[    2.006516]  really_probe.part.0+0x94/0x270
[    2.006521]  __driver_probe_device+0x89/0x110
[    2.006524]  driver_probe_device+0x19/0xd0
[    2.006527]  __driver_attach+0x78/0x160
[    2.006529]  ? __device_attach_driver+0x100/0x100
[    2.006532]  bus_for_each_dev+0x6a/0xb0
[    2.006535]  bus_add_driver+0x114/0x1d0
[    2.006538]  driver_register+0x84/0xd0
[    2.006541]  ? 0xffffffffc0a01000
[    2.006542]  do_one_initcall+0x45/0x180
[    2.006545]  ? kmem_cache_alloc+0x2c/0x120
[    2.006547]  do_init_module+0x40/0x210
[    2.006551]  __do_sys_init_module+0x14f/0x180
[    2.006554]  do_syscall_64+0x34/0x80
[    2.006559]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[    2.006562] RIP: 0033:0x7f784077e704
[    2.006564] Code: 0e 4c 8b 44 24 10 4d 8d 48 08 4c 89 4c 24 10 44 8b 4c 24 08 4d 8b 00 4c 01 c9 41 83 f9 2f 76 05 48 8b 4c 24 10 4c 8b 09 0f 05 <48> 89 c7 e8 56 66 fe ff 48 83 c4 58 c3 56 31 d2 be 02 00 08 00 bf
[    2.006565] RSP: 002b:00007ffff5ea5930 EFLAGS: 00000212 ORIG_RAX: 00000000000000af
[    2.006567] RAX: ffffffffffffffda RBX: 00007f78407dfb7c RCX: 00007f784077e704
[    2.006569] RDX: 000055f15430bda6 RSI: 0000000000731e48 RDI: 00007f783f6f4550
[    2.006570] RBP: 000055f15430bda6 R08: 0000000000000004 R09: 0000000000731e48
[    2.006571] R10: 0000000000000000 R11: 0000000000000212 R12: 00007f783f6f4550
[    2.006572] R13: 0000000000000000 R14: 0000000000000001 R15: 000055f15430c9e4
[    2.006573]  </TASK>
[    2.006574] ---[ end trace 2d7d4523a1e62cc9 ]---
[    2.779549] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    2.804165] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[    2.813648] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    4.861781] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    6.872702] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    8.878697] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[   10.883441] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[   12.891425] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)

[pgj]

I have pushed a new version 20230715 that prints some debugging information on loading the module and probing the device. Maybe with the help of that, we could get a better understanding of what is happening. Here is an example with an Intel Wireless NIC, which is not supported and that is now visible in the logs:

[    1.380732] wl: loading out-of-tree module taints kernel.
[    1.380736] wl: module license 'MIXED/Proprietary' taints kernel.
[    1.380737] Disabling lock debugging due to kernel taint
[    1.384811] wl_pci_probe: bus 0 slot 6 func 0 irq 255
[    1.384816] wl_pci_probe: unsupported vendor 8086 device 24f3

[venoom27]

Here is the dmesg info from 20230715

[    0.771657] Run /sbin/init as init process
[    0.772580]   with arguments:
[    0.772581]     /sbin/init
[    0.772582]   with environment:
[    0.772582]     HOME=/
[    0.772583]     TERM=linux
[    0.772583]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.772585]     modules=squashfs
[    2.904731] wl: loading out-of-tree module taints kernel.
[    2.904739] wl: module license 'MIXED/Proprietary' taints kernel.
[    2.904740] Disabling lock debugging due to kernel taint
[    2.908552] wl_pci_probe: bus 0 slot 6 func 0 irq 255
[    2.908672] wl 0000:00:06.0: can't derive routing for PCI INT A
[    2.908675] wl 0000:00:06.0: PCI INT A: not connected
[    2.908768] wl_alloc_linux_if
[    2.908772] wl0: Bus: 
[    2.908773] PCI/DMA
[    3.039705] wl0: online cpus 1
[    3.039708] wl0: request_irq() failed
[    3.039709] wl: wl_free
[    3.039709] wl_free_if
[    3.039711] ERROR @wl_cfg80211_detach : 
[    3.039712] NULL ndev->ieee80211ptr, unable to deref wl
[    3.039716] ------------[ cut here ]------------
[    3.039717] name 'brcm_monitor0'
[    3.039722] WARNING: CPU: 0 PID: 446 at fs/proc/generic.c:712 remove_proc_entry+0xfb/0x150
[    3.039728] Modules linked in: wl(PO+)
[    3.039731] CPU: 0 PID: 446 Comm: modprobe Tainted: P           O      5.15.104-0-lts #1-Alpine
[    3.039734] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[    3.039735] RIP: 0010:remove_proc_entry+0xfb/0x150
[    3.039738] Code: ff e9 5e ff ff ff 48 8b b0 98 00 00 00 48 c7 c7 78 21 d1 9b e8 71 7e 3e 00 0f 0b 48 c7 c7
 b7 20 d1 9b 48 89 de e8 60 7e 3e 00 <0f> 0b e9 33 ff ff ff 48 89 df e8 76 fb 0b 00 48 8b 8d 98 00 00 00
[    3.039739] RSP: 0018:ffffa731400fbb30 EFLAGS: 00010286
[    3.039741] RAX: 0000000000000000 RBX: ffffa731400fbb68 RCX: 00000000ffffdfff
[    3.039743] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000009ffb
[    3.039744] RBP: 0000000000000000 R08: 00000000ffffdfff R09: 00000000ffffdfff
[    3.039745] R10: ffffffff9be536c0 R11: ffffffff9be536c0 R12: ffffffff9bf3b080
[    3.039746] R13: ffffa0c4c1932000 R14: ffffa0c4c13449c0 R15: ffffa0c4c194f000
[    3.039748] FS:  00007ff7bb71cb48(0000) GS:ffffffff9be4b000(0000) knlGS:0000000000000000
[    3.039749] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.039751] CR2: 00007ff7bb5a4000 CR3: 0000000001986005 CR4: 00000000001706f0
[    3.039752] Call Trace:
[    3.039754]  <TASK>
[    3.039755]  wl_free+0xf3/0x21d [wl]
[    3.039825]  ? _printk+0x53/0x6a
[    3.039830]  wl_pci_probe+0x850/0xf80 [wl]
[    3.039865]  pci_device_probe+0x8d/0xf0
[    3.039865]  pci_device_probe+0x8d/0xf0
[    3.039869]  really_probe.part.0+0x94/0x270
[    3.039874]  __driver_probe_device+0x89/0x110
[    3.039876]  driver_probe_device+0x19/0xd0
[    3.039879]  __driver_attach+0x78/0x160
[    3.039882]  ? __device_attach_driver+0x100/0x100
[    3.039885]  bus_for_each_dev+0x6a/0xb0
[    3.039887]  bus_add_driver+0x114/0x1d0
[    3.039890]  driver_register+0x84/0xd0
[    3.039893]  ? 0xffffffffc0776000
[    3.039894]  do_one_initcall+0x45/0x180
[    3.039897]  ? kmem_cache_alloc+0x2c/0x120
[    3.039899]  do_init_module+0x40/0x210
[    3.039903]  __do_sys_init_module+0x14f/0x180
[    3.039906]  do_syscall_64+0x34/0x80
[    3.039910]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[    3.039913] RIP: 0033:0x7ff7bb6bb704
[    3.039915] Code: 0e 4c 8b 44 24 10 4d 8d 48 08 4c 89 4c 24 10 44 8b 4c 24 08 4d 8b 00 4c 01 c9 41 83 f9 2f
 76 05 48 8b 4c 24 10 4c 8b 09 0f 05 <48> 89 c7 e8 56 66 fe ff 48 83 c4 58 c3 56 31 d2 be 02 00 08 00 bf
[    3.039916] RSP: 002b:00007fffbb18bcd0 EFLAGS: 00000212 ORIG_RAX: 00000000000000af
[    3.039918] RAX: ffffffffffffffda RBX: 00007ff7bb71cb7c RCX: 00007ff7bb6bb704
[    3.039920] RDX: 0000562da5bf3da6 RSI: 0000000000734960 RDI: 00007ff7ba631550
[    3.039921] RBP: 0000562da5bf3da6 R08: 0000000000000004 R09: 0000000000734960
[    3.039922] R10: 0000000000000000 R11: 0000000000000212 R12: 00007ff7ba631550
[    3.039923] R13: 0000000000000000 R14: 0000000000000001 R15: 0000562da5bf49e4
[    3.039924]  </TASK>
[    3.039925] ---[ end trace dbb3047612a9919f ]---
[    3.502454] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    3.568051] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[    3.577028] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    5.583696] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    7.602793] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    9.608621] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[   11.615200] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[   13.621333] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)

[pgj]

Awesome, thank you!

[pgj]

Could you please do a cat /proc/interrupts on the guest (using wifibox console)?

[venoom27]

wifibox:~# cat /proc/interrupts
           CPU0       
  0:        130   IO-APIC   2-edge      timer
  1:          0   IO-APIC   1-edge      i8042
  4:       2269   IO-APIC   4-edge      ttyS0
  8:          0   IO-APIC   8-edge      rtc0
  9:          0   IO-APIC   9-fasteoi   acpi
 12:          1   IO-APIC  12-edge      i8042
 20:        188   IO-APIC  20-fasteoi   eth0
 32:          0   PCI-MSI 65536-edge      virtio0-config
 33:        188   PCI-MSI 65537-edge      virtio0-req.0
 34:          0   PCI-MSI 67584-edge      virtio1-config
 35:         93   PCI-MSI 67585-edge      virtio1-requests
 36:          0   PCI-MSI 69632-edge      virtio2-config
 37:        524   PCI-MSI 69633-edge      virtio2-requests
 38:          0   PCI-MSI 71680-edge      virtio3-config
 39:          8   PCI-MSI 71681-edge      virtio3-requests
NMI:          0   Non-maskable interrupts
LOC:       4465   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:          0   IRQ work interrupts
RTR:          0   APIC ICR read retries
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
DFR:          0   Deferred Error APIC interrupts
MCE:          0   Machine check exceptions
MCP:          2   Machine check polls
ERR:          0
MIS:          0
PIN:          0   Posted-interrupt notification event
NPI:          0   Nested posted-interrupt event
PIW:          0   Posted-interrupt wakeup event

[pgj]

What is your FreeBSD version?

[venoom27]

I am running

13.2-RELEASE-p1

[pgj]

I have created version 20230716 that will display more information on the failure of the request_irq() function. Please share me the corresponding dmesg output.

[venoom27]

Here is the dmesg output from the wifibox

[    0.771608] Run /sbin/init as init process
[    0.772508]   with arguments:
[    0.772508]     /sbin/init
[    0.772509]   with environment:
[    0.772510]     HOME=/
[    0.772510]     TERM=linux
[    0.772511]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.772512]     modules=squashfs
[    1.356351] wl: loading out-of-tree module taints kernel.
[    1.356358] wl: module license 'MIXED/Proprietary' taints kernel.
[    1.356359] Disabling lock debugging due to kernel taint
[    1.360280] wl 0000:00:06.0: can't derive routing for PCI INT A
[    1.360285] wl 0000:00:06.0: PCI INT A: not connected
[    1.506288] ERROR @wl_cfg80211_detach : 
[    1.506291] NULL ndev->ieee80211ptr, unable to deref wl
[    1.506295] ------------[ cut here ]------------
[    1.506295] name 'brcm_monitor0'
[    1.506300] WARNING: CPU: 0 PID: 446 at fs/proc/generic.c:712 remove_proc_entry+0xfb/0x150
[    1.506305] Modules linked in: wl(PO+)
[    1.506309] CPU: 0 PID: 446 Comm: modprobe Tainted: P           O      5.15.104-0-lts #1-Alpine
[    1.506311] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[    1.506313] RIP: 0010:remove_proc_entry+0xfb/0x150
[    1.506315] Code: ff e9 5e ff ff ff 48 8b b0 98 00 00 00 48 c7 c7 78 21 51 ab e8 71 7e 3e 00 0f 0b 48 c7 c7 b7 20 51 ab 48 89 de e8 60 7e 3e 00 <0f> 0b e9 33 ff ff ff 48 89 df e8 76 fb 0b 00 48 8b 8d 98 00 00 00
[    1.506317] RSP: 0018:ffff9d7d40103b30 EFLAGS: 00010286
[    1.506319] RAX: 0000000000000000 RBX: ffff9d7d40103b68 RCX: 00000000ffffdfff
[    1.506320] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000009ffb
[    1.506321] RBP: 0000000000000000 R08: 00000000ffffdfff R09: 00000000ffffdfff
[    1.506323] R10: ffffffffab6536c0 R11: ffffffffab6536c0 R12: ffffffffab73b080
[    1.506324] R13: ffff8b5d8194f000 R14: ffff8b5d8194ec00 R15: 0000000000004331
[    1.506325] FS:  00007fe9adc73b48(0000) GS:ffffffffab64b000(0000) knlGS:0000000000000000
[    1.506327] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.506328] CR2: 000055c65012f224 CR3: 000000000195e005 CR4: 00000000001706f0
[    1.506330] Call Trace:
[    1.506332]  <TASK>
[    1.506333]  wl_free+0xb9/0x280 [wl]
[    1.506406]  ? wl_pci_probe+0x293/0xf80 [wl]
[    1.506441]  wl_pci_probe+0x5ff/0xf80 [wl]
[    1.506475]  pci_device_probe+0x8d/0xf0
[    1.506479]  really_probe.part.0+0x94/0x270
[    1.506484]  __driver_probe_device+0x89/0x110
[    1.506486]  driver_probe_device+0x19/0xd0
[    1.506489]  __driver_attach+0x78/0x160
[    1.506492]  ? __device_attach_driver+0x100/0x100
[    1.506494]  bus_for_each_dev+0x6a/0xb0
[    1.506497]  bus_add_driver+0x114/0x1d0
[    1.506500]  driver_register+0x84/0xd0
[    1.506503]  ? 0xffffffffc06a9000
[    1.506504]  do_one_initcall+0x45/0x180
[    1.506507]  ? kmem_cache_alloc+0x2c/0x120
[    1.506509]  do_init_module+0x40/0x210
[    1.506513]  __do_sys_init_module+0x14f/0x180
[    1.506516]  do_syscall_64+0x34/0x80
[    1.506520]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[    1.506523] RIP: 0033:0x7fe9adc12704
[    1.506526] Code: 0e 4c 8b 44 24 10 4d 8d 48 08 4c 89 4c 24 10 44 8b 4c 24 08 4d 8b 00 4c 01 c9 41 83 f9 2f 76 05 48 8b 4c 24 10 4c 8b 09 0f 05 <48> 89 c7 e8 56 66 fe ff 48 83 c4 58 c3 56 31 d2 be 02 00 08 00 bf
[    1.506527] RSP: 002b:00007ffd04f885d0 EFLAGS: 00000212 ORIG_RAX: 00000000000000af
[    1.506529] RAX: ffffffffffffffda RBX: 00007fe9adc73b7c RCX: 00007fe9adc12704
[    1.506530] RDX: 000055c650136da6 RSI: 0000000000731e48 RDI: 00007fe9acb88550
[    1.506531] RBP: 000055c650136da6 R08: 0000000000000004 R09: 0000000000731e48
[    1.506533] R10: 0000000000000000 R11: 0000000000000212 R12: 00007fe9acb88550
[    1.506534] R13: 0000000000000000 R14: 0000000000000001 R15: 000055c6501379e4
[    1.506535]  </TASK>
[    1.506536] ---[ end trace 97802034722f004e ]---

[pgj]

Oops, I think I have messed up something. There are no debug messages at all :-( I will fix it this up later.

[pgj]

Just a wild guess. I have read in the bhyve documentation that it sometimes matters for the PCI passthrough that the PCI slot configuration must match for the guest and the host (point #5). Whether is this a multi-function device? What does lspci say about the device on Arch Linux?

[pgj]

lspci -vvnn | grep -A 9 Network more specifically.

[venoom27]

On Arch Linux the command lspci -vvnn | grep -A 9 Network outputs:

02:00.0 Network controller [0280]: Broadcom Inc. and subsidiaries BCM4331 802.11a/b/g/n [14e4:4331] (rev 02)
        Subsystem: Broadcom Inc. and subsidiaries BCM4331 802.11a/b/g/n [14e4:4331]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 17
        Region 0: Memory at a0600000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: wl
        Kernel modules: bcma, wl

[pgj]

The bhyve documentation suggests that only device drivers that support Message-Signaled Interrupts (MSI) would work with PCI pass-through. Looks like the wl driver blindly accepts the IRQ number (255) that it gets from the kernel on probing the device. While looking at the code of other Linux drivers, some extra work needs to be done to find an IRQ line that could actually be used.

Do you see any bhyve errors in the logs? What does /var/log/wifibox.log say on the host? Mind that you would have to increase the level of verbosity to debug to see everything. This can be controlled in core.conf next to bhyve.conf.

loglevel=debug

It is not sure, but you may find messages like this one:

bhyve: failed to initialize MSI for PCI 5/0/0

In the meantime, I have fixed the debugging code for the driver. Please try again with wifibox-alpine version 20230718.

[venoom27]

I turned on loglevel=debug in thecore.conf . Here is the output

2023-07-18T07:31:43-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: stop
2023-07-18T07:31:43-0400 INFO  Begin: wifibox stop
2023-07-18T07:31:43-0400 DEBUG stop=GN
2023-07-18T07:31:43-0400 INFO  Tearing down Unix Domain Socket pass-through
2023-07-18T07:31:43-0400 INFO  Daemonized socat processes found: [339
344]
2023-07-18T07:31:43-0400 INFO  Stopping guest wifibox, managed by PID 257
2023-07-18T07:31:43-0400 INFO  Waiting 3 seconds for the guest to stop
2023-07-18T07:31:46-0400 INFO  Forcing shutdown of guest wifibox
2023-07-18T07:31:46-0400 INFO  Destroying guest wifibox
2023-07-18T07:31:47-0400 INFO  Destroying bhyve PPT device: pci2:0:0
2023-07-18T07:31:47-0400 INFO  Removing null-modem devices
2023-07-18T07:31:47-0400 INFO  Null-modem devices are removed
2023-07-18T07:31:47-0400 INFO  Destroying bridge interface: wifibox0
2023-07-18T07:31:47-0400 INFO  Destroying linked tap interface: tap0
2023-07-18T07:31:47-0400 INFO  End: wifibox stop
2023-07-18T07:32:31-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-07-18T07:32:31-0400 INFO  Begin: wifibox start
2023-07-18T07:32:31-0400 DEBUG start=GN
2023-07-18T07:32:31-0400 INFO  Creating bridge interface: wifibox0
2023-07-18T07:32:31-0400 DEBUG [ifconfig] wifibox0
2023-07-18T07:32:31-0400 INFO  Linking tap interface to wifibox0: tap0
2023-07-18T07:32:31-0400 DEBUG assert loaded: kmod=vmm, kmod_file=/boot/modules/vmm.ko
2023-07-18T07:32:31-0400 DEBUG Kernel module vmm is not loaded
2023-07-18T07:32:31-0400 DEBUG Kernel module vmm was loaded successfully
2023-07-18T07:32:31-0400 DEBUG assert hardware support present: iommu=1, amdvi=0
2023-07-18T07:32:31-0400 DEBUG Backends reported by bhyve:
2023-07-18T07:32:31-0400 DEBUG [bhyve] ahci
2023-07-18T07:32:31-0400 DEBUG [bhyve] ahci-hd
2023-07-18T07:32:31-0400 DEBUG [bhyve] ahci-cd
2023-07-18T07:32:31-0400 DEBUG [bhyve] e1000
2023-07-18T07:32:31-0400 DEBUG [bhyve] dummy
2023-07-18T07:32:31-0400 DEBUG [bhyve] hda
2023-07-18T07:32:31-0400 DEBUG [bhyve] fbuf
2023-07-18T07:32:31-0400 DEBUG [bhyve] amd_hostbridge
2023-07-18T07:32:31-0400 DEBUG [bhyve] hostbridge
2023-07-18T07:32:31-0400 DEBUG [bhyve] lpc
2023-07-18T07:32:31-0400 DEBUG [bhyve] nvme
2023-07-18T07:32:31-0400 DEBUG [bhyve] passthru
2023-07-18T07:32:31-0400 DEBUG [bhyve] virtio-9p
2023-07-18T07:32:31-0400 DEBUG [bhyve] virtio-blk
2023-07-18T07:32:31-0400 DEBUG [bhyve] virtio-console
2023-07-18T07:32:31-0400 DEBUG [bhyve] virtio-net
2023-07-18T07:32:31-0400 DEBUG [bhyve] virtio-rnd
2023-07-18T07:32:31-0400 DEBUG [bhyve] virtio-scsi
2023-07-18T07:32:31-0400 DEBUG [bhyve] uart
2023-07-18T07:32:31-0400 DEBUG [bhyve] xhci
2023-07-18T07:32:31-0400 INFO  Bringing up null-modem devices for console connection
2023-07-18T07:32:31-0400 DEBUG assert loaded: kmod=nmdm, kmod_file=nmdm
2023-07-18T07:32:31-0400 DEBUG Kernel module nmdm is not loaded
2023-07-18T07:32:31-0400 DEBUG Kernel module nmdm was loaded successfully
2023-07-18T07:32:31-0400 INFO  Null-modem devices are available
2023-07-18T07:32:31-0400 INFO  Waiting for bhyve to start up
2023-07-18T07:32:31-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-07-18T07:32:31-0400 DEBUG assert daemonized: parent=daemon
2023-07-18T07:32:31-0400 INFO  VM manager launched
2023-07-18T07:32:31-0400 INFO  Gathering necessary configuration files for launching the guest
2023-07-18T07:32:31-0400 INFO  Pulling bhyve options from configuration file
2023-07-18T07:32:31-0400 DEBUG cpus=1, memory=128M, passthru=2/0/0, console=yes
2023-07-18T07:32:31-0400 INFO  Guest console is configured to use
2023-07-18T07:32:31-0400 INFO  Passthru device is configured: 2/0/0
2023-07-18T07:32:31-0400 INFO  PPT driver is configured for pci2:0:0 device
2023-07-18T07:32:31-0400 INFO  tap interface is configured: tap0
2023-07-18T07:32:31-0400 INFO  Launching guest wifibox from /usr/local/share/wifibox with grub-bhyve
2023-07-18T07:32:31-0400 INFO  Application config is found at /usr/local/etc/wifibox/wpa_supplicant
2023-07-18T07:32:31-0400 INFO  Application config will be mounted writeable
2023-07-18T07:32:31-0400 DEBUG Devices: virtio-blk,/usr/local/share/wifibox/disk.img virtio-9p,config=/usr/local/etc/wifibox/appliance,ro virtio-9p,var=/var/run/wifibox/appliance virtio-9p,app_config=/usr/local/etc/wifibox/wpa_supplicant
2023-07-18T07:32:31-0400 DEBUG Arguments:  -S -M 128M -r host -c /dev/nmdm-wifibox.1A -d /usr/local/share/wifibox wifibox
2023-07-18T07:32:31-0400 INFO  Launching guest wifibox from /usr/local/share/wifibox with bhyve
2023-07-18T07:32:31-0400 DEBUG Arguments:  -c 1 -m 128M -AHP -u -S -l com1,/dev/nmdm-wifibox.1A -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/usr/local/share/wifibox/disk.img -s 4:1,virtio-9p,config=/usr/local/etc/wifibox/appliance,ro -s 4:2,virtio-9p,var=/var/run/wifibox/appliance -s 4:3,virtio-9p,app_config=/usr/local/etc/wifibox/wpa_supplicant -s 5:0,e1000,tap0 -s 6:0,passthru,2/0/0 wifibox
2023-07-18T07:32:32-0400 DEBUG [bhyve] rdmsr to register 0x140 on vcpu 0
2023-07-18T07:32:32-0400 DEBUG [bhyve] rdmsr to register 0x34 on vcpu 0
2023-07-18T07:32:32-0400 INFO  Guest wifibox has started up
2023-07-18T07:32:32-0400 INFO  Guest is managed by PID 401
2023-07-18T07:32:32-0400 INFO  Bringing up Unix Domain Socket pass-through
2023-07-18T07:32:32-0400 INFO  Found guest IP address: 10.0.0.1
2023-07-18T07:32:32-0400 INFO  Configured sockets: [path=/var/run/wpa_supplicant/wlan0,user=root,group=0,mode=770,port=1200
path=/var/run/wpa_supplicant/p2p-dev-wlan0,user=root,group=0,mode=770,port=1201]
2023-07-18T07:32:32-0400 INFO  Hooking up 10.0.0.1:1200 as /var/run/wpa_supplicant/wlan0 (root:0@770)
2023-07-18T07:32:32-0400 INFO  Hooking up 10.0.0.1:1201 as /var/run/wpa_supplicant/p2p-dev-wlan0 (root:0@770)
2023-07-18T07:32:32-0400 INFO  End: wifibox start

Here is the dmesg from the 718 build

[    0.776532] Run /sbin/init as init process
[    0.777419]   with arguments:
[    0.777420]     /sbin/init
[    0.777421]   with environment:
[    0.777421]     HOME=/
[    0.777422]     TERM=linux
[    0.777422]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.777423]     modules=squashfs
[    1.390762] wl: loading out-of-tree module taints kernel.
[    1.390770] wl: module license 'MIXED/Proprietary' taints kernel.
[    1.390771] Disabling lock debugging due to kernel taint
[    1.394574] wl_pci_probe: bus 0 slot 6 func 0 irq 255
[    1.394714] wl 0000:00:06.0: can't derive routing for PCI INT A
[    1.394717] wl 0000:00:06.0: PCI INT A: not connected
[    1.394856] wl_alloc_linux_if
[    1.394860] wl0: Bus: 
[    1.394861] PCI/DMA
[    1.540013] wl0: online cpus 1
[    1.540016] wl0: request_irq() failed: code=[-107]
[    1.540017] wl: wl_free
[    1.540018] wl_free_if
[    1.540019] ERROR @wl_cfg80211_detach : 
[    1.540020] NULL ndev->ieee80211ptr, unable to deref wl
[    1.540024] ------------[ cut here ]------------
[    1.540025] name 'brcm_monitor0'
[    1.540030] WARNING: CPU: 0 PID: 446 at fs/proc/generic.c:712 remove_proc_entry+0xfb/0x150
[    1.540036] Modules linked in: wl(PO+)
[    1.540040] CPU: 0 PID: 446 Comm: modprobe Tainted: P           O      5.15.104-0-lts #1-Alpine
[    1.540043] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[    1.540044] RIP: 0010:remove_proc_entry+0xfb/0x150
[    1.540047] Code: ff e9 5e ff ff ff 48 8b b0 98 00 00 00 48 c7 c7 78 21 11 8f e8 71 7e 3e 00 0f 0b 48 c7 c7 b7 20 11 8f 48 89 de e8 60 7e 3e 00 <0f> 0b e9 33 ff ff ff 48 89 df e8 76 fb 0b 00 48 8b 8d 98 00 00 00
[    1.540049] RSP: 0018:ffff97f6400fbb30 EFLAGS: 00010286
[    1.540051] RAX: 0000000000000000 RBX: ffff97f6400fbb68 RCX: 00000000ffffdfff
[    1.540052] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000009ffb
[    1.540053] RBP: 0000000000000000 R08: 00000000ffffdfff R09: 00000000ffffdfff
[    1.540054] R10: ffffffff8f2536c0 R11: ffffffff8f2536c0 R12: ffffffff8f33b080
[    1.540056] R13: ffff90af0192a000 R14: ffff90af013019c0 R15: ffff90af0194f000
[    1.540057] FS:  00007f0694f04b48(0000) GS:ffffffff8f24b000(0000) knlGS:0000000000000000
[    1.540059] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.540060] CR2: 00007f0694d8c000 CR3: 000000000199a005 CR4: 00000000001706f0
[    1.540061] Call Trace:
[    1.540121]  <TASK>
[    1.540124]  wl_free+0xf3/0x21d [wl]
[    1.540195]  ? _printk+0x53/0x6a
[    1.540200]  wl_pci_probe+0x852/0xf80 [wl]
[    1.540237]  pci_device_probe+0x8d/0xf0
[    1.540241]  really_probe.part.0+0x94/0x270
[    1.540245]  __driver_probe_device+0x89/0x110
[    1.540248]  driver_probe_device+0x19/0xd0
[    1.540251]  __driver_attach+0x78/0x160
[    1.540254]  ? __device_attach_driver+0x100/0x100
[    1.540257]  bus_for_each_dev+0x6a/0xb0
[    1.540259]  bus_add_driver+0x114/0x1d0
[    1.540262]  driver_register+0x84/0xd0
[    1.540265]  ? 0xffffffffc0854000
[    1.540266]  do_one_initcall+0x45/0x180
[    1.540269]  ? kmem_cache_alloc+0x2c/0x120
[    1.540271]  do_init_module+0x40/0x210
[    1.540275]  __do_sys_init_module+0x14f/0x180
[    1.540278]  do_syscall_64+0x34/0x80
[    1.540282]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[    1.540285] RIP: 0033:0x7f0694ea3704
[    1.540287] Code: 0e 4c 8b 44 24 10 4d 8d 48 08 4c 89 4c 24 10 44 8b 4c 24 08 4d 8b 00 4c 01 c9 41 83 f9 2f 76 05 48 8b 4c 24 10 4c 8b 09 0f 05 <48> 89 c7 e8 56 66 fe ff 48 83 c4 58 c3 56 31 d2 be 02 00 08 00 bf
[    1.540289] RSP: 002b:00007ffe60a94a80 EFLAGS: 00000212 ORIG_RAX: 00000000000000af
[    1.540291] RAX: ffffffffffffffda RBX: 00007f0694f04b7c RCX: 00007f0694ea3704
[    1.540292] RDX: 00005654cbab9da6 RSI: 0000000000734960 RDI: 00007f0693e19550
[    1.540293] RBP: 00005654cbab9da6 R08: 0000000000000004 R09: 0000000000734960
[    1.540294] R10: 0000000000000000 R11: 0000000000000212 R12: 00007f0693e19550
[    1.540295] R13: 0000000000000000 R14: 0000000000000001 R15: 00005654cbaba9e4
[    1.540297]  </TASK>
[    1.540298] ---[ end trace 226c4c59d2f062d1 ]---
[    2.640721] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[    2.643683] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    2.678835] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    4.754242] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    6.757995] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    8.762670] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[   10.771276] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[   12.775532] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)

[pgj]

The story is slowly coming together. The IRQ "number" passed for wl_attach() is not valid one but the IRQ_NOTCONNECTED signal, which indicates the need for MSI. That is why request_irq() returns -ENOTCONN (as an alias for request_threaded_irq()) which corresponds to the return value of -107.

[pgj]

Could you please send me the output of pciconf(8) from the FreeBSD host? Something along the following lines as root: pciconf -lc | fgrep ppt0 -A 9. The output should look like this:

ppt0@pci0:3:0:0:        class=0x028000 card=0x00108086 chip=0x24f38086 rev=0x3a hdr=0x00
    cap 01[c8] = powerspec 3  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[40] = PCI-Express 2 endpoint max data 128(128) FLR RO NS
                 max read 128
                 link x1(x1) speed 2.5(2.5) ASPM L1(L1) ClockPM enabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 7c7635ffff9903cc
    ecap 0018[14c] = LTR 1
    ecap 001e[154] = L1 PM Substates 1

[venoom27]

Here is the output of pciconf -lc | fgrep ppt0 -A 9

ppt0@pci0:2:0:0:        class=0x028000 rev=0x02 hdr=0x00 vendor=0x14e4 device=0x4331 subvendor=0x14e4 subdevice=0x4331
    cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 09[58] = vendor (length 120)
    cap 05[48] = MSI supports 1 message, 64 bit 
    cap 10[d0] = PCI-Express 1 endpoint max data 128(128)
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM L0s/L1(L0s/L1) ClockPM enabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0002[13c] = VC 1 max VC0
    ecap 0003[160] = Serial 1 eaa6cfffff22a8bb

[pgj]

Sounds like a good news. This shows that the card itself supports MSI but it is not handled by the driver.

[pgj]

I have created version 20230719, which tries to setup an MSI-based interrupt. Maybe it works. Please try that.

[venoom27]

I installed the 20230719 and here is the dmesg from the wifibox

[    0.771073] Run /sbin/init as init process
[    0.771995]   with arguments:
[    0.771996]     /sbin/init
[    0.771996]   with environment:
[    0.771997]     HOME=/
[    0.771997]     TERM=linux
[    0.771998]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.771998]     modules=squashfs
[    1.359706] wl: loading out-of-tree module taints kernel.
[    1.359712] wl: module license 'MIXED/Proprietary' taints kernel.
[    1.359713] Disabling lock debugging due to kernel taint
[    1.363530] wl_pci_probe: bus 0 slot 6 func 0 irq 255
[    1.363692] wl 0000:00:06.0: can't derive routing for PCI INT A
[    1.363695] wl 0000:00:06.0: PCI INT A: not connected
[    1.363810] wl_alloc_linux_if
[    1.363813] wl0: Bus: 
[    1.363814] PCI/DMA
[    1.506862] wl0: online cpus 1
[    1.506865] wl 0000:00:06.0: MSI enabled: false
[    1.506867] wl 0000:00:06.0: MSI disabled: false
[    1.506868] wl0: request_irq() failed, ret -107
[    1.506869] wl: wl_free
[    1.506870] wl_free_if
[    1.506871] ERROR @wl_cfg80211_detach : 
[    1.506872] NULL ndev->ieee80211ptr, unable to deref wl
[    1.506875] ------------[ cut here ]------------
[    1.506876] name 'brcm_monitor0'
[    1.506880] WARNING: CPU: 0 PID: 446 at fs/proc/generic.c:712 remove_proc_entry+0xfb/0x150
[    1.506886] Modules linked in: wl(PO+)
[    1.506889] CPU: 0 PID: 446 Comm: modprobe Tainted: P           O      5.15.104-0-lts #1-Alpine
[    1.506892] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[    1.506893] RIP: 0010:remove_proc_entry+0xfb/0x150
[    1.506895] Code: ff e9 5e ff ff ff 48 8b b0 98 00 00 00 48 c7 c7 78 21 f1 94 e8 71 7e 3e 00 0f 0b 48 c7 c7 b7 20 f1 94 48 89 de e8 60 7e 3e 00 <0f> 0b e9 33 ff ff ff 48 89 df e8 76 fb 0b 00 48 8b 8d 98 00 00 00
[    1.506897] RSP: 0018:ffff9b0ec00fbb30 EFLAGS: 00010286
[    1.506899] RAX: 0000000000000000 RBX: ffff9b0ec00fbb68 RCX: 00000000ffffdfff
[    1.506901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000009ffb
[    1.506902] RBP: 0000000000000000 R08: 00000000ffffdfff R09: 00000000ffffdfff
[    1.506903] R10: ffffffff950536c0 R11: ffffffff950536c0 R12: ffffffff9513b080
[    1.506904] R13: ffff8bcf01932000 R14: ffff8bcf01357900 R15: ffff8bcf0194f000
[    1.506906] FS:  00007f36170feb48(0000) GS:ffffffff9504b000(0000) knlGS:0000000000000000
[    1.506907] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.506909] CR2: 00007f3616748ffc CR3: 0000000001988005 CR4: 00000000001706f0
[    1.506910] Call Trace:
[    1.506912]  <TASK>
[    1.506914]  wl_free+0xfc/0x226 [wl]
[    1.506981]  ? _printk+0x53/0x6a
[    1.506985]  wl_pci_probe+0x93b/0xf80 [wl]
[    1.507020]  pci_device_probe+0x8d/0xf0
[    1.507024]  really_probe.part.0+0x94/0x270
[    1.507029]  __driver_probe_device+0x89/0x110
[    1.507031]  driver_probe_device+0x19/0xd0
[    1.507034]  __driver_attach+0x78/0x160
[    1.507037]  ? __device_attach_driver+0x100/0x100
[    1.507040]  bus_for_each_dev+0x6a/0xb0
[    1.507042]  bus_add_driver+0x114/0x1d0
[    1.507045]  driver_register+0x84/0xd0
[    1.507048]  ? 0xffffffffc07ef000
[    1.507049]  do_one_initcall+0x45/0x180
[    1.507052]  ? kmem_cache_alloc+0x2c/0x120
[    1.507054]  do_init_module+0x40/0x210
[    1.507058]  __do_sys_init_module+0x14f/0x180
[    1.507061]  do_syscall_64+0x34/0x80
[    1.507128]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[    1.507132] RIP: 0033:0x7f361709d704
[    1.507134] Code: 0e 4c 8b 44 24 10 4d 8d 48 08 4c 89 4c 24 10 44 8b 4c 24 08 4d 8b 00 4c 01 c9 41 83 f9 2f 76 05 48 8b 4c 24 10 4c 8b 09 0f 05 <48> 89 c7 e8 56 66 fe ff 48 83 c4 58 c3 56 31 d2 be 02 00 08 00 bf
[    1.507135] RSP: 002b:00007ffd0cc2b8f0 EFLAGS: 00000212 ORIG_RAX: 00000000000000af
[    1.507138] RAX: ffffffffffffffda RBX: 00007f36170feb7c RCX: 00007f361709d704
[    1.507139] RDX: 000055868b02fda6 RSI: 0000000000734ce0 RDI: 00007f3616013550
[    1.507140] RBP: 000055868b02fda6 R08: 0000000000000004 R09: 0000000000734ce0
[    1.507141] R10: 0000000000000000 R11: 0000000000000212 R12: 00007f3616013550
[    1.507142] R13: 0000000000000000 R14: 0000000000000001 R15: 000055868b0309e4
[    1.507143]  </TASK>
[    1.507144] ---[ end trace 9f069610c6f8d5ec ]---

[pgj]

Hrm, okay... I have just learned that MSI will be enabled automatically as the driver starts to use it. The reason why it was not used in the first place is because in this version it is checked first then attempted to use. But, apparently, there is no need for this extra round. Well, check only if MSI is not available at all.

[pgj]

I have added 20230719_1 (note the port revision bump) that fixes the problem with initializing the MSI layer. Please try this version.

[venoom27]

it looks like it worked but I can't connect to a network. Any tips? Here is the dmesg

[    0.768856] Run /sbin/init as init process
[    0.769816]   with arguments:
[    0.769817]     /sbin/init
[    0.769818]   with environment:
[    0.769818]     HOME=/
[    0.769819]     TERM=linux
[    0.769819]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.769820]     modules=squashfs
[    2.684147] wl: loading out-of-tree module taints kernel.
[    2.684155] wl: module license 'MIXED/Proprietary' taints kernel.
[    2.684156] Disabling lock debugging due to kernel taint
[    2.688148] wl_pci_probe: bus 0 slot 6 func 0 irq 255
[    2.688148] wl 0000:00:06.0: can't derive routing for PCI INT A
[    2.688148] wl 0000:00:06.0: PCI INT A: not connected
[    2.688148] wl_alloc_linux_if
[    2.688148] wl0: Bus: 
[    2.688148] PCI/DMA
[    2.821224] wl0: online cpus 1
[    2.821229] wl 0000:00:06.0: MSI supported: true
[    2.821593] wl 0000:00:06.0: (MSI) allocated 1 IRQ vectors
[    2.821595] wl 0000:00:06.0: (MSI) requesting IRQ: 40
[    2.821918] wl0: wlc_ioctl_internal: cmd 0x106
[    2.822009] wl0: wl_get_stats
[    2.822155] wl_set_radio_block: kernel set blocked = 0
[    2.822303] wl0: wl_up
[    2.833004] wl0: wl_init
[    2.833006] wl0: wl_reset
[    2.848004] wl_init_rfkill: rfkill registered
[    2.849539] wlan0: Broadcom BCM4331 802.11 Hybrid Wireless Controller 6.30.223.271 (r587334)
[    2.849542] 
[    3.299207] random: wpa_supplicant: uninitialized urandom read (1027 bytes read)
[    3.394500] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[    3.397347] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    3.412892] wl0: wl_get_stats
[    3.466397] wl0: wlc_ioctl_internal: cmd 0x106
[    3.466471] wl0: wlc_ioctl_internal: cmd 0x106
[    3.466692] wl0: wlc_ioctl_internal: cmd 0x106
[    3.466734] wl0: wlc_ioctl_internal: cmd 0x106
[    3.467664] wl0: wlc_ioctl_internal: cmd 0x106
[    3.467715] wl0: wl_open
[    3.467717] wl0: wl_up
[    3.467723] wl0: wlc_ioctl_internal: cmd 0x14
[    3.467727] wl0: wlc_ioctl_internal: cmd 0x76
[    3.467730] wl0: wlc_ioctl_internal: cmd 0xb4
[    3.467739] wl0: wlc_ioctl_internal: cmd 0x106
[    3.467750] wl0: wl_set_multicast_list
[    3.467771] wl0: wl_get_stats
[    3.467781] wl0: wl_set_multicast_list
[    3.467790] wl0: wlc_ioctl_internal: cmd 0x56
[    3.467958] wl0: wl_set_multicast_list
[    3.467988] wl0: wl_get_stats
[    3.467995] wl0: wl_get_stats
[    3.468940] wl0: wlc_ioctl_internal: cmd 0x107
[    3.475717] wl0: wl_start: len 90 data_len 0 summed 0 csum: 0x0
[    3.511206] wl0: wl_get_stats
[    3.511996] wl0: wl_start: len 342 data_len 0 summed 0 csum: 0x0
[    3.943124] wl0: wl_start: len 90 data_len 0 summed 0 csum: 0x0
[    3.995121] random: crng init done
[    4.451459] wl0: wl_start: len 86 data_len 0 summed 0 csum: 0x0
[    5.464929] wl0: wl_start: len 90 data_len 0 summed 0 csum: 0x0
[    5.464945] wl0: wl_start: len 70 data_len 0 summed 0 csum: 0x0
[    5.663959] wl0: wl_start: len 90 data_len 0 summed 0 csum: 0x0
[    5.920743] wl0: wl_down
[    5.920957] wl0: wl_reset

[pgj]

That is great! Is the device still available on the guest? What does ifconfig say there? Is wpa_supplicant configured?

[venoom27]

Here is the ifconfig in bsd

wifibox0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:00:34:21
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:f0:02
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 520

Here is the ifconfig for the wifibox

eth0      Link encap:Ethernet  HWaddr 00:A0:98:8A:05:71  
          inet addr:10.0.0.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::2a0:98ff:fe8a:571/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:866 (866.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr A8:BB:CF:22:EA:A6  
          inet6 addr: fe80::aabb:cfff:fe22:eaa6/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:40 

I configured the wpa_supplicant.conf as follows

ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
update_config=1

network={
        ssid="MYSSID"
        #psk=my secret code"
        psk=mysecretcode
}

[pgj]

I have published version 20230720 which includes the rfkill command that could be used if the driver is blocked either in software or hardware. On the guest, you can now use the rfkill command to query the status and change it. For example:

# rfkill list all
0: phy0: wlan
        Soft blocked: no
        Hard blocked: no

There is also iw that could be used for exploration. For example:

# iw dev wlan0 link
Connected to <REDACTED> (on wlan0)
        SSID: <REDACTED>
        freq: 5560
        RX: 1131805 bytes (3023 packets)
        TX: 435415 bytes (2904 packets)
        signal: -52 dBm
        rx bitrate: 650.0 MBit/s VHT-MCS 7 80MHz short GI VHT-NSS 2
        tx bitrate: 650.0 MBit/s VHT-MCS 7 80MHz short GI VHT-NSS 2

        bss flags:      short-slot-time
        dtim period:    1
        beacon int:     100

Or scan for the nearby networks:

# iw dev wlan0 scan
...

It would be also good to see what is happening for wpa_supplicant. You can stop the corresponding service and start WPA Supplicant in the foreground as follows:

# service wpa_supplicant stop
 * WARNING: you are stopping a boot service
 * Stopping WPA Supplicant ...
 [ ok ]
# wpa_supplicant -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf
...

[venoom27]

When I go into the console and type I get

wifibox:~# rfkill list all
-ash: rfkill: not found

iw dev wlan0 link gives

wifibox:~# iw dev wlan0 link
Not connected.

iw dev wlan0 scan shows all the available networks around me.

wpa_supplicant -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf says

wifibox:~# wpa_supplicant -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf
Successfully initialized wpa_supplicant
wlan0: CTRL-EVENT-SCAN-FAILED ret=-22 retry=1

[venoom27]

Also when I run wpa_cli from the host (the freebsd os) I get

Warning: Failed to attach to wpa_supplicant.
Could not connect to wpa_supplicant: wlan0 - re-trying
Warning: Failed to attach to wpa_supplicant.

[pgj]

That is strange. I tried the 20230720 version of wifibox-alpine myself, and the rfkill command was there. That is where I got the sample output.

[pgj]

Oh, and I think I forgot to recommend adding the -d flag for WPA Supplicant, for verbose output:

wpa_supplicant -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf -d

[pgj]

Based on the submitted logs, published version 20230721 which speculatively includes a custom fix in WPA Supplicant for Broadcom devices from Arch Linux. Apparently they care more about broadcom-wl than other distributions. Please test it and report.

[venoom27]

It’s worth a try. I can test if you do it.

[venoom27]

If it matters here is part of the dmesg from the wifibox

[  255.234095] wl0: wl_start: len 98 data_len 0 summed 0 csum: 0x3e9dc162
[  255.274759] wl0: wl_sendup: 56 bytes
[  255.373086] wl0: wl_sendup: 60 bytes
[  255.447403] wl0: wl_start: len 42 data_len 0 summed 0 csum: 0x0
[  255.449825] wl0: wl_sendup: 98 bytes
[  255.450523] wl0: wl_sendup: 98 bytes
[  255.450537] wl0: wl_sendup: 56 bytes
[  256.237308] wl0: wl_start: len 98 data_len 0 summed 0 csum: 0xab2654d9
[  256.252863] wl0: wl_sendup: 98 bytes
[  257.247458] wl0: wl_start: len 98 data_len 0 summed 0 csum: 0xb9ef461
[  257.262268] wl0: wl_sendup: 98 bytes

I also got the message from the bsd box while I was pinging.

ping: sendto: No buffer space available
64 bytes from 142.251.163.104: icmp_seq=213 ttl=56 time=86.479 ms
64 bytes from 142.251.163.104: icmp_seq=214 ttl=56 time=197.352 ms
64 bytes from 142.251.163.104: icmp_seq=215 ttl=56 time=77.852 ms
64 bytes from 142.251.163.104: icmp_seq=216 ttl=56 time=315.762 ms
64 bytes from 142.251.163.104: icmp_seq=217 ttl=56 time=320.732 ms

[venoom27]

Also when I ping my router there is no packet lost but when I ping google I get around 50% packet loss. Do I have something configured wrong?

[pgj]

Ideally, it should just be working. If the packets cannot flow reliably between the local network on the Internet, it might be due to problems with the Network Address Translation (NAT) and the packet forwarding (PF). You can check the related statistics by using iptables (on the guest), e.g.

# iptables -L -nv
Chain INPUT (policy ACCEPT 2095 packets, 113K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 3007 packets, 1576K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2567 1006K ACCEPT     all  --  eth0   wlan0   0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
  356 64748 ACCEPT     all  --  eth0   wlan0   0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 1397 packets, 86552 bytes)
 pkts bytes target     prot opt in     out     source               destination

Due to the need for NAT and PF, the Wifibox guest has a firewall set up implicitly which may block the use of certain applications and protocols, mostly the ones that rely on peer-to-peer communication. But ping , use of a browser over HTTP or HTTPS, downloading files, Bittorrent, SSH outside, etc. should just work out of the (Wifi)box.

But it might also be possible that the version of WPA Supplicant that is running on the Wifibox guest has problems with your router and the connection is not configured optimally. I have noticed that Arch Linux has extra patches for WPA Supplicant (which are not in upstream, for some reason) and we could experiment further with importing those ones as well.

[pgj]

Mind that the another difference from Arch Linux is that we had to switch to using Message-Signaled Interrupts which is not how that the driver has originally been designed. In my humble understanding, the Linux MSI and PCI stack could just work in "legacy mode" to emulate the pin-based interrupt delivery. But there may be aspects that were not considered and the driver still needs work to support PCI pass-through properly.

[pgj]

By the way, what kind of system is this? What is the CPU and the model of the computer? I am asking because I would like to add it to the list of compatible configurations.

[venoom27]

It's a MacbookPro 9,2 i5-321.

Also I maybe delayed in testing this week if you come up with anything. I need to buy a new power adapter for the computer which will take a couple of days.

[venoom27]

Also if it matters I had installed Ubuntu the current LTS on the same computer about 3 months ago and the Wi-Fi would drop out like it does in FreeBSD wifibox. I didn’t look to see what driver it was using at the time.

[pgj]

Ubuntu 22.04.2 (Jammy Jellyfish) LTS ships with Linux 5.19, while 22.04 was shipped with Linux 5.15. This is a bit odd as Linux 5.19 was only a short-lived branch and it has reached its end-of-life in last October. Anyhow, this is not Linux 6, which can give us some hope :-)

[pgj]

For your information, I have integrated the changes from the dev branch to main. Starting from version 20230728 of net/wifibox-alpine the Broadcom wireless should work.

[venoom27]

Ok thanks.

[venoom27]

I got my new power adapter in so I can use the laptop if you come up with anything else to test. Thanks again for your help.

[venoom27]

I have noticed two things with the wifbox main and the dev branch we were testing. One suspend to ram does not work when either are installed and during boot starting sendmail_submit and starting sendmail_msp_queue sit for at less 30 seconds. Once I deinstall the wifibox these two things go away.

[pgj]

I think it would be better if you created a bug report about this. It seems to be unrelated to the topic of discussion.

[pgj]

I was able to upgrade Wifibox/Alpine to Alpine Linux 3.18.2, which means that it now has Linux 6.1 (6.1.42), which you can test. The updated version of the net/wifibox-alpine port is available on the dev branch, with version 20230803.

[venoom27]

I install it and the wlan0 did not attach. Here is the dmesg

[    0.777395] Run /sbin/init as init process
[    0.777866]   with arguments:
[    0.777867]     /sbin/init
[    0.777867]   with environment:
[    0.777868]     HOME=/
[    0.777869]     TERM=linux
[    0.777869]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.777870]     modules=squashfs
[    1.450470] wl: version magic '6.1.42-0-lts SMP preempt mod_unload modversions ' should be '6.1.42-0-lts mod_unload '
[    1.718493] wl: version magic '6.1.42-0-lts SMP preempt mod_unload modversions ' should be '6.1.42-0-lts mod_unload '
[    2.520260] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[    2.521114] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    2.715122] random: crng init done

Let me know if you need any other information.

[pgj]

Ah, the third-party kernel modules were not compiled with the right headers. I have now fixed this in version 20230805.

[venoom27]

I installed the update. It connected to the wlan0. Overall I am not seeing any changes in the ping.

wifibox:~# ping www.google.com
PING www.google.com (172.253.115.104): 56 data bytes
64 bytes from 172.253.115.104: seq=0 ttl=103 time=85.650 ms
64 bytes from 172.253.115.104: seq=1 ttl=103 time=30.799 ms
64 bytes from 172.253.115.104: seq=2 ttl=103 time=58.721 ms
64 bytes from 172.253.115.104: seq=3 ttl=103 time=21.532 ms
64 bytes from 172.253.115.104: seq=4 ttl=103 time=18.842 ms
64 bytes from 172.253.115.104: seq=5 ttl=103 time=18.080 ms
64 bytes from 172.253.115.104: seq=6 ttl=103 time=16.956 ms
64 bytes from 172.253.115.104: seq=7 ttl=103 time=18.270 ms
64 bytes from 172.253.115.104: seq=38 ttl=103 time=2466.830 ms
64 bytes from 172.253.115.104: seq=39 ttl=103 time=1467.090 ms
64 bytes from 172.253.115.104: seq=40 ttl=103 time=469.022 ms
64 bytes from 172.253.115.104: seq=41 ttl=103 time=16.906 ms
64 bytes from 172.253.115.104: seq=42 ttl=103 time=14.406 ms

dmesg

[   0.762752] Run /sbin/init as init process
[    0.763280]   with arguments:
[    0.763281]     /sbin/init
[    0.763282]   with environment:
[    0.763282]     HOME=/
[    0.763283]     TERM=linux
[    0.763283]     BOOT_IMAGE=(host)/usr/local/share/wifibox/vmlinuz
[    0.763284]     modules=squashfs
[    1.522555] wl: loading out-of-tree module taints kernel.
[    1.522596] wl: module license 'MIXED/Proprietary' taints kernel.
[    1.522598] Disabling lock debugging due to kernel taint
[    1.541738] wl 0000:00:06.0: can't derive routing for PCI INT A
[    1.541744] wl 0000:00:06.0: PCI INT A: not connected
[    1.737682] wl 0000:00:06.0: MSI supported: true
[    1.739871] wl 0000:00:06.0: (MSI) allocated 1 IRQ vectors
[    1.739874] wl 0000:00:06.0: (MSI) requesting IRQ: 40
[    1.807797] wlan0: Broadcom BCM4331 802.11 Hybrid Wireless Controller 6.30.223.271 (r587334)
[    1.807801] 
[    2.352257] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[    2.353116] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    2.570123] random: crng init done

wifibox:~# uname -r
6.1.43-0-lts

[pgj]

Oh, I have just noticed that you used Linux 6.4 in Arch Linux. But I have seen there is also a linux-lts package for Arch Linux which ships version 6.1. Could you please try downgrading your Arch Linux installation to that version just to see if the problem shows up there? If not, then the issue could be caused by the different kernel configurations. But first let's try to answer this question.

[venoom27]

I installed the linux-lts pack

uname -r
6.1.43-1-lts

I then had to remove the broadcom-wl pkg and then install the broadcom-wl-dkms-6.30.223.271-36 version.
ping was fine for about 2 minutes but then the wlan0 just drops and doesn't come back at all. Its like what I saw in Ubuntu but worst.

[pgj]

And if you switch back to Linux 6.4, everything gets back to normal?

[venoom27]

Everything works like it should when switching back to 6.4.

[venoom27]

So I tried installing Freebsd on a HP laptop model 15t-da000 with a RTL8723DE card and I am seeing the same packet lose on it as I am seeing with the broadcom-wl driver. I was used pkg install to install the wifibox.

[pgj]

I have updated net/wifibox-alpine to version 20230810 which now includes experimental support for Linux 6.4. At the port configuration phase, you can choose to use that version though the default still remains the LTS kernel (for a time being, at least).

[venoom27]

I downloaded it and tried it.

wifibox:~# uname -r
6.4.9-0-edge

And I am still seeing packet drops.

wifibox:~# ping www.google.com
PING www.google.com (142.251.111.105): 56 data bytes
64 bytes from 142.251.111.105: seq=0 ttl=57 time=21.235 ms
64 bytes from 142.251.111.105: seq=1 ttl=57 time=19.575 ms
64 bytes from 142.251.111.105: seq=2 ttl=57 time=19.627 ms
64 bytes from 142.251.111.105: seq=3 ttl=57 time=17.816 ms
64 bytes from 142.251.111.105: seq=4 ttl=57 time=18.958 ms
64 bytes from 142.251.111.105: seq=5 ttl=57 time=17.634 ms
64 bytes from 142.251.111.105: seq=6 ttl=57 time=17.302 ms
64 bytes from 142.251.111.105: seq=7 ttl=57 time=20.465 ms
64 bytes from 142.251.111.105: seq=50 ttl=57 time=2169.445 ms
64 bytes from 142.251.111.105: seq=51 ttl=57 time=1161.663 ms
64 bytes from 142.251.111.105: seq=52 ttl=57 time=158.196 ms
64 bytes from 142.251.111.105: seq=53 ttl=57 time=23.180 ms
--- www.google.com ping statistics ---
93 packets transmitted, 51 packets received, 45% packet loss
round-trip min/avg/max = 13.671/84.441/2169.445 ms

Thank you so much for your help in this. Do you have any ideas why I would still be dropping packets even with the 6.4 kernel?

[pgj]

There could be many reasons for that. For example, the kernel in Wifibox/Alpine is stripped down and it employs a specialized configuration which may not be optimal for this scenario. Also, the virtualization (PCI pass-through or the MSI) itself might not work properly with this type of cards. We would need to conduct several experiments to localize the source of problem. We could start with building a vanilla Arch Linux-based image, try it, check the results and proceed to the next iteration based on that, and so on.

[ierturk]

Hello, thanks for your work.

I tried your solution in a VoidLinux VM (vm-bhyve). It seesm that the BCM4331 won't work with the brcm drivers. It needs to be installed b43 or wl driver. Also it nees to be extracted the additional b43 firmwares by using fw-cutter. However, the kernel gives some error regarding the IRQs.

Is there any further update regarding regarding this problem?

Thank you.

[pgj]

Wifibox/Alpine already supports BCM4331 cards starting from version 20230728. That said, if you install the latest Wifibox release (1.3.0 as time of writing) it should just work if you select the XX_DRIVER_WL non-default option at the port's configuration.

The error about the IRQ for the wl happens because it is not aware of Message Signaled Interrupts (MSI) which is the only option to have drivers run with bhyve. This has also been patched for Wifibox/Alpine that is why it works there.

[ierturk]

I was trying the one which is insttaled by pkg. Now it works. Thank you.