Skip to content

Latest commit

 

History

History
1093 lines (872 loc) · 34.7 KB

mpcie_ethernet.md

File metadata and controls

1093 lines (872 loc) · 34.7 KB
Raw

Intro

Device under test is mPCIe extension card with double Ethernet interfaces exposed. Their MAC addresses are:

00:e0:4c:68:05:c8
00:e0:4c:68:05:c9

  • Platform is APU2C4

  • Firmware is coreboot v4.5.8 from PC engines fork.

  • Operating system is Voyage Linux. uname -a output:

Linux voyage 3.16.7-ckt9-voyage #1 SMP Wed Apr 22 23:04:57 HKT 2015 x86_64 GNU/Linux

mPCIe slot 1 slot

Device identification

Ethernet interfaces from extension card are correctly detected:

ip a

output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:e0:4c:68:05:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::2e0:4cff:fe68:5c8/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:e0:4c:68:05:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.120/24 brd 192.168.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::2e0:4cff:fe68:5c9/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bc brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20d:b9ff:fe43:3fbc/64 scope link
       valid_lft forever preferred_lft forever
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.108/24 brd 192.168.0.255 scope global eth3
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:b9ff:fe43:3fbd/64 scope link
       valid_lft forever preferred_lft forever
6: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0d:b9:43:3f:be brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.112/24 brd 192.168.0.255 scope global eth4
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:b9ff:fe43:3fbe/64 scope link
       valid_lft forever preferred_lft forever

output:

lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1566
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 156b
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.5 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:08.0 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1537
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 11)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [IDE mode] (rev 39)
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 39)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 42)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
00:14.7 SD Host controller: Advanced Micro Devices, Inc. [AMD] FCH SD Flash Controller (rev 01)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1580
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1581
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1582
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1583
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1584
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1585
01:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
04:00.0 PCI bridge: ASMedia Technology Inc. Device 1182
05:03.0 PCI bridge: ASMedia Technology Inc. Device 1182
05:07.0 PCI bridge: ASMedia Technology Inc. Device 1182
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ether)
07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411

Our extension card controllers are detected as follows:

lspci -v -s 06:00.00

output:

06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
        Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
        Flags: bus master, fast devsel, latency 0, IRQ 75
        I/O ports at 1000 [size=256]
        Memory at f7800000 (64-bit, non-prefetchable) [size=4K]
        Memory at f7600000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Kernel driver in use: r8169

lspci -v -s 07:00.00

output:

07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
        Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
        Flags: bus master, fast devsel, latency 0, IRQ 76
        I/O ports at 2000 [size=256]
        Memory at f7900000 (64-bit, non-prefetchable) [size=4K]
        Memory at f7700000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Kernel driver in use: r8169

Establishing connection

For the purpose of following test, automatic connection initialization was disabled by erasing /etc/network/interfaces file content. No network manager was present as well.

Available Ethernet interfaces list:

2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:e0:4c:68:05:c8 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:e0:4c:68:05:c9 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bc brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bd brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:be brd ff:ff:ff:ff:ff:ff
  • First interface (00:e0:4c:68:05:c9 MAC address) from extension card:
ifconfig eth1 up

output:

[  105.204628] r8169 0000:07:00.0: firmware: direct-loading firmware rtl_nic/rtl8168e-3.fw
[  105.313609] r8169 0000:07:00.0 eth1: link down
[  105.318155] r8169 0000:07:00.0 eth1: link down
[  105.318482] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[  107.209683] r8169 0000:07:00.0 eth1: link up
[  107.214014] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

dhclient eth1
ifconfig eth1

output:

eth1      Link encap:Ethernet  HWaddr 00:e0:4c:68:05:c9
          inet addr:192.168.0.120  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:4cff:fe68:5c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:22 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2945 (2.8 KiB)  TX bytes:2060 (2.0 KiB)

Pinging some internal or external addresses from APU2 works:

ping 192.168.0.1

output:

PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.374 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.266

ping google.pl

output:

PING google.pl (172.217.20.195) 56(84) bytes of data.
64 bytes from waw02s08-in-f195.1e100.net (172.217.20.195): icmp_seq=1 ttl=57 time=6.52 ms
64 bytes from waw02s08-in-f195.1e100.net (172.217.20.195): icmp_seq=2 ttl=57 time=6.48 ms

Pinging eth1 interface address from different PC works as well:

ping 192.168.0.120

output:

PING 192.168.0.120 (192.168.0.120) 56(84) bytes of data.
64 bytes from 192.168.0.120: icmp_seq=1 ttl=64 time=0.592 ms
64 bytes from 192.168.0.120: icmp_seq=2 ttl=64 time=0.482 ms
  • Second interface (00:e0:4c:68:05:c8 MAC address) from extension card:

Reboot was executed first.

ifconfig eth0 up

output:

[  218.000514] r8169 0000:06:00.0: firmware: direct-loading firmware rtl_nic/rtl8168e-3.fw
[  218.118256] r8169 0000:06:00.0 eth0: link down
[  218.122760] r8169 0000:06:00.0 eth0: link down
[  218.127347] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  220.014881] r8169 0000:06:00.0 eth0: link up
[  220.019250] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

dhclient  eth0
ifconfig eth0

output:

eth0      Link encap:Ethernet  HWaddr 00:e0:4c:68:05:c8
          inet addr:192.168.0.123  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:4cff:fe68:5c8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1240 (1.2 KiB)  TX bytes:1262 (1.2 KiB)

Pinging some internal or external addresses from APU2 works:

ping 192.168.0.1

output:

PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.391 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.233 ms

ping google.pl

output:

PING google.pl (172.217.20.195) 56(84) bytes of data.
64 bytes from waw02s08-in-f3.1e100.net (172.217.20.195): icmp_seq=1 ttl=57 time=6.59 ms
64 bytes from waw02s08-in-f3.1e100.net (172.217.20.195): icmp_seq=2 ttl=57 time=6.44 ms

Pinging eth0 interface address from different PC works as well:

PING 192.168.0.123 (192.168.0.123) 56(84) bytes of data.
64 bytes from 192.168.0.120: icmp_seq=1 ttl=64 time=0.592 ms
64 bytes from 192.168.0.120: icmp_seq=2 ttl=64 time=0.482 ms

Operating multiple interfaces at once

Following interfaces have been connected using separate Ethernet cables:

  • both interfaces from mPCIe extension card

  • two out of three interfaces from APU2 board

  1. Interfaces configuration
ifconfig eth0 up

output:

[  400.659117] r8169 0000:06:00.0: firmware: direct-loading firmware rtl_nic/rtl8168e-3.fw
[  400.762913] r8169 0000:06:00.0 eth0: link down
[  400.767438] r8169 0000:06:00.0 eth0: link down
[  400.772017] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  402.677778] r8169 0000:06:00.0 eth0: link up
[  402.682146] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

dhclient eth0
ifconfig eth0

output:

eth0      Link encap:Ethernet  HWaddr 00:e0:4c:68:05:c8
          inet addr:192.168.0.132  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:4cff:fe68:5c8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1480 (1.4 KiB)  TX bytes:1332 (1.3 KiB)

ifconfig eth1 up

output:

[  449.690659] r8169 0000:07:00.0 eth1: link down
[  449.690686] r8169 0000:07:00.0 eth1: link down
[  449.699870] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[  451.626151] r8169 0000:07:00.0 eth1: link up
[  451.630502] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

dhclient eth1
ifconfig eth1

output:

eth1      Link encap:Ethernet  HWaddr 00:e0:4c:68:05:c9
          inet addr:192.168.0.133  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:4cff:fe68:5c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1300 (1.2 KiB)  TX bytes:1262 (1.2 KiB)

ifconfig eth2 up

output:

[  484.573775] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[  487.225321] igb 0000:01:00.0 eth2: igb: eth2 NIC Link is Up 100 Mbps Full Duplex, FlowX
[  487.234971] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

dhclient eth2
ifconfig eth2

output:

eth2      Link encap:Ethernet  HWaddr 00:0d:b9:43:3f:bc
          inet addr:192.168.0.134  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20d:b9ff:fe43:3fbc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1720 (1.6 KiB)  TX bytes:1332 (1.3 KiB)
          Memory:f7a00000-f7a1ffff

ifconfig eth4 up

output:

[  528.424071] IPv6: ADDRCONF(NETDEV_UP): eth4: link is not ready
[  531.111624] igb 0000:03:00.0 eth4: igb: eth4 NIC Link is Up 100 Mbps Full Duplex, FlowX
[  531.121238] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready

dhclient eth4
ifconfig eth4

output:

eth4      Link encap:Ethernet  HWaddr 00:0d:b9:43:3f:be
          inet addr:192.168.0.135  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20d:b9ff:fe43:3fbe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2200 (2.1 KiB)  TX bytes:1332 (1.3 KiB)
          Memory:f7c00000-f7c1ffff
  1. Each of them can be separately reached from network
ping 192.168.0.132

output:

PING 192.168.0.132 (192.168.0.132) 56(84) bytes of data.
64 bytes from 192.168.0.132: icmp_seq=2 ttl=64 time=410 ms
64 bytes from 192.168.0.132: icmp_seq=3 ttl=64 time=551 ms

ping 192.168.0.133

output:

PING 192.168.0.133 (192.168.0.133) 56(84) bytes of data.
64 bytes from 192.168.0.133: icmp_seq=1 ttl=64 time=4.31 ms
64 bytes from 192.168.0.133: icmp_seq=2 ttl=64 time=1.49 ms

ping 192.168.0.134

output:

PING 192.168.0.134 (192.168.0.134) 56(84) bytes of data.
64 bytes from 192.168.0.134: icmp_seq=1 ttl=64 time=3.02 ms
64 bytes from 192.168.0.134: icmp_seq=2 ttl=64 time=1.71 ms

ping 192.168.0.135

output:

PING 192.168.0.135 (192.168.0.135) 56(84) bytes of data.
64 bytes from 192.168.0.135: icmp_seq=1 ttl=64 time=2.80 ms
64 bytes from 192.168.0.135: icmp_seq=2 ttl=64 time=1.71 ms

mPCIe slot 2

Device identification

Ethernet interfaces from extension card are not detected:

lspci

output:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1566
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 156b
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:08.0 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1537
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 11)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [IDE mode] (rev 39)
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 39)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 42)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
00:14.7 SD Host controller: Advanced Micro Devices, Inc. [AMD] FCH SD Flash Controller (rev 01)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1580
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1581
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1582
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1583
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1584
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1585
01:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

ip a

output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.135/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:b9ff:fe43:3fbc/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.108/24 brd 192.168.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:b9ff:fe43:3fbd/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0d:b9:43:3f:be brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.112/24 brd 192.168.0.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:b9ff:fe43:3fbe/64 scope link
       valid_lft forever preferred_lft forever

mPCIe slot 2 with clock always enabled

This is different from the previous one with the following change in source code:

diff --git a/src/mainboard/pcengines/apu2/romstage.c b/src/mainboard/pcengines/apu2/romstage.c
index a1c73d1e3424..f018ac5742ce 100644
--- a/src/mainboard/pcengines/apu2/romstage.c
+++ b/src/mainboard/pcengines/apu2/romstage.c
@@ -99,7 +99,9 @@ void cache_as_ram_main(unsigned long bist, unsigned long cpu_init_detectedx)
                data = *((u32 *)(ACPI_MMIO_BASE + MISC_BASE+FCH_MISC_REG04));

                data &= 0xFFFFFF0F;
-               data |= 0xA << (1 * 4); // CLKREQ GFX to GFXCLK
+               // make GFXCLK to ignore CLKREQ# input
+               // force it to be always on
+               data |= 0xF << (1 * 4); // CLKREQ GFX to GFXCLK

Device identification

ip a

output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bc brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bd brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:be brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:e0:4c:68:05:c8 brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:e0:4c:68:05:c9 brd ff:ff:ff:ff:ff:ff

lspci

output:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1566
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 156b
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:08.0 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1537
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 11)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [IDE mode] (rev 39)
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 39)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 42)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
00:14.7 SD Host controller: Advanced Micro Devices, Inc. [AMD] FCH SD Flash Controller (rev 01)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1580
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1581
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1582
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1583
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1584
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1585
01:00.0 PCI bridge: ASMedia Technology Inc. Device 1182
02:03.0 PCI bridge: ASMedia Technology Inc. Device 1182
02:07.0 PCI bridge: ASMedia Technology Inc. Device 1182
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ether)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ether)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

Our extension card controllers are detected as follows:

lspci -v -s 03:00.00

output:

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411
PCI Express Gigabit Ethernet Controller (rev 06)
        Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
        Flags: bus master, fast devsel, latency 0, IRQ 76
        I/O ports at 1000 [size=256]
        Memory at f7800000 (64-bit, non-prefetchable) [size=4K]
        Memory at f7600000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Kernel driver in use: r8169

lspci -v -s 04:00.00

output:

04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411
PCI Express Gigabit Ethernet Controller (rev 06)
        Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
        Flags: bus master, fast devsel, latency 0, IRQ 77
        I/O ports at 2000 [size=256]
        Memory at f7900000 (64-bit, non-prefetchable) [size=4K]
        Memory at f7700000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Kernel driver in use: r8169

Establishing connection

For the purpose of following test, automatic connection initialization was disabled by erasing /etc/network/interfaces file content. No network manager was present as well.

Available Ethernet interfaces list:

2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:e0:4c:68:05:c8 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:e0:4c:68:05:c9 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bc brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:bd brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0d:b9:43:3f:be brd ff:ff:ff:ff:ff:ff
  • First interface (00:e0:4c:68:05:c9 MAC address) from extension card:

Reboot was executed first.

ifconfig eth1 up

output:

[  702.047648] r8169 0000:04:00.0: firmware: direct-loading firmware rtl_nic/rtl8168e-3.fw
[  702.161935] r8169 0000:04:00.0 eth1: link down
[  702.166586] r8169 0000:04:00.0 eth1: link down
[  702.166817] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[  704.076523] r8169 0000:04:00.0 eth1: link up
[  704.080905] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

dhclient eth1
ifconfig eth1

output:

eth1      Link encap:Ethernet  HWaddr 00:e0:4c:68:05:c9
          inet addr:192.168.0.120  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:4cff:fe68:5c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1300 (1.2 KiB)  TX bytes:1332 (1.3 KiB)

Pinging some internal or external adresses from APU2 works:

ping 192.168.0.1

output:

PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.371 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.313 ms

ping google.pl

output:

PING google.pl (172.217.20.195) 56(84) bytes of data.
64 bytes from waw02s08-in-f195.1e100.net (172.217.20.195): icmp_seq=1 ttl=57 time=6.54 ms
64 bytes from waw02s08-in-f195.1e100.net (172.217.20.195): icmp_seq=2 ttl=57 time=6.46 ms

Pinging eth1 interface address from different PC works well:

ping 192.168.0.120

output:

PING 192.168.0.120 (192.168.0.120) 56(84) bytes of data.
64 bytes from 192.168.0.120: icmp_seq=1 ttl=64 time=0.755 ms
64 bytes from 192.168.0.120: icmp_seq=2 ttl=64 time=0.306 ms
  • Second interface (00:e0:4c:68:05:c8 MAC address) from extension card:

Reboot was executed first.

ifconfig eth0 up

output:

[   63.775838] r8169 0000:03:00.0: firmware: direct-loading firmware rtl_nic/rtl8168e-3.fw
[   63.880564] r8169 0000:03:00.0 eth0: link down
[   63.885105] r8169 0000:03:00.0 eth0: link down
[   63.885192] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   65.798066] r8169 0000:03:00.0 eth0: link up
[   65.802396] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

dhclient  eth0
ifconfig eth0

output:

eth0      Link encap:Ethernet  HWaddr 00:e0:4c:68:05:c8
          inet addr:192.168.0.123  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:4cff:fe68:5c8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1300 (1.2 KiB)  TX bytes:1332 (1.3 KiB)

Pinging some internal or external adresses from APU2 works:

ping 192.168.0.1

output:

PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.750 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.266 ms

ping google.pl

output:

PING google.pl (172.217.20.195) 56(84) bytes of data.
64 bytes from waw02s08-in-f195.1e100.net (172.217.20.195): icmp_seq=1 ttl=57 time=6.87 ms
64 bytes from waw02s08-in-f195.1e100.net (172.217.20.195): icmp_seq=2 ttl=57 time=6.75 ms

Pinging eth0 interface address from different PC works well:

ping 192.168.0.123

output:

PING 192.168.0.123 (192.168.0.123) 56(84) bytes of data.
64 bytes from 192.168.0.123: icmp_seq=1 ttl=64 time=0.800 ms
64 bytes from 192.168.0.123: icmp_seq=2 ttl=64 time=0.313 ms

Operating multiple interfaces at once

Following interfaces have been connected using separate Ethernet cables:

  • both interfaces from mPCIe extension card

  • two out of three interfaces from APU2 board

  1. Interfaces configuration
ifconfig eth0 up

output:

[  158.770166] r8169 0000:03:00.0: firmware: direct-loading firmware rtl_nic/rtl8168e-3.fw
[  158.877942] r8169 0000:03:00.0 eth0: link down
[  158.882483] r8169 0000:03:00.0 eth0: link down
[  158.882576] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  160.847493] r8169 0000:03:00.0 eth0: link up
[  160.851821] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

dhclient eth0
ifconfig eth0

output:

eth0      Link encap:Ethernet  HWaddr 00:e0:4c:68:05:c8
          inet addr:192.168.0.132  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:4cff:fe68:5c8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1600 (1.5 KiB)  TX bytes:1262 (1.2 KiB)

ifconfig eth1 up

output:

[  185.188050] r8169 0000:04:00.0 eth1: link down
[  185.188107] r8169 0000:04:00.0 eth1: link down
[  185.197387] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
root@voyage:~# dh[  187.135453] r8169 0000:04:00.0 eth1: link up
[  187.139809] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

client eth1
ifconfig eth1

output:

eth1      Link encap:Ethernet  HWaddr 00:e0:4c:68:05:c9
          inet addr:192.168.0.133  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:4cff:fe68:5c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1480 (1.4 KiB)  TX bytes:1262 (1.2 KiB)

ifconfig eth2 up

output:

[  602.484926] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[  605.172947] igb 0000:05:00.0 eth2: igb: eth2 NIC Link is Up 100 Mbps Full Duplex, FlowX
[  605.182617] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

ifconfig eth2

output:

eth2      Link encap:Ethernet  HWaddr 00:0d:b9:43:3f:bc
          inet addr:192.168.0.134  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20d:b9ff:fe43:3fbc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:66 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5675 (5.5 KiB)  TX bytes:1332 (1.3 KiB)
          Memory:f7a00000-f7a1ffff

ifconfig eth4 up

output:

[ 1038.772097] IPv6: ADDRCONF(NETDEV_UP): eth4: link is not ready
[ 1041.441250] igb 0000:07:00.0 eth4: igb: eth4 NIC Link is Up 100 Mbps Full Duplex, FlowX
[ 1041.450845] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready

dhclient eth4

output:

root@voyage:~# ifconfig eth4
eth4      Link encap:Ethernet  HWaddr 00:0d:b9:43:3f:be
          inet addr:192.168.0.135  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20d:b9ff:fe43:3fbe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:13 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1931 (1.8 KiB)  TX bytes:1332 (1.3 KiB)
          Memory:f7c00000-f7c1ffff
  1. Each of them can be individually reached from network
ping 192.168.0.132

output:

PING 192.168.0.132 (192.168.0.132) 56(84) bytes of data.
64 bytes from 192.168.0.132: icmp_seq=1 ttl=64 time=26.2 ms
64 bytes from 192.168.0.132: icmp_seq=2 ttl=64 time=1.96 ms

ping 192.168.0.133

output:

PING 192.168.0.133 (192.168.0.133) 56(84) bytes of data.
64 bytes from 192.168.0.133: icmp_seq=1 ttl=64 time=1.35 ms
64 bytes from 192.168.0.133: icmp_seq=2 ttl=64 time=1.97 ms

ping 192.168.0.134

PING 192.168.0.134 (192.168.0.134) 56(84) bytes of data.
64 bytes from 192.168.0.134: icmp_seq=1 ttl=64 time=1.42 ms
64 bytes from 192.168.0.134: icmp_seq=2 ttl=64 time=1.49 ms

ping 192.168.0.135

PING 192.168.0.135 (192.168.0.135) 56(84) bytes of data.
64 bytes from 192.168.0.135: icmp_seq=1 ttl=64 time=1.81 ms
64 bytes from 192.168.0.135: icmp_seq=2 ttl=64 time=1.38 ms

Conclusion

  • Ethernet interfaces from extension card inserted into mPCIe slot 1 operate properly in terms of device enumeration and DHCP lease.

  • Ethernet interfaces from extension card inserted into mPCIe slot 2 do not operate properly. They are not enumerated by kernel.

  • After forcing mPCIe clock to be always on, Ethernet interfaces from extension card inserted into mPCIe 2 slot operate in the same fashion as when inserted into mPCIe slot 1 (when it comes to device enumeration and DHCP lease, at least). Note that corresponding change exists for mPCIe slot 1.