[RESPONDED] LAN module intermittently disconnecting from USB

Jan_Van_humbeek · November 20, 2023, 8:43am

Which OS (Operating System)?
Proxmox VE
Which release of your OS (Operating System / Windows 10, 11, Distribution of Linux)?
Proxmox VE 8.0-2
Which Framework laptop (11th, 12th or 13th generation Framework laptop) are you asking for support with?
11th Gen (i7-1165G7) stand-alone in CoolerMaster case

Hi all,

I have 2 11th gen mainboards, both i7-1165G7 that I have set up as a Proxmox cluster.
They both have the LAN module installed and are working fine when booting up.
Then every once in a while (could be every other day, daily, or even more frequent) the server stops showing up on the network (not detected in router, no IP). After a reboot I check syslog and it mentions that the LAN module disconnects from USB, and a bit later it is re-detected by the OS but it is not active as my router doesn’t detect it and it’s not given an IP.
The activity lights on the module do keep blinking though.

I initially thought it was just 1 mainboard or 1 LAN adapter as the 2nd system stayed online at first but then it also started showing the same issues late last week.
Both mainboards are updated to the latest BIOS, and I have tried plugging in the LAN module on the different USB-C ports, to no avail.
Has anyone else ever faced an issue like this before, and is there perhaps a workaround so I could have a script that detects when this happens and resets the networkstack or such? I’m pretty new to Proxmox and Linux in general, especially when it comes to hardware monitoring and activating networkdevices/restarting services so if anyone could advise, that would be greatly appreciated.

Jan_Van_humbeek · November 20, 2023, 9:37am

I just checked syslog again for the timings, and it’s really disconnecting & reconnecting within 1 second…

As an example from syslog:

Nov 09 05:26:28 pve-node01 kernel: r8152-cfgselector 2-2: USB disconnect, device number 3
Nov 09 05:26:29 pve-node01 kernel: usb 2-2: new SuperSpeed USB device number 4 using xhci_hcd
Nov 09 05:26:29 pve-node01 kernel: usb 2-2: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.04
Nov 09 05:26:29 pve-node01 kernel: usb 2-2: New USB device strings: Mfr=x, Product=x, SerialNumber=x
Nov 09 05:26:29 pve-node01 kernel: usb 2-2: Product: USB 10/100/1G/2.5G LAN
Nov 09 05:26:29 pve-node01 kernel: usb 2-2: Manufacturer: Realtek
Nov 09 05:26:29 pve-node01 kernel: usb 2-2: SerialNumber: xxxxxxxxxx
Nov 09 05:26:29 pve-node01 kernel: cdc_ncm 2-2:2.0: MAC-Address: xx:xx:xx:xx:xx:xx
Nov 09 05:26:29 pve-node01 kernel: cdc_ncm 2-2:2.0: setting rx_max = 16384
Nov 09 05:26:29 pve-node01 kernel: cdc_ncm 2-2:2.0: setting tx_max = 16384
Nov 09 05:26:29 pve-node01 kernel: cdc_ncm 2-2:2.0 eth0: register ‘cdc_ncm’ at usb-0000:00:0d.0-2, CDC NCM (NO ZLP), xx:xx:xx:xx:xx:xx
Nov 09 05:26:29 pve-node01 kernel: cdc_ncm 2-2:2.0 eth0: unregister ‘cdc_ncm’ usb-0000:00:0d.0-2, CDC NCM (NO ZLP)
Nov 09 05:26:29 pve-node01 kernel: r8152-cfgselector 2-2: reset SuperSpeed USB device number 4 using xhci_hcd
Nov 09 05:26:29 pve-node01 kernel: r8152 2-2:1.0: ram code speedup mode fail
Nov 09 05:26:29 pve-node01 kernel: r8152 2-2:1.0: load rtl8156b-2 v2 04/27/23 successfully
Nov 09 05:26:29 pve-node01 kernel: r8152 2-2:1.0 eth0: v1.12.13
Nov 09 05:26:29 pve-node01 kernel: r8152 2-2:1.0 enx9cbf0d00213f: renamed from eth0

Matt_Hartley · November 20, 2023, 5:57pm

I’ve seen this before, specifically on distros with a lot of activity - it’s not as it should be.

Test this please, live Fedora USB (for testing), playing YouTube and pinging something like yahoo for a few minutes. Wifi off. You should see the drop happen. If you do, it’s the card, then open a ticket linking to this thread for context.

Armon_Rouhani · November 23, 2023, 1:13am

I’ve been running into this issue for some time, good to know I’m not the only one!

jadam · November 24, 2023, 6:12pm

Me too

Matt_Hartley · November 27, 2023, 5:49pm

All, in dmesg or journalctl, we’re looking for:

cdc_ncm 2-2:2.0 eth0: unregister ‘cdc_ncm’ usb-0000:00:0d.0-2, CDC NCM (NO ZLP)

We can use grep to look for this:

dmesg | grep "CDC NCM (NO ZLP)"

If you have trouble, try narrowing it down:

dmesg | grep "ZLP"

If you do find CDC NCM (NO ZLP), please open a support ticket.

kumardamani · January 13, 2024, 1:42am

getting the same thing under Proxmox 8.

I actually did try a live-usb session with Ubuntu 22.04, but did not try playing Youtube, or pinging anything so maybe that is why I only saw:

[    2.596631] cdc_ncm 2-1:2.0 eth0: register 'cdc_ncm' at usb-0000:00:0d.0-1, CDC NCM (NO ZLP), xx:xx:...
[    2.596782] usbcore: registered new interface driver cdc_ncm

I guess I will have to re-retry with those actions, and report back

kumardamani · January 13, 2024, 6:02pm

ok I was not able to see those unregister messages in the live-usb environment .

Going through the logs in the live-usb, I noticed that it was missing any lines with r8152 which seems to be the driver that is loaded for this adapter on Proxmox.

Ubuntu:

...
[    2.547896] usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd
[    2.568496] usb 2-1: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.04
[    2.568499] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[    2.568501] usb 2-1: Product: USB 10/100/1G/2.5G LAN
[    2.568502] usb 2-1: Manufacturer: Realtek
[    2.568503] usb 2-1: SerialNumber: 4013000001
[    2.574550] usbcore: registered new interface driver cdc_ether
[    2.595928] hid-generic 0003:17EF:60EE.0004: input,hidraw2: USB HID v1.11 Keyboard [Lenovo TrackPoint Keyboard II] on usb-0000:00:14.0-3.2/input0
[    2.596350] cdc_ncm 2-1:2.0: MAC-Address: xx:xx:xx...
[    2.596353] cdc_ncm 2-1:2.0: setting rx_max = 16384
[    2.596389] cdc_ncm 2-1:2.0: setting tx_max = 16384
[    2.596631] cdc_ncm 2-1:2.0 eth0: register 'cdc_ncm' at usb-0000:00:0d.0-1, CDC NCM (NO ZLP), xx:xx...
[    2.596782] usbcore: registered new interface driver cdc_ncm
[    2.598254] cdc_ncm 2-1:2.0 enx9cbf0d000a43: renamed from eth0
...

where as Proxmox:

...
[    1.708638] usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd
[    1.729890] usb 2-1: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.04
[    1.729894] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[    1.729895] usb 2-1: Product: USB 10/100/1G/2.5G LAN
[    1.729896] usb 2-1: Manufacturer: Realtek
[    1.729897] usb 2-1: SerialNumber: 4013000001
[    1.734849] usbcore: registered new device driver r8152-cfgselector
[    1.864968] r8152-cfgselector 2-1: reset SuperSpeed USB device number 2 using xhci_hcd
[    1.908672] r8152 2-1:1.0: load rtl8156b-2 v3 10/20/23 successfully
[    1.945486] r8152 2-1:1.0 (unnamed net_device) (uninitialized): netif_napi_add_weight() called with weight 256
[    1.945501] fbcon: Taking over console
[    1.945847] r8152 2-1:1.0 eth0: v1.12.13
[    1.945868] usbcore: registered new interface driver r8152
[    1.948330] usbcore: registered new interface driver cdc_ether
[    1.949407] usbcore: registered new interface driver cdc_ncm
[    1.953043] r8152 2-1:1.0 enx9cbf0d000a43: renamed from eth0
...

Am I to take this mean that not using the r8152 is the desired behavior?

Additionally I noticed the following differences with ethtool:
Ubuntu:

Features for enx9cbf0d000a43:
rx-checksumming: off [fixed]
tx-checksumming: off
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: off [fixed]
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: off
	tx-scatter-gather: off [fixed]
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
	tx-tcp-segmentation: off [fixed]
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp-mangleid-segmentation: off [fixed]
	tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]

Proxmox:

Features for enx9cbf0d000a43:
rx-checksumming: on
tx-checksumming: on
	tx-checksum-ipv4: on
	tx-checksum-ip-generic: off [fixed]
	tx-checksum-ipv6: on
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp-mangleid-segmentation: off
	tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]

Matt_Hartley · January 19, 2024, 12:06am

I looked into the ticket. @Loell_Framework will be replying shortly. I do indeed, see the error I was looking for from the previous logs.

Alexander_Keusch · February 9, 2024, 1:46pm

I ran into the same issue using an 11th gen mainboard and the framework network adapter with Ubuntu 22.04.

The issue started slowly, seeing some network “lags” from time to time. After a longer period of time, the network connectivity was totally unusable. In the syslog I also could see messages of the usb device being disconnecting and reconnecting.

The root cause seems to be a bug in the Realtek r8152 Kernel driver, also used by the Framework network adapter: https://bugzilla.kernel.org/show_bug.cgi?id=198931

I’ve also applied the workaround described in the ticket:

edit /etc/default/grub to add the usbcore.quirks parameter, make sure to use the proper id “0bda:8156” for the specific chip used in the framework adapter

GRUB_CMDLINE_LINUX_DEFAULT="usbcore.quirks=0bda:8156:k"

apply the changes

sudo update-grub

reboot the system

With these changes applied, my system is running stable.

Matt_Hartley · February 12, 2024, 6:59pm

Hi @Alexander_Keusch, welcome to the community!

GRUB_CMDLINE_LINUX_DEFAULT="usbcore.quirks=0bda:8156:k"

This is used to disable auto-suspend which is helpful, but should have been fixed at this stage. May I ask what kernel you’re using for 22.04.3?

uname -r

Alexander_Keusch · February 12, 2024, 8:45pm

Hi @Matt_Hartley,

this is my kernel version:

$ uname -r
5.15.0-94-generic

Maybe I also should mention that my setup is pretty similar to the one described by Jan_Van_humbeek. My i7-1165G7 is running in a home server setup: CoolerMaster case, Ubuntu 22.04.3 Server. No display connected, no desktop environment installed, so also some tools installed with the Desktop version - like power-profiles-daemon or tlp - are missing.

I could not find any information about a fix for the 0bda:8156 2.5G chip used with the framework adapter. For the older 0bda:8153 1G chip the quirks mode now is enabled by default, while I cannot find the newer 0bda:8165 2.5G in this file:

github.com

torvalds/linux/blob/master/drivers/usb/core/quirks.c#L429


      
          	{ USB_DEVICE(0x0b05, 0x17e0), .driver_info =
          			USB_QUIRK_IGNORE_REMOTE_WAKEUP },
          
          	/* Realtek Semiconductor Corp. Mass Storage Device (Multicard Reader)*/
          	{ USB_DEVICE(0x0bda, 0x0151), .driver_info = USB_QUIRK_CONFIG_INTF_STRINGS },
          
          	/* Realtek hub in Dell WD19 (Type-C) */
          	{ USB_DEVICE(0x0bda, 0x0487), .driver_info = USB_QUIRK_NO_LPM },
          
          	/* Generic RTL8153 based ethernet adapters */
          	{ USB_DEVICE(0x0bda, 0x8153), .driver_info = USB_QUIRK_NO_LPM },
          
          	/* SONiX USB DEVICE Touchpad */
          	{ USB_DEVICE(0x0c45, 0x7056), .driver_info =
          			USB_QUIRK_IGNORE_REMOTE_WAKEUP },
          
          	/* Action Semiconductor flash disk */
          	{ USB_DEVICE(0x10d6, 0x2200), .driver_info =
          			USB_QUIRK_STRING_FETCH_255 },
          
          	/* novation SoundControl XL */

Matt_Hartley · February 12, 2024, 10:49pm

Ah, okay, so it’s with an untested lts kernel. We test against the OEM C/D series kernels per our docs.

This is likely something that was patched a long time ago. So even though it’s an lts kernel, it’s not what we test against.

Please continue using the kernel you feel works best for you as this quirk is addressing the issue found in this older lts kernel.

Eric_Brown · August 29, 2024, 8:55pm

This fixed my problems with Proxmox 8.2.4, incidentally.

Alex_Haydock · September 30, 2024, 9:15pm

Are you still looking for examples of this? I can reproduce this with the Framework 2.5G LAN adapter, as well as two others. Using a Framework 11th Gen and I can run tests if needed.

I’ve seen this on a few different kernels but haven’t kept track but most recently I’ve seen it on 6.11.0-63 from Fedora 41 so I ought to be about as update as can possibly be with regard to upstream kernel patches, if this is a bug which was patched any significant time ago.

Edit: They’re not all Realtek chipsets either, so I’m not convinced this is all down to Realtek drivers. One is an ASIX AX88179A.