Help wanted - eGPU bootloop issues

A random thing to try.
In the FW16 bios, there is a setting called “USB4 device measured boot”.
It does a lot more than the simple description.
Try turning that on or off, to see if it affects the DEG2 at all in Linux.

There might be a similar setting in the Tuxedo BIOS.

I did end up with some bad news. After a longer game session, Windows locked up too.. but, I’m a Linux support engineer, and know nothing about windows.

I’ll go into event logs later to see what happened.

That being said, apparently there are known issues with the 50 series cards and egpu over USBC (that I found out about.. AFTER I got this card)

Ideally, the framework Dev kit gets a price soon and this becomes a nothing issue for me… But while I have the deg2 I’d love to help debug if framework or amd want / need logs.

I can also file a bug on gitlab but not sure what the best project is for that. I also emailed minisforum… So far, silence.

Not to bother @Mario_Limonciello on a Sunday, but if there is an ideal place to report this, please let me know :slight_smile:

@knipp30
You don’t have to wait for the dev kit.
The community have done a 4i Oculink and a 8i Oculink that you could get now, until the usb 4 starts working.

Regarding quirks:

It seems that there are 3 quirks in drivers/thunderbolt/quirks.c not related to DisplayPort:

  • quirk_force_power_link
  • quirk_clx_disable
  • quirk_usb3_maximum_bandwidth

quirk_force_power_link forces power to a link controller in case a specific peripheral is connected, so it doesn’t seem like it could help.

quirk_usb3_maximum_bandwidth made me speculate, that maybe when a TB5 peripheral is connected, the host tries to match TB5/USB4v2 transfer rates and as it exceeds its physical capabilities, errors are produced. Therefore, I created a similar quirk for USB4 targeting PCIe tunneling and just in case limiting the bandwidth slightly below the actual USB4 capabilities of 3.6GB/s: patch
The quirk was applied as evidenced by the logs below, but the laptop still rebooted after connecting DEG2 with my 3090.

Apr 26 22:09:58 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0: running quirk_usb4_maximum_bandwidth [thunderbolt]
Apr 26 22:09:58 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0: applying USB4 bandwidth limits
Apr 26 22:09:58 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:1: skipping USB4 bandwidth limitting
Apr 26 22:09:58 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:2: skipping USB4 bandwidth limitting
Apr 26 22:09:58 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:3: skipping USB4 bandwidth limitting
Apr 26 22:09:58 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:4: skipping USB4 bandwidth limitting
Apr 26 22:09:58 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:5: USB4 maximum bandwidth limited to 28664 Mb/s
Apr 26 22:09:58 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:6: skipping USB4 bandwidth limitting
Apr 26 22:09:58 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:7: skipping USB4 bandwidth limitting

Next, I tried more aggressive matching of the bandwidth quirk to all ports and also enabled quirk_clx_disable that disables some advanced power management features on some older AMD chipsests: patch and quirk application logs:

Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0: running quirk_clx_disable [thunderbolt]
Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0: disabling CL states
Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0: running quirk_usb4_maximum_bandwidth [thunderbolt]
Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:1: USB4 maximum bandwidth limited to 28664 Mb/s
Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:2: USB4 maximum bandwidth limited to 28664 Mb/s
Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:3: USB4 maximum bandwidth limited to 28664 Mb/s
Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:4: USB4 maximum bandwidth limited to 28664 Mb/s
Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:5: USB4 maximum bandwidth limited to 28664 Mb/s
Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:6: USB4 maximum bandwidth limited to 28664 Mb/s
Apr 26 23:14:08 morgwai-extm2-toshiba1gb kernel: thunderbolt 0000:68:00.6: 0:7: USB4 maximum bandwidth limited to 28664 Mb/s

…but it also hasn’t changed anything :frowning:

1 Like

Yeah, the deg2 is 4i - I’ll consider that.

The good news? I played for about 3 hours on windows and no crashes.

Also, changing the thunderbolt settings in bios seem to make no difference

Ideally to debug it should be reproduced with a USB4 analyzer connected between the system and the eGPU. Minisforum should have these and be able to do that. They can then tell folks what exactly is going wrong.

Another potentially useful data point would be to look at lspci.exe verbose output for the whole topology in Windows and compare it to lspci output in Linux.

1 Like

I have emailed Minisforum support, quoted the above message and asked to forward it to their engineers. A few days ago I sent them the link to the kernel.org bug report and they did replied they would forward it to the engineering team, so there’s some chance it will not go to /dev/null@knipp30 if you did so also, it would probably increase the chances :wink:

Adding the output of “sudo lspci -vvv” and “sudo lspci -n” would help.

Hi! I’m not a Framework user(yet), but I’ve been using the DEG2 dock for about a month now with a 9070XT and I’ve had similar issues. The DEG2 froze up and, even under Windows, completely rebooted the system. These freezes were completely random, sometimes it worked flawlessly for a full day, other times it froze up 4 times a day.

As for Linux, I had zero success running anything for more than 5 minutes before a grey screen took over and then reset the OS.(Tried Bazzite and SteamOS)

All tests were done using the included(and unmarked) TB5 cable, so I’m planning on ordering a certified one to see if it’s any better. I slotted in my old GTX1060, and after a week, it’s been much more stable, only encountered a freeze once, so I’m thinking it’s the amount of data transmitted making it lock up.(Perhaps overheating the dock? CPU temps were monitored at all times and weren’t the cause.)

On Windows or Linux or both?

Also, what CPU do you currently have? (AMD/Intel, what model exactly)
Also the exact laptop model may be helpful.

Cheers! :slight_smile:

I’m currently in the process of installing Windows on the Tuxedo laptop again (deleted the previous install a few weeks ago) and will post both lspci and lspci.exe to the kernel.org bug report probably later today :slight_smile:

I am guessing that a possible cause might be an incompatibility between TB5 and USB4v1 (Essentially TB3). Incompatibility in the sense of maybe a driver bug. TB5 is relatively new tech.
I have a thunderbolt 3/USB4 device. An NVME SSD in an external enclosure.
When doing lspci -vv
I get the following PCIe related devices:
I think the interesting part is the PCIe bandwidth and lanes.
Either end of the USB4/Thunderbolt link is:
Speed 2.5GT/s, Width x1 ← Based on FIO, this looks about right, but I have been told that the real negotiated speed is not populated into “lspci -vvv output” when using USB4/Thunderbolt, so this is maybe bogus.

The end NVME device is (PCIe-gen4):
LnkSta: Speed 16GT/s, Width x4

Doing a data write test with fio, I get only:
1259993303 bytes / second. Or 1.2 GBytes/s. I would expect closer to 3 GBytes/s.

That is definitely not the 40Gbps that USB4 advertises in boltctl:
│ ├─ rx speed: 40 Gb/s = 2 lanes * 20 Gb/s (Gbits/s)
│ ├─ tx speed: 40 Gb/s = 2 lanes * 20 Gb/s

So, that is why seeing your “sudo lspci -vvv” and “sudo lspci -n” would be so useful.
There is also a possibility of mismatched “MaxPayload”, that the lspci output would help identify.

00:04.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel (prog-if 00 [Normal decode])
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin ? routed to IRQ 41
        IOMMU group: 5
        Bus: primary=00, secondary=62, subordinate=c0, sec-latency=0
        I/O behind bridge: 2000-5fff [size=16K] [16-bit]
        Memory behind bridge: 60000000-77ffffff [size=384M] [32-bit]
        Prefetchable memory behind bridge: 6800000000-77ffffffff [size=64G] [32-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Root Port (Slot+), IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+ TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
                        Slot #0, PowerLimit 0W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt- HPIrq+ LinkChg+
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCap: CRSVisible+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
                RootSta: PME ReqID 0021, PMEStatus+ PMEPending+
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- LN System CLS Not Supported, TPHComp+ ExtTPHComp- ARIFwd+
                         AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                         IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00000  Data: 0000
        Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
        Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP+ Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF-
                AERCap: First Error Pointer: 14, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn+ NFERptEn+ FERptEn+
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsgNum 0
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0021
        Capabilities: [270 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [400 v1] Data Link Feature <?>
        Kernel driver in use: pcieport
        Kernel modules: shpchp

62:00.0 PCI bridge: ASMedia Technology Inc. ASM2464PD USB4 Device Controller 40G (prog-if 00 [Normal decode])
        Subsystem: ASMedia Technology Inc. ASM2464PD USB4 Device Controller 40G
        Physical Slot: 0-1
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 133
        IOMMU group: 5
        Bus: primary=62, secondary=63, subordinate=64, sec-latency=0
        I/O behind bridge: 2000-5fff [size=16K] [16-bit]
        Memory behind bridge: 60000000-77ffffff [size=384M] [32-bit]
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v2) Upstream Port, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: Routing-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: EgressBlck-
                         IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [c0] Subsystem: ASMedia Technology Inc. ASM2464PD USB4 Device Controller 40G
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [160 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [1c0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [200 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=128us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Capabilities: [220 v1] Data Link Feature <?>
        Capabilities: [240 v1] Physical Layer 16.0 GT/s
                Phy16Sta: EquComplete- EquPhase1- EquPhase2- EquPhase3- LinkEquRequest-
        Capabilities: [280 v1] Lane Margining at the Receiver
                PortCap: Uses Driver-
                PortSta: MargReady- MargSoftReady-
        Kernel driver in use: pcieport
        Kernel modules: shpchp

63:00.0 PCI bridge: ASMedia Technology Inc. ASM2464PD USB4 Device Controller 40G (prog-if 00 [Normal decode])
        Subsystem: ASMedia Technology Inc. ASM2464PD USB4 Device Controller 40G
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 134
        IOMMU group: 5
        Bus: primary=63, secondary=64, subordinate=64, sec-latency=0
        I/O behind bridge: 2000-5fff [size=16K] [16-bit]
        Memory behind bridge: 60000000-77ffffff [size=384M] [32-bit]
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00000  Data: 0000
        Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v2) Downstream Port (Slot+), IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+ TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                        ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt+ AutBWInt+ FltModeDis-
                LnkSta: Speed 16GT/s, Width x4
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #0, PowerLimit 0W; Interlock- NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet+ LinkState+
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- ARIFwd-
                         AtomicOpsCap: Routing-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
                         AtomicOpsCtl: EgressBlck-
                         IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [c0] Subsystem: ASMedia Technology Inc. ASM2464PD USB4 Device Controller 40G
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [1c0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [200 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=128us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=128us LTR1.2_Threshold=144384ns
                L1SubCtl2: T_PwrOn=10us
        Capabilities: [220 v1] Data Link Feature <?>
        Capabilities: [240 v1] Physical Layer 16.0 GT/s
                Phy16Sta: EquComplete+ EquPhase1+ EquPhase2+ EquPhase3+ LinkEquRequest-
        Capabilities: [280 v1] Lane Margining at the Receiver
                PortCap: Uses Driver-
                PortSta: MargReady+ MargSoftReady+

64:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN850X NVMe SSD (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Sandisk Corp WD Black SN850X NVMe SSD
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 133
        IOMMU group: 5
        Region 0: Memory at 60000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [b0] MSI-X: Enable+ Count=65 Masked-
                Vector table: BAR=0 offset=00003000
                PBA: BAR=0 offset=00002000
        Capabilities: [c0] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend+
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch+ ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 16GT/s, Width x4
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
                        ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
                        PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CorrIntErr- HeaderOF-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [1b8 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [300 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [900 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+
                          PortCommonModeRestoreTime=32us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=144384ns
                L1SubCtl2: T_PwrOn=10us
        Capabilities: [910 v1] Data Link Feature <?>
        Capabilities: [920 v1] Lane Margining at the Receiver
                PortCap: Uses Driver+
                PortSta: MargReady+ MargSoftReady+
        Capabilities: [9c0 v1] Physical Layer 16.0 GT/s
                Phy16Sta: EquComplete+ EquPhase1+ EquPhase2+ EquPhase3+ LinkEquRequest-
        Kernel driver in use: nvme
        Kernel modules: nvme
1 Like

Will definitely do later today or tomorrow morning.

Thanks for your help and involvement! :slight_smile:

Windows only at the moment, I’m quite afraid to connect Nvidia to Linux due to past experiences… :smiley:

CPU is AMD Ryzen Z1 Extreme, which is effectively the Ryzen 7 7840U. James mentioned an USB4v1 incompatibility, which is definitely possible, as I’m observing the issues through USB4v1 aswell.

It is also difficult to find good USB4/Thunderbolt cables.
For example, the cables that come with my NVME enclosure generally hardly ever work and result in disconnections from the device. I have a lot of NVME enclosures.
It was only after I purchased about 4 different cables for various web sites, did I finally find a cable that worked reliably. My guess is that cable manufacturers are just lying about the quality of their cables, and the labels they put on them.
It is a shame that we don’t have access to the link training data the USB4 chips do. We could use that to help determine the quality of cables, but lack of datasheets / NDAs on the PCIe hardware prevents that.

I think a useful test would also be trying the DEG2 with an Intel CPU laptop with USB4 ports on Linux, and see if reboots happen or not?

Here is the lspci outputs from Fedora 44:

Windows output:

C:\Users\rknipp\Downloads\pciutils-3.5.5-win64\pciutils-3.5.5-win64>lspci.exe
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e8
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14ea
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ed
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ed
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14ea
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ee
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ee
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14ea
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ef
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14ea
00:04.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ef
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14ea
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14eb
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14eb
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14eb
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 71)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14f0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14f1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14f2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14f3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14f4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14f5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14f6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14f7
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1478 (rev 12)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1479 (rev 12)
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 7480 (rev c1)
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device ab30
04:00.0 Non-Volatile memory controller: Device 1987:5021 (rev 01)
05:00.0 Network controller: Intel Corporation Device 2725 (rev 1a)
06:00.0 Non-Volatile memory controller: Sandisk Corp Device 5017 (rev 01)
66:00.0 PCI bridge: Intel Corporation Device 5786 (rev 85)
67:00.0 PCI bridge: Intel Corporation Device 5786 (rev 85)
67:01.0 PCI bridge: Intel Corporation Device 5786 (rev 85)
67:02.0 PCI bridge: Intel Corporation Device 5786 (rev 85)
67:03.0 PCI bridge: Intel Corporation Device 5786 (rev 85)
68:00.0 VGA compatible controller: NVIDIA Corporation Device 2d04 (rev a1)
68:00.1 Audio device: NVIDIA Corporation Device 22eb (rev a1)
c5:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 15bf (rev c2)
c5:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 1640
c5:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 15c7
c5:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b9
c5:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15ba
c5:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Device 15e2 (rev 63)
c5:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Device 15e3
c6:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 14ec
c6:00.1 Signal processing controller: Advanced Micro Devices, Inc. [AMD] Device 1502
c7:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 14ec
c7:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15c0
c7:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15c1
c7:00.5 USB controller: Advanced Micro Devices, Inc. [AMD] Device 1668
c7:00.6 USB controller: Advanced Micro Devices, Inc. [AMD] Device 1669

@James3 for you as well

I wish I had an intel laptop to test with :confused:

It would be helpful if the “lspci.exe” output for Windows, also had the output of “lspci.exe -vv”
Similar to the Linux output in the pastebin you did.

My apologies there :slight_smile:
Windows lspci -vvv~~~00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] - Pastebin.com

This is lspci -vvv and lspci -n on windows.

Ill stay booted into windows for now if you need anything else

Well, the “lspci -vvv” is definitely different between Windows and Linux.
The speed and amount of PCIe lanes for the eGPU is different for a start.
Linux:

68:00.0 VGA compatible controller: NVIDIA Corporation GB206 [GeForce RTX 5060 Ti] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. Device 8a11

	LnkSta:	Speed 2.5GT/s (downgraded), Width x4 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

Vs
Windows:

68:00.0 VGA compatible controller: NVIDIA Corporation Device 2d04 (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. Device 8a11

	LnkSta:	Speed 16GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-