Curious Thunderbolt 3 eGPU link speed case (Linux)

Framework Laptop 16 Linux
1

I’m facing an issue that was already mentioned on this forum, but discussions are scattered across multiple threads, out of which none treats link speed directly. I also found some mentions of such cases in the Internet, but without clear conclusions.

I’m using Arch Linux with 6.13.1 kernel. Given that (as I mentioned) it seems to be more widespread problem I decided to post on the main forum.

Let’s start from the very root of the problem - I’ve bought AKiTiO Node Titan eGPU case. Upon connection it refuses to negotiate 40Gb/s linkspeed, at least according to boltctl:

 ● AKiTiO Node Titan
   ├─ type:          peripheral
   ├─ name:          Node Titan
   ├─ vendor:        AKiTiO
   ├─ uuid:          -
   ├─ generation:    Thunderbolt 3
   ├─ status:        authorized
   │  ├─ domain:     -
   │  ├─ rx speed:   20 Gb/s = 2 lanes * 10 Gb/s
   │  ├─ tx speed:   20 Gb/s = 2 lanes * 10 Gb/s
   │  └─ authflags:  none
   ├─ authorized:    Sun 02 Feb 2025 12:04:50 PM UTC
   ├─ connected:     Sun 02 Feb 2025 12:04:50 PM UTC
   └─ stored:        Sat 01 Feb 2025 12:22:32 PM UTC
      ├─ policy:     iommu
      └─ key:        no

This is a well battle tested eGPU with known controller:

62:00.0 PCI bridge: Intel Corporation JHL7440 Thunderbolt 3 Bridge [Titan Ridge DD 2018]

First thing that has drawn my attention is kernel complaining about limited PCI bandwidth due to Thunderbolt bridge running in PCIe 1x mode, however I don’t believe this is the culprit, as nvidia drivers and lspci reports full speed on the card itself (I also see similar logs on different system which negotiates full speed, but more on that later).

So I started experimenting, things I tried so far (with a bit of context):

  1. different cables - I have two TB3 certified cables, one 70cm (27.5 inch) and another 50cm (20 inch) - as cables longer than 1m may actually downgrade the linkspeed to 20Gb/s,
  2. different kernels - because of rolling upgrade distro limitations I tried 6.12 and 6.13, I didn’t go as far as compiling older revisions,
  3. attaching original PSU - as I normally power my laptop via dock integrated with my monitor and it’s rated for 100W only, I made sure to try original 180W PSU
  4. mangling with UEFI settings - there’s no much, but I tried to disable “PCIE Link Power Management” (which apparently affects link parameters when run on battery) and adaptive battery, in case it was triggering some link power management logic
  5. attaching the eGPU to another, Intel system - it negotiated 40GB/sek (2 lanes, 20Gb/s each) without slightest issues
  6. disconnecting battery and letting the system discharge - just to be super sure that hardware didn’t get into some ephemeral, broken state
  7. however it didn’t seem to be connected with linkspeed in any way, I also tried disabling AMD IOMMU as it was reported broken some time ago
  8. both of USB4 capable ports - I also tried replacing the expansion adapters to rule out fault on their side

None of the above helped.

Thinks I still consider trying, but only if I don’t find any other low hanging fruits (or kids and wife let me):

  1. compile kernel 6.8, as I’ve seen reports of folks having full speed on those while they also experienced issues with older revisions
  2. getting and installing Windows 11 to check how it behaves
  3. downgrading UEFI to 3.03, as I run the newest version atm
  4. pray to the gods of hardware so they automagically fix the problem overnight for me

That said, for now I’m stuck and lack of ideas. I may also try to enable / build kernel with more logging from around Thunderbolt and see if it reports something valuable.

Does anyone experience something similar / rootcaused the problem or maybe even found the solution?

2

That seems to be just a reporting issue. The amd pcie tunnel device reports as pcie1x1 but has way more effective bandwidth than that.

The JHL7440 in my TH4G3 seem to do effectively much over 20gbit on my amd fw13 (but nowhere near as close to 40gbit as the asm controller does) but I don’t recall what it reports in boltctl.

3

Hi,
I have an asm controller for a nvme enclosure.
I get similar pcie 1x and pcie warning, but using a disk speed test, it is transferring at expected max bandwidth of about 3 GBytes/sec.
So, the pcie 1x and warnings seem bogus.

4

Assuming you mean GB/s, still you should get close to 4 (3.7+) with a pcie4 ssd on an asm controller and amd host. 2.8-3GB/s is well within the range I get with even first gen intel controllers.

Jup, my ass ran after that red herring for a while last year when it turned out I just need a multi threaded load to get the full bandwidth XD

5

Thanks for the answers folks. I believe it’s your post that I’ve found about this being red herring, thus my remark that I don’t really believe this is the case here. Especially that speeds on the device itself are reported correctly and if there were inherited from the bridge then they should’ve been capped.

I just tried certified Thunderbolt 4 cables (80cm/31.5inch), as I’ve seen report that it helped to someone from this community, but no luck in my case. I’m having a feeling that this may be somehow driver related.

I’m going to try (1) another distro from liveusb, (2) once I find a bit more time - Windows 11.

If not that then my attention is going to shift to hardware and I’ll be contacting the customer support.

6

Oh I’ve found something interesting.

I’m editing the whole post, as I was looking at it wrongly. I mixed the domains in tbdump command, here’s what’s really happening:

# tbdump -d 1 -r 0 -a 2 -vv -N2 LANE_ADP_CS_0 
0x0080 0x003c01c0 0b00000000 00111100 00000001 11000000 .<.. LANE_ADP_CS_0  
  [00:07]       0xc0 Next Capability Pointer
  [08:15]        0x1 Capability ID
  [16:19]        0xc Supported Link Speeds
  [20:21]        0x3 Supported Link Widths (SLW)
  [22:23]        0x0 Gen 4 Asymmetric Support (G4AS)
  [26:26]        0x0 CL0s Support
  [27:27]        0x0 CL1 Support
  [28:28]        0x0 CL2 Support
0x0081 0x4828003c 0b01001000 00101000 00000000 00111100 H(.< LANE_ADP_CS_1  
  [00:03]        0xc Target Link Speed → Router shall attempt Gen 3 speed
  [04:05]        0x3 Target Link Width → Establish a Symmetric Link
  [06:07]        0x0 Target Asymmetric Link → Establish Symmetric Link
  [10:10]        0x0 CL0s Enable
  [11:11]        0x0 CL1 Enable
  [12:12]        0x0 CL2 Enable
  [14:14]        0x0 Lane Disable (LD)
  [15:15]        0x0 Lane Bonding (LB)
  [16:19]        0x8 Current Link Speed → Gen 2
  [20:25]        0x2 Negotiated Link Width → Symmetric Link (x2)
  [26:29]        0x2 Adapter State → CL0
  [30:30]        0x1 PM Secondary (PMS)

So it does fall back to Gen2 speed, even though Gen3 is supported (both sides). I’m stuck now.

From what I know it usually means that the cable is too long, but there’s simply no way for that. I’ve tried USB4 cable, two certified TB3 cables and certified TB4 cable. Apparently it may also be a case of too long routes from the port to the controller (in that case CPU as far as I understand), but come on, it works for others… Unless this is faulty hardware.

Ok, different distro test goes as next.

7

I’ve tried Ubuntu LTS 24.04, running 6.8 kernel, same result - only 20 Gb/s link speed. I may need to experiment with Windows as next, but I have a feeling that it won’t bring any difference.

I didn’t find anything fishy in dyndbg thunderbolt kernel driver logs.

I also triple checked that it runs full 40Gb/s on ThinkPad 14s (Intel system) on the cables I use - it does.

8

I just tried the 16in with a thunderbolt dock and a usb4 dongle (HyperDrive Dual Monitor USB4 Mobile Dock"

Both connect at 40Gb/s based on the output of boltctl

 ● ANKER PowerExpand Elite 13-in-1 Thunderbolt 3 Dock
   ├─ type:          peripheral
   ├─ name:          PowerExpand Elite 13-in-1 Thunderbolt 3 Dock
   ├─ vendor:        ANKER
   ├─ uuid:          c7010000-0082-840e-834a-b1c828539106
   ├─ generation:    Thunderbolt 3
   ├─ status:        authorized
   │  ├─ domain:     ffce3804-8052-90e7-ffff-ffffffffffff
   │  ├─ rx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  ├─ tx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  └─ authflags:  none
   ├─ authorized:    Wed 26 Feb 2025 07:29:48 PM UTC
   ├─ connected:     Wed 26 Feb 2025 07:29:48 PM UTC
   └─ stored:        Sun 01 Sep 2024 05:12:03 AM UTC
      ├─ policy:     iommu
      └─ key:        no

 ● generic vendor generic model
   ├─ type:          peripheral
   ├─ name:          generic model
   ├─ vendor:        generic vendor
   ├─ uuid:          30080921-00b0-6635-ffff-ffffffffffff
   ├─ generation:    USB4
   ├─ status:        authorized
   │  ├─ domain:     ffce3804-8152-90e7-ffff-ffffffffffff
   │  ├─ rx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  ├─ tx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  └─ authflags:  none
   ├─ authorized:    Thu 27 Feb 2025 10:09:57 AM UTC
   ├─ connected:     Thu 27 Feb 2025 10:09:57 AM UTC
   └─ stored:        Thu 20 Feb 2025 07:15:18 PM UTC
      ├─ policy:     iommu
      └─ key:        no
9

On a FW16: Slot / Port 1
I use the Sonnect 10G SFP to get a 10Gbps Ethernet port on my FW16.

boltctl reports: 2x 20Gbps == 40Gbps
lspci reports: LnkSta: Speed 8GT/s, Width x4 == 32Gbps.
Note: Where does tbdump come from? I don’t have that on my distro - Ubuntu.

I think there is overhead caused by the USB4 bit, so the 32Gbps seems to be a more realistic figure of what speed is actually usable over the link.

boltctl:

 ● Sonnet Technologies, Inc Solo 10G SFP+ Thunderbolt 3 Edition
   ├─ type:          peripheral
   ├─ name:          Solo 10G SFP+ Thunderbolt 3 Edition
   ├─ vendor:        Sonnet Technologies, Inc
   ├─ uuid:          cd010000-0090-8708-2323-a285c853e001
   ├─ generation:    Thunderbolt 3
   ├─ status:        authorized
   │  ├─ domain:     4c243804-d1f1-5294-ffff-ffffffffffff
   │  ├─ rx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  ├─ tx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  └─ authflags:  none
   ├─ authorized:    Thu 27 Feb 2025 11:43:47 UTC
   ├─ connected:     Thu 27 Feb 2025 11:43:47 UTC
   └─ stored:        Mon 04 Nov 2024 15:36:45 UTC
      ├─ policy:     iommu
      └─ key:        no

When is do lspci -vvv I get:

64:00.0 Ethernet controller: Aquantia Corp. AQtion AQC100S NBase-T/IEEE 802.3an Ethernet Controller [Atlantic 10G] (rev 02)
        Subsystem: Sonnet Technologies, Inc. Device 72e5
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 118
        IOMMU group: 5
        Region 0: Memory at 60440000 (64-bit, non-prefetchable) [size=64K]
        Region 2: Memory at 60450000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at 60000000 (64-bit, non-prefetchable) [size=4M]
        Expansion ROM at 60400000 [virtual] [disabled] [size=256K]
        Capabilities: [40] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x4
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink+ Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00000200
        Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] Vital Product Data
                Not readable
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [180 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: atlantic
10

Thanks a ton for your reply folks! Yeah I suspect I may have a hardware issue of some sort.

I contacted support and my case got escalated to engineering 2 weeks ago. Since then no response whatsoever. I’m a bit disappointed, but still waiting.

11

Holy moly! I just connected my eGPU after a week (I was on a business trip) and voila:

 ● AKiTiO Node Titan
   ├─ type:          peripheral
   ├─ name:          Node Titan
   ├─ vendor:        AKiTiO
   ├─ uuid:          c7010000-0092-9c1e-8352-9482c8018a01
   ├─ generation:    Thunderbolt 3
   ├─ status:        authorized
   │  ├─ domain:     a8a63804-61ec-9004-ffff-ffffffffffff
   │  ├─ rx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  ├─ tx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  └─ authflags:  none
   ├─ authorized:    Mon 03 Mar 2025 05:34:31 PM UTC
   ├─ connected:     Mon 03 Mar 2025 05:34:31 PM UTC
   └─ stored:        Tue 04 Feb 2025 10:08:33 PM UTC
      ├─ policy:     iommu
      └─ key:        no

The only thing I changed is the location of USB-A module. Gonna try some plugging/unplugging/module shuffle later on.

12

It was one-time off 40Gb/s negotiation. I can’t reproduce it anymore, stuck with 20Gb/s as before. So sadly, the problem remains. Seems like RMA candidate.

13

Another update - tomorrow I’ll receive a replacement mainboard, gonna report about the result.

The whole RMA process was super smooth, maybe apart of ridiculous courier service - it took 7 (!) working days to get the package from Netherlands to Poland, it’s absurd. Mediocare service would’ve deliver it within 3 days easily.

14

Ah, my apology, I didn’t spot your question. I installed it from Aur (Arch Linux), but also compiled on Fedora laptop for needs of testing, available here: GitHub - intel/tbtools: Thunderbolt/USB4 debugging tools

15

32gbps is the theoretical maximum any TB3/TB4 eGPU is able to deliver for plain connectivity to the GPU as the jhl7440 only connects pcie gen3x4 to the pcie Slot, BUT the Thunderbolt Protocoll requires to spare 10gbps for additional USB3 Lanes over the Connection for an “Dock”
So the practical Maximum on an tb3/4 eGPU is 22gbps. Thunderbolt also reserves a 50/50 Split for Data In and Out.

If you want more Datatransfer to your GPU or a full 40gbps effectivly saturating the usb4 connection you need an eGPU Adapter with the ASM2464PD Chip, there is only the ADT-Link UT3G on the Market, it provides pcie Gen4x4 to the pcie socket (theoretical 64gps) which is connected via usb4 pcie tunnel to the AMD Laptop and thereby throttled to the full 40gbps. I use this setup for my RX9070XT on my FW16 and get within 10% of the performance the Card should have. The only Downside of the UT3G is, that it is no able to do Powerdelivery, so you need a charger besides the Dock

And yes i tested the System also on Linux (Nobara 41) and it is confirmed the full Bus, but the new GFX Card works better with windows.

20405-grafik
20405-grafik592×975 208 KB

20397-Screenshot-2025-03-11-172745
20397-Screenshot-2025-03-11-1727451920×1186 203 KB

20403-Screenshot-2025-03-12-192232
20403-Screenshot-2025-03-12-192232832×201 86.9 KB

1741714290429
17417142904291707×1280 391 KB

1 Like
16

So motherboard replacement didn’t help. I pushed me towards further experimentation and I have few interesting findings.

Starting from further exploration of Thunderbolt registers - I found out that it is not recognizing the cable as Thunderbolt Gen3/4.

0x00d2 0x00000400 0b00000000 00000000 00000100 00000000 .... PORT_CS_18     
  [00:07]        0x0 Cable USB4 Version
  [08:08]        0x0 Bonding Enabled (BE)
  [09:09]        0x0 TBT3-Compatible Mode (TCM)
  [10:10]        0x1 CLx Protocol Support (CPS)
  [11:11]        0x0 RS-FEC Enabled (Gen 2) (RE2)
  [12:12]        0x0 RS-FEC Enabled (Gen 3) (RE3)
  [13:13]        0x0 Router Detected (RD)
  [16:16]        0x0 Wake on Connect Status
  [17:17]        0x0 Wake on Disconnect Status
  [18:18]        0x0 Wake on USB4 Wake Status
  [19:19]        0x0 Wake on Inter-Domain Status
  [20:20]        0x0 Cable Gen 3 Support (CG3)              <---- here...
  [21:21]        0x0 Cable Gen 4 Support (CG4)              <---- ...and here
  [22:22]        0x0 Cable Asymmetric Support (CSA)
  [23:23]        0x0 Cable CLx Support (CSC)
  [24:24]        0x0 AsymmetricTransitionInProgress (TIP)

Above values were set to 1 on ThinkPad.

So I bought USB-C cable analyzer to confirm that e-marker on my cables is presenting them as proper TBT3/4 cables with 40Gbps capabilities. It’s confirmed beyond any doubts.

Then I attached logic analyzer into CC line to check what’s actually happening there.
Comparing the output of ThinkPad T14 and Framework 16 clearly see that the first discovers cable Thunderbolt capabilities, while Framework doesn’t - since it’s unaware what cable is attached to the eGPU it defaults to 20Gbps.

I intend to do a more elaborate writeup about the problem, but first I’d like to get a response from Framework support team. Also cc @Mario_Limonciello , as far as I know this is AMD firmware what negotiates the capabilities, maybe you could be of help here as well?

Edit: if I was about to guess I’d bet that people who are getting full 40Gbps negotiated are using enclosures that don’t provide power and the fact that I have the issue is somehow related to power negotiation. I must emphasize again that this is just a guess.

1 Like
17

It is worth sharing this patch:

It helps with an issue that sideband traffic causes problems with enumeration in Linux. If that doesn’t help
I would be most suspicious of the being PD controller bug from what you have shared.

You should bring your traces to GitHub · Where software is built and hopefully Framework can work with PD controller vendor on them.

1 Like
18

Lovely, thanks. Gonna try the path and follow your suggestions.

19

Sadly, no luck. I have open support case, going to push it there further + ticket github as well. Thanks!

20

So apparently the issue got reproduced and kinda confirmed on the Framework side with suspicion that it may be firmware related. Sadly the resolution may take “quite some time”, what doesn’t make me overly optimistic about the ETA.

I mentioned “kinda confirmed” as from what I’ve been told it’s not as consistent as mine or it was manifesting in other ways. A bit cryptic, but that’s understandable.

I intend to continue my investigation on this and keep the github case updated. I don’t know if the fix lays within my capabilities, but maybe [at least] I’ll be able to provide more data points or even find a way to make it negotiate full 40Gbps - there’s still option to fiddle with CC transmission or listen what’s happening on i2c / configuration of the PD controller. Any other ideas are mostly welcome.