[RESPONDED] Framework 13 USB 4 PCIe Lanes

Framework Laptop 13 Linux
1

I was trying to figure out why one of my eGPUs worked with my Framework 13 Intel i7-1370p but doesn’t work with my new Framework 13 AMD 7840U, and noticed that I’m only getting an x1 link on AMD while I’m getting and x4 link on Intel. This is what I’m seeing in dmesg:

[  134.193344] pci 0000:62:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:04.1 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)

lspci for the thunderbolt tunnel also shows an x1 link:

00:04.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel (prog-if 00 [Normal decode])
        Subsystem: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel
                ...
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                ...

Are the USB ports on the AMD Framework 13’s only 1-lane ports instead of 4 like on the Intels? If they are in fact 4-lane, is there some trick to make them run at full speed?

Note that this happens with a few Thunderbolt devices I have, not just with an eGPU.

Edit: I’m running an updated Arch install with the latest kernel:

Linux grazier 6.6.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 08 Nov 2023 16:05:38 +0000 x86_64 GNU/Linux
2

We don’t test eGPUs per se, however can you tell me which port you are connecting to?

4800×3600 181 KB

3

I tested both ports 1 and 3. I know the bottom two aren’t USB4.

4

It looks like this might be the issue here: [Linux 6.5.7] External GPU not working on AMD ThinkPad Z16 laptop via USB4 (#2925) · Issues · drm / amd · GitLab

Also the link speed showing as a single x1 PCIe gen 2 link might be a red herring: https://gitlab.freedesktop.org/drm/amd/-/issues/2925#note_2147083

We’ll see if it’s fixed in 6.7…

1 Like
5

I guess I should have mentioned, the eGPU that “doesn’t work” does connect and shows up in lspci, it’s just the amdgpu driver that gives an error. I have two though, one works and one doesn’t.

I have a RX7800 in an Akitio Node Titan, which is the one that “doesn’t work”, and an RX6800 in a Razer Core X, which does, both however report x1 at 2.5GT/s; I thought maybe the newer card doesn’t like running with only that much bandwidth so it might be the problem. It’s not important enough for me to start swapping enclosures and whatnot to try and get it working, so I’ll just wait for this fix to go through which hopefully fixes it.

6

This should fix performance issues.

https://lore.kernel.org/amd-gfx/20231110223452.13439-1-mario.limonciello@amd.com/

The messaging from PCI core is still under discussion.

1 Like
7

Thanks, I saw that in the thread I had posted. It’s not important enough right now for me to run a patched kernel so I’m just waiting for it to (hopefully) show up in kernel 6.7.

1 Like
8

Might not be relevant to your problem, but you might give this solution a shot in order to force amdgpu driver to use x3 instead of x1 speeds. I had a similar issue but on 12th gen Intel and this fixed it.

9

Did you get this fixed? I have a 7640U (running NixOS, so Linux as well) and I’d like to get an eGPU to be able to get rid of my desktop possibly.

10

I can get pretty much full bandwidth on a usb4 ssd so I suppose it probably is. Th reporting as 1 lane thing was still there last time I checked but it doesn’t actually effect performance.

1 Like
11

Thanks a lot!

12

Hi all. I’m using an eGPU (NVIDIA 2070 Super in a Razer Core X) connected by a TB3 cable to a Ryzen 5 7640. I’m running Ubuntu 22.04. The connection also appears to be bottlenecked by the 00:04.1 PCI Bridge.

00:04.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ef (prog-if 00 [Normal decode])
...
LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
...

I’ve connected to both ports 1 & 3 (the USB4 capable ones), with the same issue on both (PCIE bridge 00:03.1 on Port 3). I saw that this reported value might not be accurate, so I tested it with the cuda samples bandwidthTest.

[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: NVIDIA GeForce RTX 2070 SUPER
 Quick Mode

 Host to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(GB/s)
   32000000			2.6

 Device to Host Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(GB/s)
   32000000			2.9

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(GB/s)
   32000000			379.2

Result = PASS

This suggests to me that it is using a faster PCIe protocol than the lspci reported 1.0, but is still limited in its number of lanes. Is there a way to increase the number of lanes used by the ports?

Insights would be appreciated, and I’m happy to provide any logs that might clarify things. I’m fairly new to this subject and may have misinterpreted things.

13

PCIe bandwidth with a Thunderbolt 3 enclosure like the Razer Core X is expected to be limited to 22 Gbps because of the limitations of the original Thunderbolt chip in the enclosure. egpu.io has an extensive list of expected host to device transfer speeds here and your tested speeds look exactly in line with what is expected from this enclosure.

As was said above the reported speed of 2.5x1 is a red herring and not actually impacting or reflecting the actual speed of the connection. The AMD USB4 controller is capable of fully saturating the 40 Gbps USB4/Thunderbolt bandwidth with PCIe traffic, but the bottleneck is the Thunderbolt chip in the enclosure. An enclosure that uses the ASM2464 USB4 chip like the ADT-link UT3G and a GPU that supports PCIe 4.0 should allow for maximum bandwidth expected to be around 3.8 to 3.9 GB/s

4 Likes
14

Ah, thank you. That makes sense. I was a bit thrown off by every game running at around 20fps without high CPU nor eGPU usage (~30% ish for Rocket League) and assumed it was a data-transfer bottleneck issue. But I checked the external display (connected to eGPU) without any games running and all other displays disabled and it was reporting ~23fps with glxgears. So something else is the problem there; maybe the display itself is being powered by the iGPU? Either way, not a PCIe Lane problem so I’ll move these questions to a more appropriate forum. Thanks for your help!

1 Like
15

Are you sure you’re actually using the external GPU for rendering? In general, this doesn’t happen automatically, even if that’s the display it’s connected to.

Now, I don’t personally connect displays to the eGPU, however I’ll just describe what I have and do which I hope helps:

Setup:

  1. Computer: Various Framework laptops (both AMD and Intel), among other machines like 12/13/14th gen Intel/ASUS NUCs
  2. GPU Enclosure: AKiTiO Node Titan and a Razer Core X (non-RGB version)
  3. Graphics Card: GIGABYTE Radeon RX 7800 XT GAMING OC 16G (in the AKiTiO) and a PowerColor Fighter AMD Radeon RX 6800 16G (in the Razer)
  4. Linux Distro: Arch
  5. Desktop Environment: GNOME

By default, GNOME will not use the dedicated GPU for anything, and will generally be asleep. There’s a flag in some .desktop files, such as steam, that tell GNOME to use the dedicated GPU (if available) when launching those applications (I think switcheroo is required for this which I talk about next…), which it will then use the dedicated gpu for rendering and then do some sort of framebuffer copying to display this on the integrated gpu/display.

The other way of using the dedicated gpu is to use something like switcherooctl, which you can use to choose which gpu you want to use for rendering. I have this installed and have the switcheroo service enabled which allows me to either manually launch an application using the deciated gpu from the command line, and also allows me to right click any application within GNOME and tell it to run on the dedicated GPU.

Now, all switcheroo is doing is setting some environment variables such as DRI_PRIME among others to make the opengl and vulkan libraries to use (or at least prever) a specific card for rendering. So you can just set these environment variables yourself, but you’ll have to figure out which ones apply to you and set them either globally (I don’t think this is recommended) or when you want to run an application using the dedicated gpu. It’s probably easier to use something like switcheroo, however.