Details about USB, Thunderbolt and dock operation

[joevt]

@Rupert_Woods

Qestion, if I have a Thunderbolt 4 dock and a Thunderbolt 3 eGPU does the Thunderbolt bandwidth get divided between each of the controllers (each side of the motherboard) or only if multiple Thunderbolt connected to the same controller (same side)?

In M1, all Thunderbolt ports are a separate Thunderbolt controller. In Alpine Ridge, Titan Ridge, Maple Ridge, Ice Lake, and Tiger Lake, a Thunderbolt controller has two ports (usually two per side of a laptop in the case of Ice Lake and Tiger Lake). Each Thunderbolt controller has two DisplayPort In Adapters, so in Ice Lake and Tiger Lake, you can only connect two displays per side.

With integrated Thunderbolt implementations such as M1, Ice Lake, and Tiger Lake, I believe there’s no division of PCIe bandwidth because the upstream of the Thunderbolt controller isn’t really PCIe. So you can software RAID-0 two Thunderbolt NVMe together and get the same forty-something Gbps bandwidth from any two Thunderbolt ports but you can’t get 22Gbps from all ports at the same time (at least in the case of Ice Lake - I haven’t seen benchmarks for M1 or Tiger Lake).

In your case, Instead of two Thunderbolt NVMe, you want to connect a dock and an eGPU. Like I said above, you won’t see a difference no matter which two ports you select. If you look at Device Manager in Windows and view by Connection Type (or use HWiNFO), you’ll see that each Thunderbolt port is a separate PCIe root port (but it’s not really PCIe so you can ignore the ridiculous/inaccurate/meaningless PCIe 1.0 2.5 GT/s link speed).

@Ray519

But if you are using a Dock (presumably to attach monitors) and eGPU at the same time, the eGPU will have to transfer every frame through PCIe tunneled through TB back to the CPU/iGPU which will then output those frames via DP through the other TB controller.

So no matter where you attach the dock, you further limit the already limited PCIe bandwidth between eGPU and CPU, by not using the display outputs of a GPU.

You’ll definitely get better performance if the display you are outputting to is connected directly to the GPU in the eGPU enclosure. See the eGPU.io website for benchmarks and info.

@Ray519

@all can anybody confirm whether USB3 devices on Tiger Lakes USB4 ports appear on the system-wide USB3 controller or on the TB controllers own USB3 controller just like with Maple Ridge? (with USBTreeView for example)

All eGPUs are Thunderbolt 3 (Alpine Ridge or Titan Ridge) so their USB is controlled by their USB controllers which will use tunnelled PCIe. I don’t think I’ve seen a tunnelled USB on Tiger Lake (using a Thunderbolt 4 dock/hub) so a screenshot of Device Manager or HWiNFO or USBTreeView would be interesting.

@Matthijs_Kooijman

As an overall observation while reading these replies, I realized that bandwidth and output number limitations with all this tunneling really mostly come from the hardware interfaces present on the controllers/routers used (e.g. dual 4xHBR2 interfaces on Alpine Ridge, dual 4xHBR3 interfaces on Titan Ridge). These limitations are mostly linked to the hardware used, not so much limited by the protocol (though the protocols do seem to typically specify minimum bandwidth/features for each port).

The dual HBR3 problem with Titan Ridge is similar to the dual HBR2 problem with Falcon Ridge (Thunderbolt 2). Alpine Ridge is easy since it can only do HBR2 and two HBR2 can’t exceed 40 Gbps. But if you connect with a 20 Gbps cable, then you have the Falcon Ridge problem.

@Matthijs_Kooijman

I’m not sure I follow this. Where does this 22Gbps limit come from? It sounds like all USB goes through PCIe, so is this the total PCIe bandwidth?

It’s a limit of PCIe data through Thunderbolt. 22Gbps is one of the earliest numbers given by Intel in their marketing materials. It is 2750 MB/s which is a number often seen on the product pages for Thunderbolt NVMe storage devices. This is sometimes rounded up to 2800 MB/s. There have been benchmarks that reach 3000 MB/s (24 Gbps) and sometimes exceed that so the max is something less than 25 Gbps. The difference between 22 and 25 Gbps may be a setting in the firmware of the Thunderbolt peripheral. For example, eGPU enclosures had unexpected lower H2D or D2H bandwidth (host to device or device to host) benchmarks as measured by CL!ng.app in macOS or AIDA64 GPGPU Benchmark or CUDA-Z in Windows. A firmware update of the eGPU enclosure would often correct this. But in the USB4 spec there’s mention of a tradeoff between bandwidth and latency in regards to PCIe and USB tunnelling. So maybe the eGPUs with the lower bandwidth were tuned more for latency?

@Matthijs_Kooijman

But I thought that as only 16Gbps for TB3?

@Matthijs_Kooijman

Isn’t this also a property of the protocol? I.e. if you connect a TB3 dock to a TB4 port, you’d be limited to 16Gbps PCIe whatever your connect downstream? Or can TB3 also happily tunnel 32Gbps (or 22/24/25 as you say), but it’s just that TB3 controllers currently (and probably forever) have just 16Gbps (PCIe 3.0 x2?) uplink to the CPU?

A Thunderbolt 3 host controller (Alpine Ridge or Titan Ridge) has a max upstream link of PCIe 3.0 x4 (31.5 Gbps) same as Maple Ridge (Ice Lake and Tiger Lake don’t have this limit since they are integrated Thunderbolt controllers). The host controller is a PCIe device, so really, you can connect it to any PCIe connection all the way down to PCIe 1.0 x1 (2 Gbps). Some PC laptops used PCIe 3.0 x2 (15.75 Gbps) (some Mac laptops too) or PCIe 2.0 x4 (16 Gbps) which saves power or lanes. Regardless of the PCIe limit, the Thunderbolt controller can still do 34.56Gbps of DisplayPort if it has two DisplayPort connections to the GPU so it’s not a complete waste.

@Matthijs_Kooijman

Also, why this 8/4Gbps limits?

There’s two Fresco Logic FL1100 USB controllers which each use a PCIe 2.0 x1 connection (5 GT/s * 8b/10b = 4 Gbps PCIe, or 5 Gbps * 8b/10b = 4 Gbps USB). The ASMedia ASM1142 USB controller uses a PCIe 3.0 x1 connection (8 GT/s * 128b/130b = 7.877 Gbps). The USB controller in the Alpine Ridge Thunderbolt controller is not limited by PCIe (10 Gbps * 128b/132b = 9.7 Gbps USB). Actually, in regards to the FL1100, although both PCIe and USB are limited to 4 Gbps, the extra PCIe overhead compared to USB overhead makes it slower than USB 3.0 from the ASM1142 or Alpine Ridge. In regards to the differences between 5 and 10 Gbps USB, the 10 Gbps USB is more than twice as fast because it uses a more efficient encoding.

@Matthijs_Kooijman

Does Alpine Ridge have HW interface for 4 lanes of PCIe, one at 8Gbps and three at 4Gbps or so?

Alpine Ridge and Titan Ridge in a Thunderbolt peripheral have 4 PCIe 3.0 downstream lanes. They can be divided by the manufacture as x1x1x1, x2x1x1, x2x2, or x4. The TS3+ uses x1x1x1x1 and the fourth PCIe lane to the Alpine Ridge Thunderbolt controller is used by an Intel Gigabit Ethernet controller. A PCIe device can connect using any link rate up to the max supported by the PCIe device and the root port or downstream bridge device it is connected to. You can change the link rate while they are connected (using pciutils) so you can simulate a lower speed device if you like. On a Mac Pro 2008, a PCIe 3.0 device installed in a PCIe 2.0 slot will boot as PCIe 1.0 link rate but you can set it to PCIe 2.0 using pciutils.

@Matthijs_Kooijman

Wow, when I thought this couldn’t get more complex in terms of options you have, then there’s this :-p Makes sense, though I’m not sure where and if at all I should put this in my post…

Maybe there are situations where having more USB controllers is beneficial. It depends on the capabilities of the USB controller that would be used for USB tunnelling compared to the USB controller in the downstream Thunderbolt peripheral.

@Matthijs_Kooijman

This is rather unexpected, but I guess it might be because TB3 evolved from TB1/2 over a DP connector without USB 2.0 wires.

I think it’s just following the USB-C Alt Mode spec.

USB-C

The Thunderbolt 3 device doesn’t do USB so it shouldn’t try to do USB except to provide a USB 2.0 billboard device in case the USB-C port can’t switch from USB mode to Thunderbolt alt mode. Windows will see the billboard device and display a message. The same may exist in USB-C to DisplayPort adapters and cables.

@Matthijs_Kooijman

Doesn’t TB support routing DP data from the eGPU directly to a display attached to the same dock (or even somewhere in the same TB tree), without going through the host? I cannot find anything definitive on this, but skimming the USB4 spec I do see that docks/hubs can contain DP IN adapters (which would allow attaching the eGPU DP output to the TB network), and I cannot find any explicit restriction that all tunnels should either start or end at the host (though all examples I can find do involve the host)…

The only eGPUs that use the DisplayPort In Adapter of their Thunderbolt controller are the Blackmagic eGPUs and the Sonnet eGPU Breakaway Puck RX 5500 XT/5700. I don’t know if these work with Windows. I don’t know if the DisplayPort from the eGPU can be tunnelled upstream or only downstream (I guess it’s a decision made by Apple’s software Thunderbolt connection manager). I don’t know if the DisplayPort from the host Thunderbolt controller can be tunnelled downstream of the eGPU such that there are 4 tunnelled DisplayPort connections downstream of the eGPU.

I do know one DisplayPort tunnelling trick where the DisplayPort from one host is tunnelled to the DisplayPort of another host’s Thunderbolt controller. This is called Thunderbolt Target Display Mode which allows an old Thunderbolt iMac (1440p) to be a display for another Mac. So it seems you can have software that pokes some arbitrary DisplayPort paths even across domains. The iMac though has a switch to move its display connection from its GPU to a DisplayPort Out Adapter of its Thunderbolt controller so it can receive the tunnelled DisplayPort. I don’t know if the DisplayPort tunnelling path could extend deeper into the iMac’s Thunderbolt domain.

[Ray519]

@joevt

All eGPUs are Thunderbolt 3 (Alpine Ridge or Titan Ridge) so their USB is controlled by their USB controllers which will use tunnelled PCIe. I don’t think I’ve seen a tunnelled USB on Tiger Lake

Not what I was asking, I was already aware of that. And Tiger Lake should very much support tunneling of USB3 through USB4, as that is precisely what the USB4 spec demands. What I meant was the following: Maple, Alpine and Titan Ridge host controllers are their own USB3-PCIe controllers. If you directly attach a USB3 device to one of those, it appears on the TB controllers internal USB3 controller. But Titan Ridge and Maple Ridge forward USB2 signals. (See all Titan Ridge AICs having an additional USB2 input connected to the motherboard). Consequently USB2 devices attached to my Titan Ridge notebook and my Maple Ridge desktop appear not on the TB controllers own USB controller, as they did with Alpine Ridge or my Titan Ridge dock, but on the chipsets controller. When using USB4 hubs on Maple Ridge, nothing changes, USB2 devices appear an the chipset controller, USB3 devices on the TB controller. Only my Alpine Ridge AIC uses its own USB3 controller also for USB2 devices.

My question was, due to Tiger Lake having on-die TB controllers whether USB2 and 3 are handled just as with Maple Ridge, in that every TB controller has its own USB3 controller, plus the chipset having its own (3rd) hard wired USB3 controller on the same die. Is USB2 still forwarded to the chipsets’ own controller?. This would give an indication, of how closely related the implementations actually are, because I am assuming that the reason for still having dual-port TB controllers is also simply because that is what they were going to use for the dedicated controllers and so they can share more HW and just strip away the physical PCIe parts, because backwards compatibility and configurable lanes are not needed on-die.
Then I would presume, that although the uplink might not be limited to 32 GBit/s, the TB controller itself is most likely still somewhat PCIe bandwidth limited, because it never had a need to achieve much higher than 32 GBit/s throughput. Apple showcases, that if you design a TB controller only for on-die purposes, there is no need in doing dual-port controllers, because every subcomponent is either per-port anyway, or can be combined into one component for the whole die, or whatever amount is still feasible.

@joevt

so a screenshot of Device Manager or HWiNFO or USBTreeView would be interesting

Is that directed at me?

Yellow is Maple Ridge USB-Controller, Green is Alder Lake Chipset USB-Controller.
Blue is a USB2 stick, Magenta a USB3 stick.
Both sticks are attached the USB3-A ports of the CalDigit ElementHub behind Maple Ridge in the screenshot (physical ports right next to each other). The cursor highlight shows that USBTreeView detects the paired USB2 and USB3 ports of the CalDigit Hub, even though the billboard device only shows up on USB2.
As you can see, the USB2 stick shows up on the chipset controller in both cases (I am only allowed one picture per post, so the other one showing both attached to the chipset via front USB3-A ports is missing).
I speculated previously, that this is at least in part to achieve the energy efficient waking up from keyboard and mouse etc. as the TB controller needs no active PCIe connection to forward USB2 data to the chipset and the existing wake-on-USB HW inside the chipset can handle it in basically the same way as without TB.

@joevt

The Thunderbolt 3 device doesn’t do USB so it shouldn’t try to do USB except to provide a USB 2.0 billboard device

When connecting my WD19TB to a simple USB3 port, a USB2 device behind the dock shows up on the USB2 hierarchy of the chipset and not the USB3 hierarchy, so presumably Titan Ridge can still forward USB2 signals in back-compat mode, just not use them in TB3 mode. As to why USB2 lines remain unused in TB3, I do not know. But since it is optional for TB4 fiber cables to even support USB2, it might have just been to simplify the different possible configurations with different types of TB cables. Or like Matthiijs speculated, due to TB’s history.

[joevt]

@Ray519

because I am assuming that the reason for still having dual-port TB controllers is also simply because that is what they were going to use for the dedicated controllers and so they can share more HW and just strip away the physical PCIe parts, because backwards compatibility and configurable lanes are not needed on-die.

I think the dual port decision was based more on the number of available DisplayPort connections in relation to the number of Thunderbolt ports. The Tiger Lake GPU only has 4 possible DisplayPort connections. The simplest method of allowing two DisplayPort connections for a single Thunderbolt port (a requirement of Thunderbolt 4) when there are four Thunderbolt ports is to pair two Thunderbolt ports to a controller which has two of the DisplayPort connections. Then the controller can output the two DisplayPort connections to the same port (to meet the Thunderbolt 4 requirement) or to different ports. Intel would have needed to add more complexity to get 4 DisplayPort connections to a single side of the laptop. Apple’s Mac Pro 2019 has a lot of DisplayPort muxes to get DisplayPort from the two MPX module slots (4 DisplayPort connections per slot) to the two built-in Thunderbolt controllers (two DisplayPort connections each).

M1 Macs are weird - For example, the Mac Studio can have 8 DisplayPort connections, two per Thunderbolt port, but can only connect 4 displays (up to two displays per Thunderbolt port to meet the Thunderbolt 4 requirement) - the extra DisplayPort connections can only be used for tiled displays like the LG UltraFine 5K or Dell UP2715K. I don’t know if there’s a 9th DisplayPort connection for the HDMI port.

@Ray519

Then I would presume, that although the uplink might not be limited to 32 GBit/s, the TB controller itself is most likely still somewhat PCIe bandwidth limited, because it never had a need to achieve much higher than 32 GBit/s throughput.

There is a limit but it’s not exactly PCIe since none of the upstream is real PCIe. There’s the tunnelled PCIe limit and there’s whatever limit that exists in the Tiger Lake CPU. The TB controller needs to allow the PCIe bandwidth of at least one port (22-25Gbps). Ice Lake can do around 38 Gbps from all ports
https://egpu.io/forums/laptop-computing/ice-lake-cpu-on-die-thunderbolt-3-controller-bandwidth/paged/2/
(I think ATTO multi-disk non-raid test would be better than AJA in macOS, or AmorphousDiskMark for software RAID 0)

@Ray519

Yellow is Maple Ridge USB-Controller, Green is Alder Lake Chipset USB-Controller.
Blue is a USB2 stick, Magenta a USB3 stick.
Both sticks are attached the USB3-A ports of the CalDigit ElementHub behind Maple Ridge in the screenshot (physical ports right next to each other). The cursor highlight shows that USBTreeView detects the paired USB2 and USB3 ports of the CalDigit Hub, even though the billboard device only shows up on USB2.
As you can see, the USB2 stick shows up on the chipset controller in both cases

For a Tiger Lake laptop, the following link shows PCI device:
https://egpu.io/forums/builds/2021-framework-11th4cg-2x-3090-2070-1080ti-32gbps-2x-razer-core-x-agb-3090-agb-2070-win10-21h1/

• There’s a separate root port for each Thunderbolt port. Since the Thunderbolt controller is integrated, the root ports don’t have an upstream PCIe device that is limited to 8 GT/s x4.
• There’s a USB 3.x controller specifically for the Thunderbolt ports but the question about USB 2.0 for Tiger Lake is not answered here.
• I wonder how fast a software raid of 4 USB NVMe devices can get when connected to the Tiger Lake Thunderbolt USB controller?
• The Thunderbolt NHI #0 and #1 are the two Thunderbolt controllers responsible for setting up paths for PCIe and DisplayPort tunnelling and for doing Thunderbolt networking (using DMA tunnelling).
  An image from Device Manager or HWiNFO would be best because they include both PCIe and USB devices (and other devices). ioreg or IORegistryExplorer.app in macOS are similar but also include drivers (so does Device Manager but not in the connection view).

@Ray519

When connecting my WD19TB to a simple USB3 port, a USB2 device behind the dock shows up on the USB2 hierarchy of the chipset and not the USB3 hierarchy, so presumably Titan Ridge can still forward USB2 signals in back-compat mode, just not use them in TB3 mode.

Right, the Thunderbolt 3 device I was referring to uses Alpine Ridge. Titan Ridge (like Goshen Ridge) peripherals usually can connect as USB 3.x with DisplayPort Alt Mode devices to non-Thunderbolt USB-C ports. And with USB 3.x there is usually always also a USB 2.x connection and this connection may be made with a different USB controller than the the Thunderbolt USB controller that is used for the USB 3.x connection.

[turns2stone]

Can someone advise how/if we can use the Framework laptop with a Thunderbolt display? I’ve had good luck using my DIY Framework with a variety of Thunderbolt docks and eGPU. But has anyone use it with a Thunderbolt Display, like Apple’s Pro Display XDR?

I have this display, but can’t get any video output from any of the Frameworks USB-C ports. Normally, DisplayPort needs to be tunneled into the Thunderbolt output/stream, but I’m unclear on how to make this happen, as it doesn’t seem automatic.

I used a different Windows Thunderbolt PC with this same display (4K60Hz output, as expected), but I can’t get the Framework to recognize it.

[Matthijs_Kooijman]

@turns2stone

But has anyone use it with a Thunderbolt Display, like Apple’s Pro Display XDR?

I haven’t got much hands-on experience, nor suggestions for debugging tools yet (haven’t received my framework yet), but maybe others have some ideas. However, might I suggest to create a new topic for this (or maybe the mods could move your post to a new topic), to keep this topic centered around the technology involved, rather that a specific display and debugging advice?

[Matthijs_Kooijman]

While debugging Laptop powers *on* after hibernate when unplugging/unpowering dock - #2 by Azure, I found that the Linux kernel can provide quite a bit of detail on the ports/interfaces/adapters exposed by a USB4/TB4 router (like a dock) if you enable debug printing.

i.e. run:

echo "module thunderbolt +p" > /sys/kernel/debug/dynamic_debug/control

Then plug in the dock and check dmesg for debug output.

For example, with my Caldigit TS4, this gives:

[37305.688348] thunderbolt 0000:00:0d.2: current switch config:
[37305.688357] thunderbolt 0000:00:0d.2:  USB4 Switch: 8087:b26 (Revision: 3, TB Version: 32)
[37305.688364] thunderbolt 0000:00:0d.2:   Max Port Number: 19
[37305.688367] thunderbolt 0000:00:0d.2:   Config:
[37305.688369] thunderbolt 0000:00:0d.2:    Upstream Port Number: 1 Depth: 1 Route String: 0x1 Enabled: 1, PlugEventsDelay: 254ms
[37305.688375] thunderbolt 0000:00:0d.2:    unknown1: 0x0 unknown4: 0x0
[37305.747299] thunderbolt 0000:00:0d.2: 1: reading drom (length: 0x9e)
[37305.764321] thunderbolt 0000:00:0d.2: 1: DROM version: 1
[37305.792245] thunderbolt 0000:00:0d.2: 1: uid: 0x8087a68941184600
[37305.818400] thunderbolt 0000:00:0d.2:  Port 1: 8086:b26 (Revision: 3, TB Version: 1, Type: Port (0x1))
[37305.818413] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[37305.818418] thunderbolt 0000:00:0d.2:   Max counters: 16
[37305.818422] thunderbolt 0000:00:0d.2:   NFC Credits: 0xc7800000
[37305.818425] thunderbolt 0000:00:0d.2:   Credits (total/control): 120/2
[37305.849770] thunderbolt 0000:00:0d.2:  Port 2: 8086:b26 (Revision: 3, TB Version: 1, Type: Port (0x1))
[37305.849783] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[37305.849787] thunderbolt 0000:00:0d.2:   Max counters: 16
[37305.849790] thunderbolt 0000:00:0d.2:   NFC Credits: 0x80000000
[37305.849795] thunderbolt 0000:00:0d.2:   Credits (total/control): 0/2
[37305.875913] thunderbolt 0000:00:0d.2:  Port 3: 8086:b26 (Revision: 3, TB Version: 1, Type: Port (0x1))
[37305.875929] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[37305.875933] thunderbolt 0000:00:0d.2:   Max counters: 16
[37305.875936] thunderbolt 0000:00:0d.2:   NFC Credits: 0x83c00000
[37305.875940] thunderbolt 0000:00:0d.2:   Credits (total/control): 60/2
[37305.907278] thunderbolt 0000:00:0d.2:  Port 4: 8086:b26 (Revision: 3, TB Version: 1, Type: Port (0x1))
[37305.907286] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[37305.907289] thunderbolt 0000:00:0d.2:   Max counters: 16
[37305.907291] thunderbolt 0000:00:0d.2:   NFC Credits: 0x83c00000
[37305.907294] thunderbolt 0000:00:0d.2:   Credits (total/control): 60/2
[37305.907298] thunderbolt 0000:00:0d.2: 1:5: disabled by eeprom
[37305.907301] thunderbolt 0000:00:0d.2: 1:6: disabled by eeprom
[37305.919796] pcieport 0000:00:07.0: pciehp: Slot(0): Card present
[37305.919805] pcieport 0000:00:07.0: pciehp: Slot(0): Link Up
[37305.933423] thunderbolt 0000:00:0d.2:  Port 7: 8086:b26 (Revision: 3, TB Version: 1, Type: Port (0x1))
[37305.933435] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[37305.933440] thunderbolt 0000:00:0d.2:   Max counters: 16
[37305.933443] thunderbolt 0000:00:0d.2:   NFC Credits: 0x83c00000
[37305.933446] thunderbolt 0000:00:0d.2:   Credits (total/control): 60/2
[37305.964804] thunderbolt 0000:00:0d.2:  Port 8: 8086:b26 (Revision: 3, TB Version: 1, Type: Port (0x1))
[37305.964818] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[37305.964823] thunderbolt 0000:00:0d.2:   Max counters: 16
[37305.964826] thunderbolt 0000:00:0d.2:   NFC Credits: 0x83c00000
[37305.964829] thunderbolt 0000:00:0d.2:   Credits (total/control): 60/2
[37305.970033] thunderbolt 0000:00:0d.2:  Port 9: 8086:b26 (Revision: 3, TB Version: 1, Type: PCIe (0x100102))
[37305.970044] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[37305.970049] thunderbolt 0000:00:0d.2:   Max counters: 2
[37305.970052] thunderbolt 0000:00:0d.2:   NFC Credits: 0x40800000
[37305.970056] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[37305.975274] thunderbolt 0000:00:0d.2:  Port 10: 8086:b26 (Revision: 3, TB Version: 1, Type: PCIe (0x100101))
[37305.975282] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[37305.975285] thunderbolt 0000:00:0d.2:   Max counters: 2
[37305.975288] thunderbolt 0000:00:0d.2:   NFC Credits: 0x40800000
[37305.975291] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[37305.975295] thunderbolt 0000:00:0d.2: 1:b: disabled by eeprom
[37305.980508] thunderbolt 0000:00:0d.2:  Port 12: 8086:b26 (Revision: 3, TB Version: 1, Type: PCIe (0x100101))
[37305.980523] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[37305.980528] thunderbolt 0000:00:0d.2:   Max counters: 2
[37305.980531] thunderbolt 0000:00:0d.2:   NFC Credits: 0x40800000
[37305.980534] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[37305.990960] thunderbolt 0000:00:0d.2:  Port 13: 8086:b26 (Revision: 1, TB Version: 1, Type: DP/HDMI (0xe0102))
[37305.990970] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 9/9
[37305.990974] thunderbolt 0000:00:0d.2:   Max counters: 2
[37305.990977] thunderbolt 0000:00:0d.2:   NFC Credits: 0x40800000
[37305.990980] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[37306.035751] thunderbolt 0000:00:0d.2:  Port 14: 8086:b26 (Revision: 1, TB Version: 1, Type: DP/HDMI (0xe0102))
[37306.035766] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 9/9
[37306.035771] thunderbolt 0000:00:0d.2:   Max counters: 2
[37306.035775] thunderbolt 0000:00:0d.2:   NFC Credits: 0x800000
[37306.035779] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[37306.051922] thunderbolt 0000:00:0d.2:  Port 15: 8086:b26 (Revision: 1, TB Version: 1, Type: Inactive (0x0))
[37306.051938] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 0/0
[37306.051943] thunderbolt 0000:00:0d.2:   Max counters: 0
[37306.051946] thunderbolt 0000:00:0d.2:   NFC Credits: 0x0
[37306.051950] thunderbolt 0000:00:0d.2:   Credits (total/control): 0/0
[37306.057162] thunderbolt 0000:00:0d.2:  Port 16: 8086:b26 (Revision: 3, TB Version: 1, Type: USB (0x200102))
[37306.057172] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[37306.057176] thunderbolt 0000:00:0d.2:   Max counters: 2
[37306.057179] thunderbolt 0000:00:0d.2:   NFC Credits: 0x800000
[37306.057182] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[37306.057422] pci 0000:02:00.0: [8086:0b26] type 01 class 0x060400
[37306.057528] pci 0000:02:00.0: enabling Extended Tags
[37306.057783] pci 0000:02:00.0: supports D1 D2
[37306.057787] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[37306.057996] pci 0000:02:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x4 link at 0000:00:07.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[37306.059200] pci 0000:02:00.0: Adding to iommu group 18
[37306.059427] pcieport 0000:00:07.0: ASPM: current common clock configuration is inconsistent, reconfiguring
[37306.062402] thunderbolt 0000:00:0d.2:  Port 17: 8086:b26 (Revision: 3, TB Version: 1, Type: USB (0x200101))
[37306.062415] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[37306.062418] thunderbolt 0000:00:0d.2:   Max counters: 2
[37306.062422] thunderbolt 0000:00:0d.2:   NFC Credits: 0x800000
[37306.062425] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[37306.067619] thunderbolt 0000:00:0d.2:  Port 18: 8086:b26 (Revision: 3, TB Version: 1, Type: USB (0x200101))
[37306.067628] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[37306.067631] thunderbolt 0000:00:0d.2:   Max counters: 2
[37306.067634] thunderbolt 0000:00:0d.2:   NFC Credits: 0x800000
[37306.067637] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[37306.072846] thunderbolt 0000:00:0d.2:  Port 19: 8086:b26 (Revision: 3, TB Version: 1, Type: USB (0x200101))
[37306.072852] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[37306.072855] thunderbolt 0000:00:0d.2:   Max counters: 2
[37306.072858] thunderbolt 0000:00:0d.2:   NFC Credits: 0x800000
[37306.072860] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[37306.073348] pci 0000:02:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[37306.073593] pci 0000:03:00.0: [8086:0b26] type 01 class 0x060400
[37306.073693] pci 0000:03:00.0: enabling Extended Tags
[37306.073954] pci 0000:03:00.0: supports D1 D2
[37306.073957] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[37306.074407] pci 0000:03:00.0: Adding to iommu group 19
[37306.074761] pci 0000:03:01.0: [8086:0b26] type 01 class 0x060400
[37306.074862] pci 0000:03:01.0: enabling Extended Tags
[37306.075088] pci 0000:03:01.0: supports D1 D2
[37306.075091] pci 0000:03:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[37306.076409] pci 0000:03:01.0: Adding to iommu group 20
[37306.076662] pci 0000:03:02.0: [8086:0b26] type 01 class 0x060400
[37306.076765] pci 0000:03:02.0: enabling Extended Tags
[37306.077010] pci 0000:03:02.0: supports D1 D2
[37306.077015] pci 0000:03:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[37306.077270] pci 0000:03:02.0: Adding to iommu group 21
[37306.077466] usb 2-1: new SuperSpeed Plus Gen 2x1 USB device number 36 using xhci_hcd
[37306.077538] pci 0000:03:03.0: [8086:0b26] type 01 class 0x060400
[37306.077640] pci 0000:03:03.0: enabling Extended Tags
[37306.077882] pci 0000:03:03.0: supports D1 D2
[37306.077887] pci 0000:03:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[37306.081247] pci 0000:03:03.0: Adding to iommu group 22
[37306.081978] thunderbolt 0-1: new device found, vendor=0x3d device=0x26
[37306.081989] thunderbolt 0-1: CalDigit, Inc. TS4
[37306.082089] pci 0000:03:04.0: [8086:0b26] type 01 class 0x060400
[37306.082250] pci 0000:03:04.0: enabling Extended Tags
[37306.082493] pci 0000:03:04.0: supports D1 D2
[37306.082498] pci 0000:03:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[37306.086186] pci 0000:03:04.0: Adding to iommu group 23
[37306.086481] pci 0000:02:00.0: PCI bridge to [bus 03-2b]
[37306.086500] pci 0000:02:00.0:   bridge window [io  0x0000-0x0fff]
[37306.086511] pci 0000:02:00.0:   bridge window [mem 0x00000000-0x000fffff]
[37306.086531] pci 0000:02:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[37306.086541] pci 0000:03:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[37306.086567] pci 0000:03:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[37306.086590] pci 0000:03:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[37306.086610] pci 0000:03:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[37306.086633] pci 0000:03:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[37306.086834] pci 0000:03:00.0: PCI bridge to [bus 04-2b]
[37306.086849] pci 0000:03:00.0:   bridge window [io  0x0000-0x0fff]
[37306.086857] pci 0000:03:00.0:   bridge window [mem 0x00000000-0x000fffff]
[37306.086870] pci 0000:03:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[37306.086876] pci_bus 0000:04: busn_res: [bus 04-2b] end is updated to 04
[37306.087011] pci 0000:03:01.0: PCI bridge to [bus 05-2b]
[37306.087025] pci 0000:03:01.0:   bridge window [io  0x0000-0x0fff]
[37306.087034] pci 0000:03:01.0:   bridge window [mem 0x00000000-0x000fffff]
[37306.087048] pci 0000:03:01.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[37306.087053] pci_bus 0000:05: busn_res: [bus 05-2b] end is updated to 11
[37306.087659] pci 0000:03:02.0: PCI bridge to [bus 12-2b]
[37306.087676] pci 0000:03:02.0:   bridge window [io  0x0000-0x0fff]
[37306.087686] pci 0000:03:02.0:   bridge window [mem 0x00000000-0x000fffff]
[37306.087699] pci 0000:03:02.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[37306.087705] pci_bus 0000:12: busn_res: [bus 12-2b] end is updated to 1e
[37306.087855] pci 0000:03:03.0: PCI bridge to [bus 1f-2b]
[37306.087869] pci 0000:03:03.0:   bridge window [io  0x0000-0x0fff]
[37306.087878] pci 0000:03:03.0:   bridge window [mem 0x00000000-0x000fffff]
[37306.087891] pci 0000:03:03.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[37306.087896] pci_bus 0000:1f: busn_res: [bus 1f-2b] end is updated to 2a
[37306.090752] pci 0000:2b:00.0: [8086:5502] type 00 class 0x020000
[37306.090798] pci 0000:2b:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
[37306.090852] pci 0000:2b:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
[37306.091168] pci 0000:2b:00.0: PME# supported from D0 D3hot D3cold
[37306.094787] pci 0000:2b:00.0: Adding to iommu group 24
[37306.101088] thunderbolt 0000:00:0d.2: ICM rtd3 veto=0x00000004
[37306.101692] usb 2-1: New USB device found, idVendor=8087, idProduct=0b40, bcdDevice=12.34
[37306.101703] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[37306.101708] usb 2-1: Product: USB3.0 Hub
[37306.101711] usb 2-1: Manufacturer: Intel Corporation.
[37306.102774] hub 2-1:1.0: USB hub found
[37306.102908] hub 2-1:1.0: 4 ports detected
[37306.105362] pci 0000:03:04.0: PCI bridge to [bus 2b]
[37306.105380] pci 0000:03:04.0:   bridge window [io  0x0000-0x0fff]
[37306.105389] pci 0000:03:04.0:   bridge window [mem 0x00000000-0x000fffff]
[37306.105402] pci 0000:03:04.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[37306.105409] pci_bus 0000:2b: busn_res: [bus 2b] end is updated to 2b
[37306.105421] pci_bus 0000:03: busn_res: [bus 03-2b] end is updated to 2b
[37306.105435] pci 0000:03:01.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 05-11] add_size 100000 add_align 100000
[37306.105442] pci 0000:03:01.0: bridge window [mem 0x00100000-0x001fffff] to [bus 05-11] add_size 100000 add_align 100000
[37306.105449] pci 0000:03:02.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 12-1e] add_size 100000 add_align 100000
[37306.105456] pci 0000:03:02.0: bridge window [mem 0x00100000-0x001fffff] to [bus 12-1e] add_size 100000 add_align 100000
[37306.105462] pci 0000:03:03.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 1f-2a] add_size 100000 add_align 100000
[37306.105468] pci 0000:03:03.0: bridge window [mem 0x00100000-0x001fffff] to [bus 1f-2a] add_size 100000 add_align 100000
[37306.105480] pci 0000:02:00.0: bridge window [mem 0x00100000-0x005fffff 64bit pref] to [bus 03-2b] add_size 300000 add_align 100000
[37306.105488] pci 0000:02:00.0: bridge window [mem 0x00100000-0x006fffff] to [bus 03-2b] add_size 300000 add_align 100000
[37306.105506] pci 0000:02:00.0: BAR 14: assigned [mem 0x7c000000-0x881fffff]
[37306.105513] pci 0000:02:00.0: BAR 15: assigned [mem 0x6000000000-0x601c1fffff 64bit pref]
[37306.105518] pci 0000:02:00.0: BAR 13: assigned [io  0x4000-0x7fff]
[37306.105529] pci 0000:03:00.0: BAR 14: assigned [mem 0x7c000000-0x7c0fffff]
[37306.105533] pci 0000:03:00.0: BAR 15: assigned [mem 0x6000000000-0x60000fffff 64bit pref]
[37306.105538] pci 0000:03:01.0: BAR 14: assigned [mem 0x7c100000-0x800aaaa9]
[37306.105543] pci 0000:03:01.0: BAR 15: assigned [mem 0x6000100000-0x6009655554 64bit pref]
[37306.105549] pci 0000:03:02.0: BAR 14: assigned [mem 0x80100000-0x84055553]
[37306.105553] pci 0000:03:02.0: BAR 15: assigned [mem 0x6009700000-0x6012baaaa9 64bit pref]
[37306.105559] pci 0000:03:03.0: BAR 14: assigned [mem 0x84100000-0x87fffffd]
[37306.105564] pci 0000:03:03.0: BAR 15: assigned [mem 0x6012c00000-0x601c0ffffe 64bit pref]
[37306.105569] pci 0000:03:04.0: BAR 14: assigned [mem 0x88000000-0x881fffff]
[37306.105575] pci 0000:03:04.0: BAR 15: assigned [mem 0x601c100000-0x601c1fffff 64bit pref]
[37306.105580] pci 0000:03:00.0: BAR 13: assigned [io  0x4000-0x4fff]
[37306.105584] pci 0000:03:01.0: BAR 13: assigned [io  0x5000-0x5aa9]
[37306.105588] pci 0000:03:02.0: BAR 13: assigned [io  0x6000-0x6553]
[37306.105592] pci 0000:03:03.0: BAR 13: assigned [io  0x7000-0x7fff]
[37306.105597] pci 0000:03:04.0: BAR 13: no space for [io  size 0x1000]
[37306.105602] pci 0000:03:04.0: BAR 13: failed to assign [io  size 0x1000]
[37306.105608] pci 0000:03:00.0: PCI bridge to [bus 04]
[37306.105614] pci 0000:03:00.0:   bridge window [io  0x4000-0x4fff]
[37306.105626] pci 0000:03:00.0:   bridge window [mem 0x7c000000-0x7c0fffff]
[37306.105634] pci 0000:03:00.0:   bridge window [mem 0x6000000000-0x60000fffff 64bit pref]
[37306.105647] pci 0000:03:01.0: PCI bridge to [bus 05-11]
[37306.105654] pci 0000:03:01.0:   bridge window [io  0x5000-0x5aa9]
[37306.105665] pci 0000:03:01.0:   bridge window [mem 0x7c100000-0x800aaaa9]
[37306.105673] pci 0000:03:01.0:   bridge window [mem 0x6000100000-0x6009655554 64bit pref]
[37306.105686] pci 0000:03:02.0: PCI bridge to [bus 12-1e]
[37306.105691] pci 0000:03:02.0:   bridge window [io  0x6000-0x6553]
[37306.105701] pci 0000:03:02.0:   bridge window [mem 0x80100000-0x84055553]
[37306.105708] pci 0000:03:02.0:   bridge window [mem 0x6009700000-0x6012baaaa9 64bit pref]
[37306.105720] pci 0000:03:03.0: PCI bridge to [bus 1f-2a]
[37306.105724] pci 0000:03:03.0:   bridge window [io  0x7000-0x7fff]
[37306.105734] pci 0000:03:03.0:   bridge window [mem 0x84100000-0x87fffffd]
[37306.105741] pci 0000:03:03.0:   bridge window [mem 0x6012c00000-0x601c0ffffe 64bit pref]
[37306.105754] pci 0000:2b:00.0: BAR 0: assigned [mem 0x88000000-0x880fffff]
[37306.105765] pci 0000:2b:00.0: BAR 3: assigned [mem 0x88100000-0x88103fff]
[37306.105775] pci 0000:03:04.0: PCI bridge to [bus 2b]
[37306.105784] pci 0000:03:04.0:   bridge window [mem 0x88000000-0x881fffff]
[37306.105792] pci 0000:03:04.0:   bridge window [mem 0x601c100000-0x601c1fffff 64bit pref]
[37306.105803] pci 0000:02:00.0: PCI bridge to [bus 03-2b]
[37306.105808] pci 0000:02:00.0:   bridge window [io  0x4000-0x7fff]
[37306.105817] pci 0000:02:00.0:   bridge window [mem 0x7c000000-0x881fffff]
[37306.105825] pci 0000:02:00.0:   bridge window [mem 0x6000000000-0x601c1fffff 64bit pref]
[37306.105836] pcieport 0000:00:07.0: PCI bridge to [bus 02-2b]
[37306.105841] pcieport 0000:00:07.0:   bridge window [io  0x4000-0x7fff]
[37306.105847] pcieport 0000:00:07.0:   bridge window [mem 0x7c000000-0x881fffff]
[37306.105853] pcieport 0000:00:07.0:   bridge window [mem 0x6000000000-0x601c1fffff 64bit pref]
[37306.105860] PCI: No. 2 try to assign unassigned res
[37306.105863] pci 0000:03:00.0: resource 13 [io  0x4000-0x4fff] released
[37306.105867] pci 0000:03:00.0: PCI bridge to [bus 04]
[37306.105879] pci 0000:03:01.0: resource 13 [io  0x5000-0x5aa9] released
[37306.105882] pci 0000:03:01.0: PCI bridge to [bus 05-11]
[37306.105893] pci 0000:03:02.0: resource 13 [io  0x6000-0x6553] released
[37306.105896] pci 0000:03:02.0: PCI bridge to [bus 12-1e]
[37306.105908] pci 0000:03:03.0: resource 13 [io  0x7000-0x7fff] released
[37306.105911] pci 0000:03:03.0: PCI bridge to [bus 1f-2a]
[37306.105922] pci 0000:02:00.0: resource 13 [io  0x4000-0x7fff] released
[37306.105925] pci 0000:02:00.0: PCI bridge to [bus 03-2b]
[37306.105943] pci 0000:02:00.0: BAR 13: assigned [io  0x4000-0x7fff]
[37306.105949] pci 0000:03:00.0: BAR 13: assigned [io  0x4000-0x4fff]
[37306.105953] pci 0000:03:01.0: BAR 13: assigned [io  0x5000-0x5aa9]
[37306.105956] pci 0000:03:02.0: BAR 13: assigned [io  0x6000-0x6553]
[37306.105959] pci 0000:03:03.0: BAR 13: assigned [io  0x7000-0x7fff]
[37306.105963] pci 0000:03:04.0: BAR 13: no space for [io  size 0x1000]
[37306.105966] pci 0000:03:04.0: BAR 13: failed to assign [io  size 0x1000]
[37306.105969] pci 0000:03:00.0: PCI bridge to [bus 04]
[37306.105974] pci 0000:03:00.0:   bridge window [io  0x4000-0x4fff]
[37306.105984] pci 0000:03:00.0:   bridge window [mem 0x7c000000-0x7c0fffff]
[37306.105991] pci 0000:03:00.0:   bridge window [mem 0x6000000000-0x60000fffff 64bit pref]
[37306.106003] pci 0000:03:01.0: PCI bridge to [bus 05-11]
[37306.106008] pci 0000:03:01.0:   bridge window [io  0x5000-0x5aa9]
[37306.106017] pci 0000:03:01.0:   bridge window [mem 0x7c100000-0x800aaaa9]
[37306.106024] pci 0000:03:01.0:   bridge window [mem 0x6000100000-0x6009655554 64bit pref]
[37306.106036] pci 0000:03:02.0: PCI bridge to [bus 12-1e]
[37306.106041] pci 0000:03:02.0:   bridge window [io  0x6000-0x6553]
[37306.106050] pci 0000:03:02.0:   bridge window [mem 0x80100000-0x84055553]
[37306.106057] pci 0000:03:02.0:   bridge window [mem 0x6009700000-0x6012baaaa9 64bit pref]
[37306.106069] pci 0000:03:03.0: PCI bridge to [bus 1f-2a]
[37306.106073] pci 0000:03:03.0:   bridge window [io  0x7000-0x7fff]
[37306.106082] pci 0000:03:03.0:   bridge window [mem 0x84100000-0x87fffffd]
[37306.106090] pci 0000:03:03.0:   bridge window [mem 0x6012c00000-0x601c0ffffe 64bit pref]
[37306.106102] pci 0000:03:04.0: PCI bridge to [bus 2b]
[37306.106111] pci 0000:03:04.0:   bridge window [mem 0x88000000-0x881fffff]
[37306.106120] pci 0000:03:04.0:   bridge window [mem 0x601c100000-0x601c1fffff 64bit pref]
[37306.106133] pci 0000:02:00.0: PCI bridge to [bus 03-2b]
[37306.106138] pci 0000:02:00.0:   bridge window [io  0x4000-0x7fff]
[37306.106148] pci 0000:02:00.0:   bridge window [mem 0x7c000000-0x881fffff]
[37306.106157] pci 0000:02:00.0:   bridge window [mem 0x6000000000-0x601c1fffff 64bit pref]
[37306.106170] pcieport 0000:00:07.0: PCI bridge to [bus 02-2b]
[37306.106174] pcieport 0000:00:07.0:   bridge window [io  0x4000-0x7fff]
[37306.106180] pcieport 0000:00:07.0:   bridge window [mem 0x7c000000-0x881fffff]
[37306.106185] pcieport 0000:00:07.0:   bridge window [mem 0x6000000000-0x601c1fffff 64bit pref]
[37306.106293] pcieport 0000:02:00.0: enabling device (0000 -> 0003)
[37306.106909] pcieport 0000:03:00.0: enabling device (0000 -> 0003)
[37306.108105] pcieport 0000:03:01.0: enabling device (0000 -> 0003)
[37306.108568] pcieport 0000:03:01.0: pciehp: Slot #1 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl+ IbPresDis- LLActRep+
[37306.109802] pcieport 0000:03:02.0: enabling device (0000 -> 0003)
[37306.110701] pcieport 0000:03:02.0: pciehp: Slot #2 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl+ IbPresDis- LLActRep+
[37306.116971] pcieport 0000:03:03.0: enabling device (0000 -> 0003)
[37306.117342] usb 3-1: new high-speed USB device number 13 using xhci_hcd
[37306.119011] pcieport 0000:03:03.0: pciehp: Slot #3 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl+ IbPresDis- LLActRep+
[37306.122683] pcieport 0000:03:04.0: enabling device (0000 -> 0002)
[37306.127261] igc 0000:2b:00.0: enabling device (0000 -> 0002)
[37306.127573] igc 0000:2b:00.0: PCIe PTM not supported by PCIe bus/controller
[37306.185919] pps pps0: new PPS source ptp0
[37306.186081] igc 0000:2b:00.0 (unnamed net_device) (uninitialized): PHC added
[37306.245420] igc 0000:2b:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[37306.245435] igc 0000:2b:00.0 eth0: MAC: 64:4b:f0:37:08:97
[37306.252044] igc 0000:2b:00.0 enp43s0: renamed from eth0
[37306.274115] kauditd_printk_skb: 8 callbacks suppressed
[37306.341933] usb 3-1: New USB device found, idVendor=2188, idProduct=5802, bcdDevice= 1.01
[37306.341945] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[37306.341949] usb 3-1: Product: TS4 USB2.0 Hub
[37306.341953] usb 3-1: Manufacturer: CalDigit, Inc.
[37306.343133] hub 3-1:1.0: USB hub found
[37306.343206] hub 3-1:1.0: 5 ports detected
[37306.598628] thunderbolt 0000:00:0d.3: control channel starting...
[37306.598640] thunderbolt 0000:00:0d.3: starting TX ring 0
[37306.598650] thunderbolt 0000:00:0d.3: enabling interrupt at register 0x38200 bit 0 (0x0 -> 0x1)
[37306.598658] thunderbolt 0000:00:0d.3: starting RX ring 0
[37306.598665] thunderbolt 0000:00:0d.3: enabling interrupt at register 0x38200 bit 12 (0x1 -> 0x1001)
[37306.605271] thunderbolt 0000:00:0d.3: ICM rtd3 veto=0x00000000
[37306.607378] thunderbolt 0000:00:0d.3: ICM rtd3 veto=0x00000001
[37306.907461] thunderbolt 0000:00:0d.3: ICM rtd3 veto=0x00000000
[Additional USB device discover omitted]

All of the pci output was already present by default (could maybe be made more verbose even with dynamic_debug), but note in particular the thunderbolt debug output that shows a long list of ports at the start. I believe these are all the ports that the USB router exposes internally. Interesting observations:

• There are 2 DP/HDMI ports, which confirms the documentation that says the dock supports up to dual displays. Note that I expect these two ports are multiplexed (probably using a multiplexer integrated into the Goshen Ridge controller) onto the two downstream TB4 ports and the DP connector (which is probably just connected to the third downstream TB port on the Goshen Ridge controller that is not exposed). Interestingly this says “DP/HDMI” except HDMI/DP++ is not actually supported it seems.
• There are 4 USB ports, which are probably endpoints of USB3 tunneling, so this is probably essentially the 4-port USB3 hub that is listed in lsusb (though interestingly the first one has a slightly different numerical type, not sure what that means).
• There are 3 PCIe ports, of which the first one has a slightly different numerical type. I guess one of these is connected to the PCIe ethernet chip, but why two more? Could be downstream TB4 ports, but I thought those would typically be both connected to an external PCIe switch (so needing only one port on the USB4 switch)? Note sure how to relate these ports to the lspci output and pci kernel output, either.
• The first 6 ports just have type “Port”, so no idea what those are.
• Some ports are “disabled in EEPROM”, which I assume are ports that Goshen Ridge supports but are not connected to anything in this particular dock, and one port is “Inactive”, not sure what that means.
• No idea what these counters and (NFC) credits mean…

[Matthijs_Kooijman]

Using modprobe thunderbolt dyndbg=+p you can actually get debug printing enabled at module load, so this allows capturing the info on the host-side USB4 routers (two of them, each serving two ports) when nothing is plugged in:

[63297.159042] ACPI: bus type thunderbolt registered
[63297.159624] thunderbolt 0000:00:0d.2: total paths: 12
[63297.172612] thunderbolt 0000:00:0d.2: allocating TX ring 0 of size 10
[63297.172658] thunderbolt 0000:00:0d.2: allocating RX ring 0 of size 10
[63297.172683] thunderbolt 0000:00:0d.2: control channel created
[63297.172689] thunderbolt 0000:00:0d.2: using firmware connection manager
[63297.172691] thunderbolt 0000:00:0d.2: NHI initialized, starting thunderbolt
[63297.172693] thunderbolt 0000:00:0d.2: control channel starting...
[63297.172695] thunderbolt 0000:00:0d.2: starting TX ring 0
[63297.172700] thunderbolt 0000:00:0d.2: enabling interrupt at register 0x38200 bit 0 (0x0 -> 0x1)
[63297.172705] thunderbolt 0000:00:0d.2: starting RX ring 0
[63297.172710] thunderbolt 0000:00:0d.2: enabling interrupt at register 0x38200 bit 12 (0x1 -> 0x1001)
[63297.182282] thunderbolt 0000:00:0d.2: USB4 proxy operations supported
[63297.182287] thunderbolt 0000:00:0d.2: security level set to none
[63297.187131] thunderbolt 0000:00:0d.2: current switch config:
[63297.187136] thunderbolt 0000:00:0d.2:  USB4 Switch: 8087:9a1b (Revision: 1, TB Version: 32)
[63297.187142] thunderbolt 0000:00:0d.2:   Max Port Number: 13
[63297.187144] thunderbolt 0000:00:0d.2:   Config:
[63297.187145] thunderbolt 0000:00:0d.2:    Upstream Port Number: 7 Depth: 0 Route String: 0x0 Enabled: 1, PlugEventsDelay: 10ms
[63297.187150] thunderbolt 0000:00:0d.2:    unknown1: 0x0 unknown4: 0x0
[63297.238443] thunderbolt 0000:00:0d.2: 0: uid: 0xb90a0b708087e118
[63297.277519] thunderbolt 0000:00:0d.2:  Port 1: 8086:9a1b (Revision: 1, TB Version: 1, Type: Port (0x1))
[63297.277531] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[63297.277535] thunderbolt 0000:00:0d.2:   Max counters: 16
[63297.277537] thunderbolt 0000:00:0d.2:   NFC Credits: 0x83c00000
[63297.277540] thunderbolt 0000:00:0d.2:   Credits (total/control): 60/2
[63297.317662] thunderbolt 0000:00:0d.2:  Port 2: 8086:9a1b (Revision: 1, TB Version: 1, Type: Port (0x1))
[63297.317673] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[63297.317676] thunderbolt 0000:00:0d.2:   Max counters: 16
[63297.317677] thunderbolt 0000:00:0d.2:   NFC Credits: 0x83c00000
[63297.317680] thunderbolt 0000:00:0d.2:   Credits (total/control): 60/2
[63297.358043] thunderbolt 0000:00:0d.2:  Port 3: 8086:9a1b (Revision: 1, TB Version: 1, Type: Port (0x1))
[63297.358054] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[63297.358058] thunderbolt 0000:00:0d.2:   Max counters: 16
[63297.358060] thunderbolt 0000:00:0d.2:   NFC Credits: 0x83c00000
[63297.358062] thunderbolt 0000:00:0d.2:   Credits (total/control): 60/2
[63297.398397] thunderbolt 0000:00:0d.2:  Port 4: 8086:9a1b (Revision: 1, TB Version: 1, Type: Port (0x1))
[63297.398408] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 19/19
[63297.398412] thunderbolt 0000:00:0d.2:   Max counters: 16
[63297.398414] thunderbolt 0000:00:0d.2:   NFC Credits: 0x83c00000
[63297.398417] thunderbolt 0000:00:0d.2:   Credits (total/control): 60/2
[63297.403356] thunderbolt 0000:00:0d.2:  Port 5: 8086:9a1b (Revision: 1, TB Version: 1, Type: DP/HDMI (0xe0101))
[63297.403368] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 9/9
[63297.403371] thunderbolt 0000:00:0d.2:   Max counters: 2
[63297.403373] thunderbolt 0000:00:0d.2:   NFC Credits: 0x100000d
[63297.403375] thunderbolt 0000:00:0d.2:   Credits (total/control): 16/0
[63297.408373] thunderbolt 0000:00:0d.2:  Port 6: 8086:9a1b (Revision: 1, TB Version: 1, Type: DP/HDMI (0xe0101))
[63297.408385] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 9/9
[63297.408388] thunderbolt 0000:00:0d.2:   Max counters: 2
[63297.408390] thunderbolt 0000:00:0d.2:   NFC Credits: 0x100000d
[63297.408392] thunderbolt 0000:00:0d.2:   Credits (total/control): 16/0
[63297.423603] thunderbolt 0000:00:0d.2:  Port 7: 8086:9a1b (Revision: 1, TB Version: 1, Type: NHI (0x2))
[63297.423614] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 11/11
[63297.423618] thunderbolt 0000:00:0d.2:   Max counters: 16
[63297.423620] thunderbolt 0000:00:0d.2:   NFC Credits: 0x41800000
[63297.423622] thunderbolt 0000:00:0d.2:   Credits (total/control): 24/0
[63297.428530] thunderbolt 0000:00:0d.2:  Port 8: 8086:9a1b (Revision: 1, TB Version: 1, Type: PCIe (0x100101))
[63297.428542] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[63297.428545] thunderbolt 0000:00:0d.2:   Max counters: 2
[63297.428547] thunderbolt 0000:00:0d.2:   NFC Credits: 0x40800000
[63297.428549] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[63297.433542] thunderbolt 0000:00:0d.2:  Port 9: 8086:9a1b (Revision: 1, TB Version: 1, Type: PCIe (0x100101))
[63297.433554] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[63297.433557] thunderbolt 0000:00:0d.2:   Max counters: 2
[63297.433559] thunderbolt 0000:00:0d.2:   NFC Credits: 0x40800000
[63297.433561] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[63297.436176] thunderbolt 0000:00:0d.2:  Port 10: not implemented
[63297.438872] thunderbolt 0000:00:0d.2:  Port 11: not implemented
[63297.444070] thunderbolt 0000:00:0d.2:  Port 12: 8086:9a1b (Revision: 1, TB Version: 1, Type: USB (0x200101))
[63297.444081] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[63297.444085] thunderbolt 0000:00:0d.2:   Max counters: 2
[63297.444086] thunderbolt 0000:00:0d.2:   NFC Credits: 0x800000
[63297.444089] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[63297.448918] thunderbolt 0000:00:0d.2:  Port 13: 8086:9a1b (Revision: 1, TB Version: 1, Type: USB (0x200101))
[63297.448928] thunderbolt 0000:00:0d.2:   Max hop id (in/out): 8/8
[63297.448931] thunderbolt 0000:00:0d.2:   Max counters: 2
[63297.448933] thunderbolt 0000:00:0d.2:   NFC Credits: 0x800000
[63297.448935] thunderbolt 0000:00:0d.2:   Credits (total/control): 8/0
[63297.448938] thunderbolt 0000:00:0d.2: 0: linked ports 1 <-> 2
[63297.448941] thunderbolt 0000:00:0d.2: 0: linked ports 3 <-> 4
[63297.457392] thunderbolt 0000:00:0d.2: ICM rtd3 veto=0x00000000
[63297.457416] thunderbolt 0000:00:0d.2: ICM rtd3 veto=0x00000001
[63297.458028] thunderbolt 0000:00:0d.3: total paths: 12
[63297.471015] thunderbolt 0000:00:0d.3: allocating TX ring 0 of size 10
[63297.471064] thunderbolt 0000:00:0d.3: allocating RX ring 0 of size 10
[63297.471088] thunderbolt 0000:00:0d.3: control channel created
[63297.471095] thunderbolt 0000:00:0d.3: using firmware connection manager
[63297.471097] thunderbolt 0000:00:0d.3: NHI initialized, starting thunderbolt
[63297.471099] thunderbolt 0000:00:0d.3: control channel starting...
[63297.471101] thunderbolt 0000:00:0d.3: starting TX ring 0
[63297.471106] thunderbolt 0000:00:0d.3: enabling interrupt at register 0x38200 bit 0 (0x0 -> 0x1)
[63297.471110] thunderbolt 0000:00:0d.3: starting RX ring 0
[63297.471115] thunderbolt 0000:00:0d.3: enabling interrupt at register 0x38200 bit 12 (0x1 -> 0x1001)
[63297.480278] thunderbolt 0000:00:0d.3: USB4 proxy operations supported
[63297.480285] thunderbolt 0000:00:0d.3: security level set to none
[63297.483413] thunderbolt 0000:00:0d.2: ICM rtd3 veto=0x00000000
[63297.485196] thunderbolt 0000:00:0d.3: current switch config:
[63297.485200] thunderbolt 0000:00:0d.3:  USB4 Switch: 8087:9a1d (Revision: 1, TB Version: 32)
[63297.485205] thunderbolt 0000:00:0d.3:   Max Port Number: 13
[63297.485207] thunderbolt 0000:00:0d.3:   Config:
[63297.485208] thunderbolt 0000:00:0d.3:    Upstream Port Number: 7 Depth: 0 Route String: 0x0 Enabled: 1, PlugEventsDelay: 10ms
[63297.485212] thunderbolt 0000:00:0d.3:    unknown1: 0x0 unknown4: 0x0
[63297.536875] thunderbolt 0000:00:0d.3: 0: uid: 0x10d7eb2180874d14
[63297.576464] thunderbolt 0000:00:0d.3:  Port 1: 8086:9a1d (Revision: 1, TB Version: 1, Type: Port (0x1))
[63297.576476] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 19/19
[63297.576480] thunderbolt 0000:00:0d.3:   Max counters: 16
[63297.576482] thunderbolt 0000:00:0d.3:   NFC Credits: 0x83c00000
[63297.576484] thunderbolt 0000:00:0d.3:   Credits (total/control): 60/2
[63297.616748] thunderbolt 0000:00:0d.3:  Port 2: 8086:9a1d (Revision: 1, TB Version: 1, Type: Port (0x1))
[63297.616760] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 19/19
[63297.616763] thunderbolt 0000:00:0d.3:   Max counters: 16
[63297.616765] thunderbolt 0000:00:0d.3:   NFC Credits: 0x83c00000
[63297.616767] thunderbolt 0000:00:0d.3:   Credits (total/control): 60/2
[63297.657036] thunderbolt 0000:00:0d.3:  Port 3: 8086:9a1d (Revision: 1, TB Version: 1, Type: Port (0x1))
[63297.657045] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 19/19
[63297.657048] thunderbolt 0000:00:0d.3:   Max counters: 16
[63297.657050] thunderbolt 0000:00:0d.3:   NFC Credits: 0x83c00000
[63297.657053] thunderbolt 0000:00:0d.3:   Credits (total/control): 60/2
[63297.696912] thunderbolt 0000:00:0d.3:  Port 4: 8086:9a1d (Revision: 1, TB Version: 1, Type: Port (0x1))
[63297.696923] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 19/19
[63297.696926] thunderbolt 0000:00:0d.3:   Max counters: 16
[63297.696928] thunderbolt 0000:00:0d.3:   NFC Credits: 0x83c00000
[63297.696930] thunderbolt 0000:00:0d.3:   Credits (total/control): 60/2
[63297.701928] thunderbolt 0000:00:0d.3:  Port 5: 8086:9a1d (Revision: 1, TB Version: 1, Type: DP/HDMI (0xe0101))
[63297.701939] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 9/9
[63297.701942] thunderbolt 0000:00:0d.3:   Max counters: 2
[63297.701944] thunderbolt 0000:00:0d.3:   NFC Credits: 0x100000d
[63297.701946] thunderbolt 0000:00:0d.3:   Credits (total/control): 16/0
[63297.706899] thunderbolt 0000:00:0d.3:  Port 6: 8086:9a1d (Revision: 1, TB Version: 1, Type: DP/HDMI (0xe0101))
[63297.706909] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 9/9
[63297.706911] thunderbolt 0000:00:0d.3:   Max counters: 2
[63297.706913] thunderbolt 0000:00:0d.3:   NFC Credits: 0x100000d
[63297.706915] thunderbolt 0000:00:0d.3:   Credits (total/control): 16/0
[63297.721996] thunderbolt 0000:00:0d.3:  Port 7: 8086:9a1d (Revision: 1, TB Version: 1, Type: NHI (0x2))
[63297.722005] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 11/11
[63297.722008] thunderbolt 0000:00:0d.3:   Max counters: 16
[63297.722010] thunderbolt 0000:00:0d.3:   NFC Credits: 0x41800000
[63297.722012] thunderbolt 0000:00:0d.3:   Credits (total/control): 24/0
[63297.727054] thunderbolt 0000:00:0d.3:  Port 8: 8086:9a1d (Revision: 1, TB Version: 1, Type: PCIe (0x100101))
[63297.727065] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 8/8
[63297.727067] thunderbolt 0000:00:0d.3:   Max counters: 2
[63297.727069] thunderbolt 0000:00:0d.3:   NFC Credits: 0x40800000
[63297.727071] thunderbolt 0000:00:0d.3:   Credits (total/control): 8/0
[63297.732210] thunderbolt 0000:00:0d.3:  Port 9: 8086:9a1d (Revision: 1, TB Version: 1, Type: PCIe (0x100101))
[63297.732221] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 8/8
[63297.732224] thunderbolt 0000:00:0d.3:   Max counters: 2
[63297.732226] thunderbolt 0000:00:0d.3:   NFC Credits: 0x40800000
[63297.732228] thunderbolt 0000:00:0d.3:   Credits (total/control): 8/0
[63297.734706] thunderbolt 0000:00:0d.3:  Port 10: not implemented
[63297.737229] thunderbolt 0000:00:0d.3:  Port 11: not implemented
[63297.742554] thunderbolt 0000:00:0d.3:  Port 12: 8086:9a1d (Revision: 1, TB Version: 1, Type: USB (0x200101))
[63297.742564] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 8/8
[63297.742567] thunderbolt 0000:00:0d.3:   Max counters: 2
[63297.742569] thunderbolt 0000:00:0d.3:   NFC Credits: 0x800000
[63297.742571] thunderbolt 0000:00:0d.3:   Credits (total/control): 8/0
[63297.747562] thunderbolt 0000:00:0d.3:  Port 13: 8086:9a1d (Revision: 1, TB Version: 1, Type: USB (0x200101))
[63297.747573] thunderbolt 0000:00:0d.3:   Max hop id (in/out): 8/8
[63297.747576] thunderbolt 0000:00:0d.3:   Max counters: 2
[63297.747578] thunderbolt 0000:00:0d.3:   NFC Credits: 0x800000
[63297.747580] thunderbolt 0000:00:0d.3:   Credits (total/control): 8/0
[63297.747583] thunderbolt 0000:00:0d.3: 0: linked ports 1 <-> 2
[63297.747586] thunderbolt 0000:00:0d.3: 0: linked ports 3 <-> 4
[63297.753615] thunderbolt 0000:00:0d.3: ICM rtd3 veto=0x00000000
[63297.753628] thunderbolt 0000:00:0d.3: ICM rtd3 veto=0x00000001
[63297.782254] thunderbolt 0000:00:0d.3: ICM rtd3 veto=0x00000000

This does not seem to show whether these ports are in or out adapters, but maybe that’s encoded in the numeric port types or so (or maybe some are bidirectional even)?

[Eric_Crawford-Anton]

My curiosity is what would happen if you use the M.2 slot in a dock with a PCIE to M.2 adapter to connect a GPU (of course with external power form another source)? In the ideal situation would the GPU be able to utilize the Bandwidth to give favorable results, or would it even be recognized as a GPU by the Laptop? Basically, using a $45 adapter and external power supply that is extra anyways instead of buying an eGPU housing.

[Be_Far]

@Eric_Crawford-Anton

My curiosity is what would happen if you use the M.2 slot in a dock with a PCIE to M.2 adapter to connect a GPU

Would depend on the bandwidth of the dock. With a TB4 dock, it would probably just saturate the link and compete with anything else plugged in for bandwidth. Anything else might flat out refuse to work depending on implementation.

M.2 to PCIe adapters are really designed for an M.2 connected directly to a cpu pcie lane, anything more and you risk a chain of x16 to x4 to x2/Thunderbolt to x4 or similar and it’s just a mess.

From my experience, best practice is to keep your eGPU on a separate Thunderbolt controller from everything else for best performance. Framework’s dual-controller setup is awesome for that because I can have a fully saturated eGPU controller on one side, and all my peripherals which don’t even scratch the surface of the max of the link on the other.

[TermoZour]

So from what I understood, even with Thunderbolt 4 you can not reach full PCIe 3.0 x16 speed for a GPU using a simple eGPU dock?

According to Kingston, PCIe 3.0 x16 uses 32 GB/s of bandwidth and Thunderbolt 4 according to Intel has a maximum 40 Gbps which roughly translates to 5 GB/s which is obviously not enough.

Or is that the speed per lane (and we have 4 lanes) therefore we actually have 20 GB/s bandwidth which would almost be enough to saturate an AMD RX 580 GPU?

[Ray519]

You mixed up the numbers. 32GB/s is what is mentioned for x16 Gen 4 not Gen 3.

But those numbers are just straight conversions from the GBit/s numbers and still include encoding overhead. They are theoretical numbers that cannot ever be reached. You are better off with Wikipedia numbers.

Already available TB devices use mostly TB3 controllers and / or only have x4 Gen 3 slots for components like NVMes and GPUs.
The theoretical max. speed for TB3/USB4 with x4 Gen 3 is around 3.31-3.4 GB/s with all the overheads I know of subtracted. NVMes can in practice reach 3.1 GB/s usable speeds over that, GPUs will achieve less (more in the neighborhood of 2.6GB/s), probably for reaons having to do with the protocol and latency.

New ASM2464 controllers do support x4 Gen 4 on the port (the raw number is 64G), which would already be more than USB4 40G can transfer. That comes out to ~ 3.9-4 GB/s of theoretical max. bandwidth, with NVMe enclosures having shown 3.7 GB/s real-world.
Anything more would require newer USB4 versions with some fixes and 80G or asymmetric 120G modes. So you can expect further doubling of the bandwidth and a bit more in the next few generations.

[TermoZour]

So I would not be able to saturate the 4 lanes of Thunderbolt 3 with a GPU, and I would also not be able to use it at 100% because I would require 4x the number of lanes, which we do not have in any way, shape or form. At least for 12th gen Intel CPU thunderbolt controller.

[Ray519]

Sort of. Right now we are limited to 32G PCIe connections due to all eGPUs using TB3 controllers. Hosts like the 12th gen FW can do more, getting closer to the actual limit of USB4 (technically around 38.8G), but still nowhere near what desktop GPUs use.

What GPUs do, that they achieve less bandwidth than NVMes, I do not yet know. It’s some type of overhead. But I do not know if this could count as “not fully utilizing the existing 32G PCIe connection”. That would depend on what exactly it is doing. Maybe.

How much that affects real-world gaming perf. depends on how much bandwidth the game actually uses. Some require far less than others. Notebooks always had less bandwidth to their dGPUs. So a further doubling should get us into a region where it is much less of a concern, even if it still falls short of desktop GPUs have.

[TermoZour]

I was wondering if with these limitations in mind, I should wait for Thunderbolt 5 and buy a powerful GPU then, or buy whatever eGPU dock I can grab my hands on now and a mid-range GPU (like an AMD RX 550 or 580).

Looking at some older benchmark by GamersNexus with an Nvidia GTX 1080, there doesn’t seem to be a major difference between x8 and x16 PCIe 3.0.

The question then came to mind - is there a noticeable difference in gaming between x4 and x16?
According to some other videos I just found online, not really (subjective opinion). And these tests were run on some pretty demanding games.
Test with RTX 3060 TI
Test with RTX 3070
Test with Radeon VI 16 GB

So judging by this (rather small) sample of tests, I think TB3/4 is good enough for my needs. I’m only aiming to reach 144 FPS on Counter-Strike 2 and (soon-to-be-released) Death Stranding 2 running at either 1080p or 1440p. Now I might not reach this framerate with an RX 580 in Death Stranding 2, but I’m pretty sure I will be able to on Counter-Strike. Since I’m not planning on doing anything intensive on the GPU (like Machine Learning work or simulations) I’ll either keep this RX 580 or buy an RX 6600.

[Ray519]

I do not have a good handle on that.

The difference was noticeable with my 3090 as eGPU. But it varied widely between games.

All I can say is, bandwidth matters not much, as long as you have enough. Similar to memory sizes. How much more memory than needed you have does not impact a specific application you are interested in. You just need to have at least how ever much that application needs.
So statistical effects will increase the slower the connection, because more applications expect more. Modern GPUs run on x16 Gen 4. Halving that shows almost no effects, quartering it shows some effects etc…

Also depends very much on the game. An open world game that is constantly streaming data into the GPU will incur more bandwidth than a game that loads a specific level consisting of mostly static data and is then done with loading until the next level.
RayTracing will also have an effect, as it requires the CPU to track BVHs that have to be updated on the GPU whenever they change, further increasing the bandwidth needed.
And then it becomes a question of how latency tolerant the application is. With less bandwidth and especially if there is not enough, things will take longer to upload. Is that accounted for by the application? Some data may only cause pop-in if it arrives late. Other data is needed to coordinate the next frame and it arriving late will directly impact the frametime of the next frame.

That is why using hybrid graphics (i.e. a monitor not attached to the eGPU) has such a big effect. It requires a continuous stream of data out of the GPU, bandwidth of which depends on resolution and frame rate of that stream.

For more details on performance impact on games, you’ll probably have to go to eGPU.io or similar.

[0nn0]

I have a question about bandwidth sharing when using a TB4 dock on my 12th gen intel Framework 13. (running Kubuntu 22.04.3, if that matters) I’m using a Lenovo universal TB4 dock with 2 1440p60 displays attached as well as a QNAP 10GbE TB3 adapter. Sometimes one of the displays will either flicker or completely drop the picture when I use the 10GbE. Is this a bandwidth limitation or should I

[Adrian_Joachim]

@0nn0

I have a question about bandwidth sharing when using a TB4 dock on my 12th gen intel Framework 13. (running Kubuntu 22.04.3, if that matters) I’m using a Lenovo universal TB4 dock with 2 1440p60 displays attached as well as a QNAP 10GbE TB3 adapter. Sometimes one of the displays will either flicker or completely drop the picture when I use the 10GbE. Is this a bandwidth limitation or should I

If they are both on the same side they do share bandwidth but flickering displays may likely just be a cable quality issue.

[0nn0]

@Adrian_Joachim

If they are both on the same side they do share bandwidth but flickering displays may likely just be a cable quality issue.

They did not do this with my previous dock, so I don’t think it is the DP cables. The TB cable that I use came with the dock and is quite short. I don’t think it is a signal integrity problem. How would I analyze this without buying a new cable?

[Ray519]

Thunderbolt / USB4 was designed to reserve bandwidth, especially for DP data. So it should absolutely not be any kind of bandwidth issue. PCIe should be lower prioritized and simply be delayed or throttled when there is DP traffic ongoing and not enough bandwidth for all.

So the above mentioned cable quality could be it, or some other signal quality issue caused by power fluctuations or sth. Some monitors are really sensitive to electrostatic discharge or other effects caused by bad hardware (we had that with bad, powered USB hubs at work) But really hard to identify which, if any device is actually breaking its specs in how sensitive it is or in causing issues by beeing out of spec.