Details about USB, Thunderbolt and dock operation

[Matthijs_Kooijman]

@D.H

You are quite right about DisplayLink being a flaming pile of garbage.

Thanks for putting it like that. This is how I thought about it, but I wanted to stay a bit more objective in my post

[Lanes vs lines]

I had indeed also seen confusing things where e.g. DP uses unidirectional lanes and USB/PCIe use bidirectional lanes. To prevent confusion, I have used “lane” to mean a single differential pair in my post everywhere, which is consistent but indeed not match other sources, and apparently is not sufficient to prevent confusion (as seen from the discussion). I guess I could just prevent the word lane entirely (or maybe only use it with sufficient qualification in relation to specific protocols), and use “line” for a single unidirectional wire pair (instead of “lane” I used for that now) and keep using “full-duplex channel” for a pair of lines.

@Project_sbc

What sets tiger and ice lake apart is that the controller got integrated into the cpu dye. For ice lake, thunderbolt 3, and tiger lake thunderbolt 4. This brought down power consumption and improved some latency on the processing, which is a win win for everyone.

Cool, I’ll add a bit about this in the post (see below).

@Project_sbc

So tiger lake has a titan ridge thunderbolt controller on the cpu dye.

Shouldn’t this be Maple Ridge? According to Thunderbolt (interface) - Wikipedia Titan Ridge only does TB3, not TB4?

@Ray519

Somewhat. 1st gen / Alpine Ridge Controllers only supported HBR2, but 2x 4xHBR2 was possible. Titan Ridge added support for HBR3, but TB does not have enough bandwidth for 2 full 4xHBR3 connections. Same holds for USB4. More on that below.

Thanks, I qualified the line you quoted a bit below.

@Ray519

Correct. So far, the host TB-controllers only had at most 2 DP links available. But you can access these 2 links at any point of a TB3 / TB4 hierarchy.

Ah, I see, that clarifies things a bit. I added some remarks about this.

@Ray519

I do not know whether it is a protocol limitation of USB4 to only support 2 DP links at most. It may just be an Intel limitation as so far, all Intel TB controllers, even those inside Tiger Lake (2 controllers for 2 TB-ports each) only have 2 DP links available, which they may distribute across the 2 possible TB-Ports.

I am under the impression that the USB4 protocol is designed to just generically support arbitrary tunnels, but actual implementations will be limited by the amount of DP in/out adapters they have. E.g. the USB4 spec says “Each Router contains up to 64 Adapters.” and “A Router may contain one or more DP IN Protocol Adapters, one or more DP OUT Protocol Adapters, or a combination of DP IN and DP OUT Protocol Adapters.”

It also says “A USB4 host shall support DP tunneling. A Host Router shall contain at least one DP IN Adapter and may optionally contain one or more DP OUT Adapters. A USB4 hub shall support DP Tunneling. A USB4 Hub shall contain at least one DP OUT Adapter and may optionally contain one or more DP IN Adapters. A USB4 peripheral device may optionally support DP Tunneling. If a USB4 peripheral device supports DP Tunneling, it shall contain at least one DP Adapter.”, so it seems the USB4 spec only requires a single DP link to be supported (I think it’s interesting that a Hub is required to support a DP out adapter, does that mean it must also have a connector for that? Or is the expectation to be able to route it to DP-alt-mode on (a) downstream USB-C port(s)?)

@Ray519

Bandwidth allocation for tunneling DP though TB:

I also added some info about this below.

@Ray519

But if there is bandwidth left over, like when driving a single FHD60 display behind a TB3 dock with built-in HBR3 MST Hub (4xHBR3 / ~25.9 GBit/s allocated for this link, but only ~3.4 GBit/s actually used) the difference in bandwidth can still be used by other data than Displayport, like USB and PCIe traffic.

I can imagine that bulk USB traffic will fill up any such leftofter bandwidth, but does that also work for PCIe? Doesn’t PCIe also need/negotiate some reserved bandwidth?

@joevt

Most of the numbers shown here are for bits on the wire. However, that does not represent the amount of actual data being sent. For example, SATA, PCIe 2.0, USB 3.0, HDMI, and DisplayPort 1.4 transmit 10 bits on the wire for every 8 bits of data, otherwise known as 8b/10b encoding. This means USB 3.0 is more like 4 Gbps instead of 5 Gbps.

Good point, copied your suggestion below. I also copied your list of raw vs encoded bitrates, and added a few more entries below. I did find an error in your USB3.1 bandwidth, which I calculated as 10/132*128 = 9.697 Gbps, but you had 9.846Gbps.

@joevt

MTS: Should be MST (Multi-Stream Transport). MST is interesting because an MST hub can convert DisplayPort signals of different link width and link rates like a PCIe switch can. Also, MST can use DSC on the input and decompress that on the output for displays that don’t support DSC. DSC allows connecting 3 4K 60Hz displays from a single HBR3 x4 link.

Thanks, fixed MST below and added info about DSC.

@joevt

When connected to a Thunderbolt 3 host, PCIe tunnelling is used to communicate with a USB controller in the Goshen Ridge. In that case, the USB hub is still used, limiting upstream bandwidth to 10 Gbps. Intel could have done that differently to fully utilize the ≈23 Gbps PCIe bandwidth but i guess the hub method was easier?

Why 10Gbps? If the hub connects to the USB host in Goshen Ridge using USB3 gen2x2, it could use two 10Gbps duplex lines, so 20Gbps upstream bandwidth, right?

@joevt

or the XDR display when connected to a GPU that doesn’t support DSC, Apple has a trick to force two HBR3 links (using their software Thunderbolt connection manager). The trick works because the XDR only requires 19.5 Gbps per connection (38.93 Gbps total) and Thunderbolt doesn’t transmit the stuffing symbols used to fill the HBR3 bandwidth (25.92 Gbps).

Right, so this essentially over-allocates the two links, in the knowledge that the actual bandwidth to be used will fit, even when two HBR3 links will not. I guess this works if the host and dock can agree on this, then the GPU and display can just use two HBR3 links (with added stuffing) and never be the wiser. This does mean that the GPU and display cannot change the resolution to one that needs more bandwidth, so this means that some integration between the GPU driver / display configuration and TB controller in the host is required.

This essentially contradicts what @Ray519 said about bandwidth allocations happening based on the max bandwidth for the negotiated DP bitrate, rather than actual bandwidth, but I guess the “trick” here is that rather than just negotiating the DP link bandwidth over the actual DP channel between the GPU and TB controller (which I guess is how it normally works?), this involves some OS-level integration between the different drivers involved (to communicate about actual bandwidth needed)?

Things to change in my post
Below here, some things to be added / changed in my original post. In addition to changes based on replies, I also added some info on single vs multi-TT hubs that I missed originally.

Edit: I’ve updated the original post, and removed the proposed changes below for clarity