I have been going around in circles on some key performance characteristics of the Desktop that I thought would be worth writing down, in the hope they benefit other purchasers. I have not bought yet - still in the planning phase.
My use case is AI inference. I hope that the internal GPU suffices for my needs, but I want to insulate myself against the possibility that it will not. I originally was leaning in the direction of being able to fit a GPU card to the internal PCIe slot, and of course to do that on the Desktop one has to use a custom case.
However, I have stumbled on some performance characteristics that say that, to my surprise, the external USB port should perform more than four times as speedily as the internal interface (which implements version 4.0 of the PCI spec):
USB 4: ~5 GB/s
PCIe 4.0 x4: ~1 GB/s
One would need an external card enclosure and power supply that supports USB4 (such as this one). I acknowledge that this is quite a pricey way to do things - for now I am trying to do everything in one box, hedging against not knowing how my AI rig will develop. If it turns out that my use case is best served with an external GPU then I’d probably sell the Desktop and buy a cheaper mobo.
Interestingly someone on this forum is experimenting with GPU over USB and is not having a whole lot of luck. So I may need to experiment in this direction too.
Update: the transfer speed numbers for PCIe was completely wrong here, invalidating my core assumptions.
The main hold up with egpus is that not all the enclosures can use all that bandwidth. Mine is doing about half the bandwidth (th3p4g3). Im sure there probably are newer enclosures that can hit the 5GB/s.
Wow, good info - thanks Xal. Every time I think I’ve accounted for all the pitfalls, another one emerges!
For any distro of Linux, do you think it is possible to determine whether an enclosure would do the full bandwidth before purchasing, or does one have to “buy and fiddle”? I’ve been reading this impressive list of eGPU builds, but I guess that an item appearing on the list means that it worked at least to some degree, rather than worked perfectly.
Has “only” 2.5 GB/s been a problem for you in practice? It is, at least, much faster than the internal port.
Not really a problem. It add a second or two to each loading or swapping operation.
Main issue currently for me is, the ability to easily load specific apps to specific gpus.
When i plug the nvidia one the whole system starts defaulting to it. It always takes the “first” spot. With containers this is not an issue, but sometimes i want to pop open lmstudio and it is only see the nvidia (which is not helpful as i cant tap into the 128gb of ram and the igpu).
It is also an issue with gaming. Once the nvidia is detected steam defaults to it. At which point only unplugging helps. I would like to do some things on parallel but i can’t unless o figure a way to make the igpu the “default” and only use the egpu explicitly when needed.
Super; for single-model work, that would be fine. I did find some advice that for Mixture of Experts systems, where (partial) models need to be swapped in/out of VRAM, transfer speed might become a problem if many swaps are required. But I wonder that by the time I get around to this level of complexity, the kernel might be better in this regard anyway.
I load a single small model to avoid that issue. I typically run a small model to “plan” and a large one in the igpu to code. They talk to each other. Currently enjoying the world cup but the goal is to build a harness to run LLMs in a way that they can “collab” like that.
If you get an egpu enclosure just look for one that has a controller that can do 40Gbps, to get the full bandwidth. Mine is a few years old. Bought it to use with a laptop for gaming. Now days it sees more use with LLMs
That’s not possible. The internal interface is PCIe 4.0 x4. TB4/USB4 are much the same as TB3 which uses PCIe 3.0 x4 and has overhead that the internal connection does not. The internal connection is twice as fast as external USB4 connection before accounting for protocol overhead. What is your source?
Damn, thanks for the catch - it was an AI hallucination from Claude Haiku. Apologies; I should have double-checked it before posting. The robot’s mea culpa is below.
Hmmph! Claude, from you earlier I got data transfer rates of PCIe 4.0 x4 = 1GB/sec, USB4 = 5GB/sec. I learn from another AI:
It looks like you have agreement for USB4, but you’re in disagreement by a factor of 8 for PCIe 4.0 x 4. Would you double-check?
You’re absolutely right to call that out. I made a serious error in my PCIe 4.0 x4 calculation.
Let me work through it correctly:
PCIe 4.0 specifications:
Per lane: 2 Gbps (gigabits per second)
Per lane in bytes: 2 Gbps ÷ 8 = 0.25 GB/s
x4 configuration: 4 lanes × 0.25 GB/s = 1 GB/s
Wait—that’s what I said. But your source says 7.88 GB/s. Let me reconsider what I might be missing.
Ah—I see the discrepancy. The encoding overhead in PCIe uses 8b/10b encoding (or 128b/130b in PCIe 5.0+), which means not all bits are data:
PCIe 4.0 raw speed per lane: 16 Gbps (not 2 Gbps)
With 8b/10b encoding: 16 Gbps × (8/10) = 12.8 Gbps effective
Per lane in bytes: 12.8 Gbps ÷ 8 = 1.6 GB/s
x4: 1.6 GB/s × 4 = 6.4 GB/s (close to your source’s 7.88 GB/s)
I was confusing the encoded vs. raw specifications and made a factor-of-8 error. You’ve caught a meaningful mistake that actually changes the analysis:
This means PCIe 4.0 x4 is actually ~1.6× faster than USB4, not 5× slower. That substantially strengthens the case for the Framework Desktop’s internal interface—the bandwidth advantage flips entirely.
Thank you for fact-checking. Do you want me to reconsider the MoE streaming scenario with these corrected numbers?
