Thunderbolt domain topology and BIOS options for a dual-eGPU Linux configuration - Framework Laptop 13 (Core Ultra Series 3 / Panther Lake)

Framework Laptop 13
,
1

I am evaluating the Framework Laptop 13 Pro (Core Ultra Series 3 mainboard)
for a specific workload and need engineering-level answers before ordering.
Please escalate if needed - these are hardware architecture and firmware
questions, not usage questions.

Intended use: two Thunderbolt eGPU enclosures connected SIMULTANEOUSLY to the
same laptop, running Ubuntu 26.04 LTS, for multi-GPU LLM inference. One
enclosure holds an NVIDIA RTX 5060 Ti 16GB, the other an RTX 4060 Ti 16GB.
I understand this is not a supported or tested configuration. I am not asking
you to support it. I am asking for the hardware facts so I can judge whether
it is feasible.

  1. THUNDERBOLT DOMAIN MAPPING
    The Core Ultra Series 3 SoC exposes two Thunderbolt Native Host Interfaces
    (two separate Thunderbolt domains). All four Expansion Card slots are
    advertised as Thunderbolt 4.
    Which physical Expansion Card slots are wired to which NHI / domain?
    Specifically: which two slots must I use so that each eGPU lands on a
    different Thunderbolt domain, rather than sharing one?
    If you can supply the PCI addresses or a mainboard block diagram showing
    the slot-to-root-port mapping, that is exactly what I need.

  2. PCIe TUNNELING PER PORT
    Is PCIe tunneling enabled in firmware on all four Expansion Card slots, or
    only on some? Are there any slots where the port does USB and DisplayPort
    but will not create a PCIe tunnel?

  3. SIMULTANEOUS PCIe TUNNELS
    Can the firmware allocate PCIe resources for two PCIe tunnels at once, each
    carrying a discrete GPU with a large 64-bit prefetchable BAR? Is there a
    known limit on the number of concurrent PCIe tunnels?

  4. BIOS OPTION: ABOVE 4G DECODING
    Is “Above 4G Decoding” (64-bit BAR support) enabled, and is it exposed as a
    user-configurable BIOS setting? If it is always-on and not exposed, please
    confirm that explicitly.

  5. BIOS OPTION: RESIZABLE BAR
    Is Resizable BAR enabled by default, and can it be DISABLED from the BIOS?
    I am aware of open issue #37 in the SoftwareFirmwareIssueTracker regarding
    ReBAR over Thunderbolt. Note that I need the ability to turn ReBAR OFF - a
    published working dual-eGPU Linux configuration required disabling it for
    the NVIDIA card to initialise. Is there any plan to expose this toggle?

  6. BIOS OPTION: THUNDERBOLT PRE-BOOT / PCIe BEHIND THUNDERBOLT
    Does the BIOS execute PCIe Option ROMs for devices behind Thunderbolt during
    POST, so that the firmware enumerates a Thunderbolt-attached GPU and
    allocates its PCIe resources before the OS boots?
    Business-class machines expose this as “Thunderbolt pre-boot modules” or
    “PCIe behind TBT”. Is there an equivalent on Framework, either as a setting
    or as always-on behaviour?

  7. BIOS OPTION: THUNDERBOLT SECURITY LEVEL
    What Thunderbolt security level does the firmware set - none, user, secure,
    dponly, usbonly, or nopcie? Is it user-configurable? (dponly, usbonly and
    nopcie all prevent PCIe tunneling entirely.)

  8. POWER
    If two Expansion Card slots are occupied by self-powered eGPU enclosures,
    can the laptop still charge over a third slot at full wattage? Is there a
    power or thermal budget interaction I should know about when two Thunderbolt
    tunnels are active at once?

  9. FIRMWARE DELIVERY
    Do you publish Thunderbolt controller and retimer firmware to LVFS, so it
    can be updated with fwupd on Linux without booting Windows?

  10. HARDWARE IDs
    Can you confirm the PCI hardware IDs of the Thunderbolt NHIs on this
    mainboard, so I can verify a machine matches the expected topology?

2

Phew, this is a detailed shopping-list!

I’d say logging a support ticket is the best way to get all that information, though frankly I am not sure even a Support team would have all that info. I’d probably just buy the unit, and then using the 30-Day Return Policy if it does not match your required spec.

3

Going by previous generations of Intel: If the entire notebook has more than just 2 USB4 ports, then usually a dual-port will be from one USB4 controller. That is because they share DP inputs and if you repurpose the second USB4 port of a controller for some display out purpose, you take away 1 DP tunnel from the other port, making it no longer TB4 certifiable (which requires 2 DP tunnels).

For Framework it always was 1 controller for the left, 1 for the right.

But also, Why do you want separate USB4 controllers for each eGPU? The controllers only manage the setup. The actual PCIe traffic is, per-USB4 spec and in practice from separate PCIe Root Ports, per USB4 port. So it really does not matter if you use 2 ports on the same USB4 controller or on different ones. There are 4 separate PCIe Root ports for the PCIe tunnels and they share some undocumented amount of bandwidth together and each of them may have its own limit. Since 12th gen we could show that the per-port bandwith limit is above what you can do with 1 USB4 40G connection. So I am pretty sure if there are no other things relating to USB4 management traffic, it really does not matter.

This is regulated by Windows HLK requirements and by TB4. They all do. Maybe there is an option to disable it, but it would not be TB4 without it and we have not seen any USB4 host (so non-TB4 certified) that left it out.

PCIe tunnels in USB4 are 1 per DFP. Thats it. They follow hub-topology, just like USB3 does. If you have a USB4 hub that will need to include another PCIe switch to handle its multiple DFP from the single PCIe tunnel it will get from upstream. So this is not a thing that exists or matters.

My 12th gen FW13 could. After some firmware updates it reserved 64 GiB for 3 USB4 ports. Only 1 of them was stuck on the earlier, much lower reservation. My AMD Strix Point FW13 reserves 128 GiB for each of its 2 ports. So depends on how much you need. They probably only design for consumer GPUs and also, the address space is pre-allocated for each port. Have not tried if you can reallocate that with linux. Also no idea if linux messes / redoes some of that independent of the BIOS. At least hot …

Pretty damn sure it is by default. But there has not been a BIOS option to toggle and I would not expect it.

No option seen thus far. My AMD Strix Point FW13 board has it enabled and its working with a RTX 30.

Also, why do you need “large 64 bit prefetchable BARs” if you want ReBAR off? If the cards just have large BARs, they don’t need ReBAR. If they are consumer GPUs, without ReBAR you would get the backwards compatible small BARs.

It does. FW rather recently added a BIOS option to disable that across the various generations, though its misnamed as sth. for “measuring into a TPM PCR reg”, but in my experiment it blocked loading boot roms behind USB4. Because booting them will impact measured boot and trip bitlocker checks if the GPU is only occasionally attached at boot.

Whether the BIOS initializes the GPU enough to actually use pre-OS, I don’t know, I just know it executes the boot ROM enough to change the measured boot.

That is not a thing with USB4 anymore. Reminder, TB4 is just Intel certifying and marketing USB4. Those TB Security Levels were created to secure the old TB1-3 that were introduced before PCs had actual DMA security. USB4 requires DMA security so has no purpose for those levels. And the other thing is just disabling PCIe or TB/USB4 entirely.

Why would it not? self-powered USB-C equipment would not impact this at all and the FW13 boards always could support higher PD input then the official power supply (older ones 100W, the Panther Lake one 140W). Whether your use case might heat up the system enough were it might throttle charging, I don’t know.

You know the processor generation. You can look those up on PCI ids. You expecting somebody is lying about the processor generation AND hiding that some of the ports are unconnected and nobody has noticed? Also, the “Thunderbolt NHI” is a fun linux naming artifact. Because those are just the USB4 controllers. The name in those PCI ID databases is just irrelevant to anything and the kernel driver is only “thunderbolt” for historic reasons.

2 Likes
4

thank u reply appreciate the effort u went through! all the best!

5

I have done dual egpu stuff on my 7480u fw13 before, worked surprisingly well even daisychained.

Quite curious how your setup turns out.