Direct Thunderbolt networking between two Desktops?

Framework Desktop
1

I’m trying to get Thunderbolt networking functional between two Framework Desktops (both 128GB 395+) running Fedora 43, with the latest firmware.

I’ve got what’s supposed to be a certified TB4 cable connecting the two, but ‘boltctl list’ shows nothing. ‘boltctl list -a’ shows both the hosts (ports)’. ‘modprobe thunderbolt-net’ succeeds, but no networking device appears. Dmesg shows ‘ACPI: bus type thunderbolt registered’, but that’s it. ‘dmesg -w’ shows no state change when I unplug/replug the cable. It’s like it’s not there.

I’m getting an Anker certified TB4 cable to test Sunday, because there’s no setting in the firmware to enable or disable anything related to those ports or TB, and the OS seems to be behaving properly, it’s just not seeing the connection or giving me a network interface.

Has anyone gotten this working successfully?

2

Is there some kind of PCIE power management that Desktop uses here?
Haven’t tried this with the Desktop (mainly because I haven’t found any improvements speed-wise for llama.cpp RPC) but I also have two Minisforum MAX machines (also running Fedora 43) I did try it with and I had hard time getting them to connect via Thunderbolt as well. There it turned out that there wasn’t enough power supplied to those ports. I had to disable power management for PCIE in GRUB to make them actually work:
sudo grubby --update-kernel=ALL --args=‘pcie_port_pm=off’

1 Like
3

Thanks. Tried that, with no success.

4

Received a new, certified Thunderbolt cable and tested with it.

Still cannot get the system to acknowledge Thunderbolt networking in any way. Modprobe thunderbolt-net has no impact at all.

5

Correction: I powered down the system entirely, waited 10 minutes, and powered back up.

I then restarted bolt.service, loaded thunderbolt-net, and I now see thunderbolt0 in ifconfig output.

I’ll have to experiment some more, but perhaps bolt.service is getting started before the thunderbolt connection is detected.

6

iperf3 testing shows I’m getting a solid 10Gbps throughput over TB4 now.

EDIT: And iperf3 maxes out around 9-12Gbps, so I suspect I’m hitting an iperf3 limit, not a TB4 one.

3 Likes
7

I’m also getting 10 Gbps. I read somewhere that was the limit although it seems low for USB4. It may depend on how many pcie lanes are dedicated to this port.

I’m using this expensive cable: Amazon.com: Cable Matters [Intel Certified 80Gbps Thunderbolt 5 Cable - 1m / 3.3ft, Supports up to 120Gbps Bandwidth Boost and 240W Charging, Compatible with Thunderbolt 4, USB4, and USB C, Black : Electronics

Edit: Just noticed dmesg has this single line: TCP: thunderbolt0: Driver has suspect GRO implementation, TCP performance may be compromised.

No idea what thats about.