What PCIE network card for 40+ GB/s

Framework Desktop
, , ,
18

This is ai generated but I actually have some networking knowledge, so I did guide it. Please let me know how well it works! I would recommend using ipv6 on it for the addressing since it uses SlAAC to do stuff. AI solution follows

To get raw speed (40Gbps+) between two Strix Halos, you must set the IOMMU to “Passthrough” mode. This allows the USB4 controller to write directly to RAM without the CPU checking every permission bit.

Run on BOTH Strix Halos:

  1. Open Grub config:

    sudo nano /etc/default/grub

  2. Find GRUB_CMDLINE_LINUX_DEFAULT and append:

    iommu=pt amd_iommu=on

  3. Update Grub and Reboot:

    Bash

    sudo update-grub  # or 'grub-mkconfig -o /boot/grub/grub.cfg' on Arch/Alpine
sudo reboot

2. The “Forgiving” Kernel Tuning

The Strix Halo is fast, but balance-rr is chaotic. You must tune the TCP stack to accept out-of-order IPv6 packets without panicking.

Run on BOTH computers:

# Allow massive packet reordering (Essential for Bond Mode 0)
sudo sysctl -w net.ipv4.tcp_reordering=127

# Use BBR (Better for high-throughput/variable-latency links)
sudo sysctl -w net.core.default_qdisc=fq
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

# Maximize Memory Buffers (Crucial for >25Gbps)
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 67108864"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 67108864"

(Note: I doubled the buffer sizes from the previous guide because two Strix Halos can actually fill them.)

3. The IPv6-Only Bond Configuration

This creates a single, non-load-balanced pipe (bond0) that stripes packets across both cables.

Run on BOTH computers:

# 1. Reset links
sudo ip link set thunderbolt0 down
sudo ip link set thunderbolt1 down

# 2. Create Bond (Mode 0 = Round Robin Striping)
sudo ip link add bond0 type bond mode balance-rr miimon=100

# 3. Enable Jumbo Frames (Required for CPU efficiency)
sudo ip link set bond0 mtu 65528

# 4. Enslave physical ports
sudo ip link set thunderbolt0 master bond0
sudo ip link set thunderbolt1 master bond0

# 5. HARD DISABLE IPv4 (Forces IPv6 only)
sudo sysctl -w net.ipv4.conf.bond0.disable_ipv4=1

# 6. Bring everything up
sudo ip link set thunderbolt0 up
sudo ip link set thunderbolt1 up
sudo ip link set bond0 up

4. Assigning Addresses (IPv6 ULA)

We use fd00:: (Unique Local Address) so it behaves like a static private LAN.

On Strix Halo A:

Bash

sudo ip -6 addr add fd00::1/64 dev bond0

On Strix Halo B:

Bash

sudo ip -6 addr add fd00::2/64 dev bond0
1 Like