Has anyone managed to pass through a Thunderbolt/USB4 port to a VM?
Not the individual devices, but the whole port. The goal is, that everything that happens on that (physical) port is managed by the VM and not by the host (including plugging in and removing devices).
After digging into this for a while, I concluded that this is probably not possible (yet)?
This is what I tried:
After identifying the port (I’m using Framework 13 AMD):
$ boltctl domains -v
● domain1 3ab63804-b1c3-fb1e-ffff-ffffffffffff
├─ online: yes
├─ syspath: /sys/devices/pci0000:00/0000:00:08.3/0000:c3:00.6/domain1
├─ bootacl: 0/0
└─ security: iommu+user
├─ iommu: yes
└─ level: user
I can identify consumers:
$ find "/sys/devices/pci0000:00/0000:00:08.3/0000:c3:00.6/" -name "consumer\*" -type l
/sys/devices/pci0000:00/0000:00:08.3/0000:c3:00.6/consumer:pci:0000:00:04.1
/sys/devices/pci0000:00/0000:00:08.3/0000:c3:00.6/consumer:pci:0000:c3:00.4
$ ls /sys/bus/pci/devices/0000:c3:00.6/iommu_group/devices0000:c3:00.6$ ls /sys/bus/pci/devices/0000:00:04.1/iommu_group/devices0000:00:04.0 0000:00:04.1$ ls /sys/bus/pci/devices/0000:c3:00.4/iommu_group/devices0000:c3:00.4
Details for these devices:
$ lspci -k
...
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14ea
00:04.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel
Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
Kernel driver in use: pcieport
...
c3:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15c1
Subsystem: Framework Computer Inc. Device 0006
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
...
c3:00.6 USB controller: Advanced Micro Devices, Inc. [AMD] Pink Sardine USB4/Thunderbolt NHI controller #2
Subsystem: Framework Computer Inc. Device 0006
Kernel driver in use: thunderbolt
Kernel modules: thunderbolt
Passing through c3:00.4 and c3:00.6 works just fine for “normal” USB devices, but not for USB-4/TB4/eGPU type of things.
If I plug in such a device, it neither shows up on the host nor the guest. There is only an error:
$ journalctl -f
kernel: ucsi_acpi USBC000:00: unknown error 256
kernel: ucsi_acpi USBC000:00: GET_CABLE_PROPERTY failed (-5)
If I don’t attach these devices or unbind them and reattach them to the host, the devices show up on the host just fine (I’m using Pocket AI RTX A500 here):
IOMMU Group 5:
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
00:04.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel [1022:14ef]
62:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
63:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
63:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
63:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
64:00.0 3D controller [0302]: NVIDIA Corporation GA107 [RTX A500 Embedded GPU] [10de:25fb] (rev a1)
92:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge DD 2018] [8086:15f0] (rev 06)
I could try to attach all these devices individually, but these defeats the purpose of what I want to achieve here.
If no devices are connected, only the bridges are in this group:
IOMMU Group 5:
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
00:04.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel [1022:14ef]
00:04.1 (PCI bridge) says Kernel driver in use: pcieport, so I was thinking maybe this bridge can be attached to the VM, but this doesn’t seem to be the intended way of doing things.
Virt manager says “Non-endpoint PCI devices cannot be assigned to guests”. If I try to do it anyway, it fails:
$qemu-system-x86_64 -boot d -cdrom "linux.iso" -m 512 -device vfio-pci,host=0000:00:04.1
qemu-system-x86_64: -device vfio-pci,host=0000:00:04.1: vfio 0000:00:04.1: Could not open '/dev/vfio/5': No such file or directory
Further investigating shows, that
$echo "0x1022 0x14ef" > /sys/bus/pci/drivers/vfio-pci/new_id
does not create a file in /dev/vfio. Also, there is no error in journalctl.
I did some digging into old versions of the Arch Wiki:
As it seems, KVM could pass through arbitrary devices using the pci-stub-driver. That would then be used in qemu with -device pci-assign,host=xx:xx.x. The driver still exists, but support for it has been dropped in kvm and qemu. So there is only vfio-pci left. The kernel docs just say “vfio-pci does not currently support PCI bridges”, without giving any further details about why this is not supported or if it will be supported after all. As it seems, support for passing through bridges was lost when the kvm pass-through was deprecated in favor of VFIO. Since I haven’t tested this, so I can’t tell if passing through the bridge is what would be needed to achive this goal.
So I’m somewhat stuck what to do next.
Framework 13 AMD
6.10.13-3-MANJARO
Compiled against library: libvirt 10.7.0
Using library: libvirt 10.7.0
Using API: QEMU 10.7.0
Running hypervisor: QEMU 9.1.0