Which Linux distro are you using? Gentoo
Which release version? Rolling release, recent
Which kernel are you using? 6.19.0, self-compiled
Which BIOS version are you using? 3.19
Which Framework Laptop 13 model are you using? 12th Gen Intel® Core™
Other hardware:
Razer Core X (JHL6340)
Sonnettech Echo 5
Nvidia RTX3090
Hi.
Trying to hook up my 5th GPU via a USB4 hub, but failing because the kernel cannot fit/move the required MMIO space anywhere.
I just know enough to sorta/kinda explain the issue here. I know the kernel can ‘reorganize’ and shuffle stuff around to fit the required hardware MMIO-space into the overall /proc/iomem. And this works for 4 GPUs, one in each port. As evidenced by the following dmesg snippet:
[ 553.043784] PCI: No. 2 try to assign unassigned res
[ 553.045504] pcieport 0000:00:07.1: bridge window [io size 0x200000]: can’t assign; no space
[ 553.047238] pcieport 0000:00:07.1: bridge window [io size 0x200000]: failed to assign
[ 553.048935] pcieport 0000:00:07.1: bridge window [io size 0x200000]: can’t assign; no space
[ 553.050615] pcieport 0000:00:07.1: bridge window [io size 0x200000]: failed to assign
[ 553.052271] pci 0000:2b:00.0: bridge window [io size 0x200000]: can’t assign; no space
[ 553.053943] pci 0000:2b:00.0: bridge window [io size 0x200000]: failed to assign
[ 553.055614] pci 0000:2b:00.0: bridge window [io size 0x200000]: can’t assign; no space
[ 553.057292] pci 0000:2b:00.0: bridge window [io size 0x200000]: failed to assign
[ 553.058950] pci 0000:2c:01.0: bridge window [io size 0x200000]: can’t assign; no space
[ 553.060613] pci 0000:2c:01.0: bridge window [io size 0x200000]: failed to assign
[ 553.062263] pci 0000:2c:01.0: bridge window [io size 0x200000]: can’t assign; no space
[ 553.063908] pci 0000:2c:01.0: bridge window [io size 0x200000]: failed to assign
[ 553.065544] pci 0000:2d:00.0: BAR 5 [io size 0x0080]: can’t assign; no space
[ 553.067177] pci 0000:2d:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 553.068787] pci 0000:2d:00.0: BAR 5 [io size 0x0080]: can’t assign; no space
[ 553.070385] pci 0000:2d:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 553.071981] pci 0000:2c:01.0: PCI bridge to [bus 2d]
[ 553.073593] pci 0000:2c:01.0: bridge window [mem 0x6e000000-0x79ffffff]
[ 553.075219] pci 0000:2c:01.0: bridge window [mem 0x6020000000-0x6037ffffff 64bit pref]
[ 553.076856] pci 0000:2b:00.0: PCI bridge to [bus 2c-2d]
[ 553.078485] pci 0000:2b:00.0: bridge window [mem 0x6e000000-0x79ffffff]
[ 553.080078] pci 0000:2b:00.0: bridge window [mem 0x6020000000-0x6037ffffff 64bit pref]
[ 553.078485] pci 0000:2b:00.0: bridge window [mem 0x6e000000-0x79ffffff]
[ 553.080078] pci 0000:2b:00.0: bridge window [mem 0x6020000000-0x6037ffffff 64bit pref]
[ 553.081631] pcieport 0000:00:07.1: PCI bridge to [bus 2b-53]
[ 553.083184] pcieport 0000:00:07.1: bridge window [mem 0x6e000000-0x7a1fffff]
[ 553.084741] pcieport 0000:00:07.1: bridge window [mem 0x6020000000-0x603bffffff 64bit pref]
[ 553.086379] pcieport 0000:2b:00.0: enabling device (0000 -> 0002)
[ 553.088125] pcieport 0000:2c:01.0: enabling device (0000 -> 0002)
[ 553.089962] nvidia 0000:2d:00.0: enabling device (0000 -> 0002)
Adding a Sonnettech Echo 5 (USB4 3-port hub), I can physically connect my 5th GPU. But I cannot actually make use of it. The best I have achieved is the PCI subsystem accepting that there are 5 GPUs, but the nvidia driver cannot initialize the 5th GPU.
Do note that I am not concerned about system/GPU bandwidth.
My current linux command line contains:
pci=realloc=on,hpmmiosize=256M,hpiosize=2M,hpmemsize=1G,hpbussize=0x33,pcie_scan_all,big_root_window pcie_port_pm=off pcie_aspm.policy=performance thunderbolt.clx=0
Anyone got any good explanations/theories/ideas to offer here?
Or maybe even a personal relationship with Bjørn H or Ilpo J? 