Virtualization on FW16's Ryzen 9 (7940HS)

@CodeMichael, thank you for your research. It is truly helpful and I know I will finally have my perfect non-dual booting configuration with close to native performance inside VMs. I mostly use Linux but once in a while I have to work with Adobe or Microsoft products for a few projects and I hate having to keep a Windows partition for dual booting. I do have a Windows VM but without graphics pass-through, performance is poor. So when I can afford a FW16 I will be ready for that perfect setup. Thanks a lot!

I’ve got my notes about my nix setup published, and I discovered how to successfully detach and re-attach the the dgpu without (seemingly) any major issues.

4 Likes

Awesome! Thank you for the effort. I’m considering trying out nix with this laptop. Might steal your config for it :smiley:

Hey there,

I m running Ubuntu 22.04 with recommended OEM-Kernel on my Framework 16.
I am trying to forward the 7700S to my Windows VM. Therefore I forwarded the GPU to the VM using PCIE-forward in VirtManager.

Windows detects the card and i can install the AMD driver but im stuck with error 43 in device manager.

I checked lspci -nnv and it appears to be forwarded just fine. Do i need to forward another device to make it work?
Thanks for your help

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7480] (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Device [f111:0007]
	Flags: bus master, fast devsel, latency 0, IRQ 49, IOMMU group 16
	Memory at 8400000000 (64-bit, prefetchable) [size=8G]
	Memory at 8600000000 (64-bit, prefetchable) [size=256M]
	I/O ports at a000 [size=256]
	Memory at 90d00000 (32-bit, non-prefetchable) [size=1M]
	Expansion ROM at 90e20000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu

03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
	Subsystem: Device [f111:0005]
	Flags: bus master, fast devsel, latency 0, IRQ 132, IOMMU group 17
	Memory at 90e00000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

I address that in my notes (link above). There are additional links in those notes that talk about this issue directly if you need further input.

<features>
  <hyperv mode="custom">
    <vendor_id state="on" value="someRandomBits"/>
  </hyperv>
  <kvm> <hidden state="on"/> </kvm>
</features>

Thank you for your reply. For some reason including the hidden tag and forwarding the gpu at the same time let’s the vm reboot every few seconds. That causes my system to crash since i am unable to let the host ignore the gpu. I tried

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio-pci.ids=1002:7480,1002:ab30 pci-stub.ids=1002:7480,1002:ab30 kvm_amd.npt=1 kvm_amd.avic=1 quiet splash "

but still get

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7480] (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Device [f111:0007]
	Flags: bus master, fast devsel, latency 0, IRQ 129, IOMMU group 16
	Memory at 8400000000 (64-bit, prefetchable) [size=8G]
	Memory at 8600000000 (64-bit, prefetchable) [size=256M]
	I/O ports at a000 [size=256]
	Memory at 90d00000 (32-bit, non-prefetchable) [size=1M]
	Expansion ROM at 90e20000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
	Subsystem: Device [f111:0005]
	Flags: bus master, fast devsel, latency 0, IRQ 141, IOMMU group 17
	Memory at 90e00000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

after booting the host.
I guess that is the first problem I have to figure out before i can adress the VM rebooting issue. I am trying to adapt your solution to Ubuntu, so far with limited success :smiley:

you need to make sure to load the vfio modules before the amdgpu module.

Got my crash problem solved. As it turns out i did not have anything to do with what I am currently working on. So i added the mentioned xml-tags, but that did not change anything. I managed to isolate the gpu. In addition to my changes in GRUB i changed the initramfs:

  1. Create the conf-File
sudo nano /etc/modprobe.d/vfio.conf
  1. Fill it with:
options vfio-pci ids=1002:7480,1002:ab30
softdep amdgpu pre: vfio-pci
  1. Update and reboot
sudo update-initramfs -c -k $(uname -r)

  1. Check with lspci -v
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 7480 (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Device f111:0007
	Flags: fast devsel, IRQ 255, IOMMU group 16
	Memory at 8400000000 (64-bit, prefetchable) [disabled] [size=8G]
	Memory at 8600000000 (64-bit, prefetchable) [disabled] [size=256M]
	I/O ports at a000 [disabled] [size=256]
	Memory at 90d00000 (32-bit, non-prefetchable) [disabled] [size=1M]
	Expansion ROM at 90e20000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu

03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device ab30
	Subsystem: Device f111:0005
	Flags: fast devsel, IRQ 255, IOMMU group 17
	Memory at 90e00000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

GPU isolated!

VM-Config:

domain type="kvm">
  <name>Windows11</name>
  <uuid>681d0b64-7d1b-4b73-a499-55c851919c46</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/11"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">32498688</memory>
  <currentMemory unit="KiB">32498688</currentMemory>
  <vcpu placement="static">16</vcpu>
  <os firmware="efi">
    <type arch="x86_64" machine="pc-q35-6.2">hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode="custom">
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <runtime state="on"/>
      <synic state="on"/>
      <stimer state="on">
        <direct state="on"/>
      </stimer>
      <reset state="on"/>
      <vendor_id state="on" value="KVM Hv"/>
      <frequencies state="on"/>
      <reenlightenment state="on"/>
      <tlbflush state="on"/>
      <ipi state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on"/>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/matthias/Schreibtisch/Win11_23H2_German_x64v2.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native" discard="unmap"/>
      <source dev="/dev/nvme1n1"/>
      <target dev="vda" bus="virtio"/>
      <boot order="2"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:4c:e7:52"/>
      <source network="default"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <channel type="unix">
      <target type="virtio" name="org.qemu.guest_agent.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="2"/>
    </channel>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <tpm model="tpm-crb">
      <backend type="emulator" version="2.0"/>
    </tpm>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
      <image compression="off"/>
    </graphics>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
    <audio id="1" type="spice"/>
    <video>
      <model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="1"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>


Ideas?

2 Likes