FYI I have a desktop PC I built myself with AMD 5800x3d. I run ubuntu on it as host, and a windows KVM guest as my gaming machine, with GPU & ssd passthrough.
Most notably, nested virtualization seems to be either not working, not supported or very slow. I don’t remember what was the exactly conclusion back then when I did that experiment, but it seems to me that nested virtualization on amd is not as good as on intel.
Other than that I didn’t encounter any amd-specific issues. Qemu upgrade broke my setup once going from 5 to 6 but it’s not AMD related.
1 Like
I’m a little distracted, but getting virtualization working in on the short list here. I’m going back through my notes around getting everything set up and doing what I can to move that whole process into my Nix configuration, so I’m being a little slow with getting things working. I’ll try to work on it this evening and this weekend and get back to you with results.
Thank you for your efforts! There’s no real rush so take your time. Please remember this is just a convenience for the rest of us, lol. I’ve ordered a framework 16 but it’s a couple months out still I think.
1 Like
Figured I’d go ahead an post my current blocker and see if anyone had some ideas.
As you can from the above post the 7700S has the following devices: 1002:7480 and 1002:ab30 I’ve added those to the vfio-pci.ids and pci-sub.ids kernel parameter lists, but when I boot the vga device is using the amdgpu driver, rather than the vfio-pci driver. My understanding is that the vfio-pci.ids list should prevent the amdgpu from grabbing it, but that’s one of the earliest modules available during boot if I remove amdgpu from the initrd kernel modules its not booting at all.
1 Like
Nevermind, I forgot to put the vfio and vfio-pci modules first in the list. Work continues…
1 Like
Discourse doesn’t let me post more than 3 in a row, so I’m keeping this most recent item updated until and unless the conversation kicks off again.
Running in an ubuntu VM through sunshine/moonlight.
And here’s with windows 11 VM with sunshine
I have also gotten Looking-glass working.
In general things seem to be working just fine. I’ll pull together some documentation along with my nix configs so that you can reproduce if you like. I guess I’m taking requests if anyone has anything they’d like me to try or info they’d like me to report back on.
6 Likes
Nice! looking great! Seems you’re already doing what I want to do. What’s the base OS? Ubuntu? Mind sharing a guide on how you accomplished this when done? Or at least the general steps?
It sounds like he’s using NixOS, but I’m sure he can elaborate for us with the docs he’s writing. I’m planning on using QEMU in Kinoite or Solus, but Nix is all the rage.
Yep. Not sure how I glossed over the word nix twice, but sure enough, there it is.
Thanks for all the info so far @CodeMichael. You’re making me excited for my batch 17 machine.
Kinoite sounds neat. I’ve never gotten around to playing with Fedora (for more than a few min) but the whole Silverblue effort sounds like something I would be interested in (might have to load up sway atomic in a VM). Don’t judge me too harshly for using Nix.
Glad to do it. Lovely to be chatting with people with similar interests. If you have something you’d like to me to test for you let me know.
I’ll try to clean up my notes and make a legible document, but I suspect you all won’t need much of it, I really didn’t run into any significant challenges other than my own silliness. The board layout was the biggest unknown. I think the last checkbox on my own list for virtualization was to see if I could pass USB via PCI. I will try to knock that out this week.
1 Like
@CodeMichael, thank you for your research. It is truly helpful and I know I will finally have my perfect non-dual booting configuration with close to native performance inside VMs. I mostly use Linux but once in a while I have to work with Adobe or Microsoft products for a few projects and I hate having to keep a Windows partition for dual booting. I do have a Windows VM but without graphics pass-through, performance is poor. So when I can afford a FW16 I will be ready for that perfect setup. Thanks a lot!
I’ve got my notes about my nix setup published, and I discovered how to successfully detach and re-attach the the dgpu without (seemingly) any major issues.
4 Likes
Awesome! Thank you for the effort. I’m considering trying out nix with this laptop. Might steal your config for it 
Hey there,
I m running Ubuntu 22.04 with recommended OEM-Kernel on my Framework 16.
I am trying to forward the 7700S to my Windows VM. Therefore I forwarded the GPU to the VM using PCIE-forward in VirtManager.
Windows detects the card and i can install the AMD driver but im stuck with error 43 in device manager.
I checked lspci -nnv and it appears to be forwarded just fine. Do i need to forward another device to make it work?
Thanks for your help
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7480] (rev c1) (prog-if 00 [VGA controller])
Subsystem: Device [f111:0007]
Flags: bus master, fast devsel, latency 0, IRQ 49, IOMMU group 16
Memory at 8400000000 (64-bit, prefetchable) [size=8G]
Memory at 8600000000 (64-bit, prefetchable) [size=256M]
I/O ports at a000 [size=256]
Memory at 90d00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at 90e20000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
Subsystem: Device [f111:0005]
Flags: bus master, fast devsel, latency 0, IRQ 132, IOMMU group 17
Memory at 90e00000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
I address that in my notes (link above). There are additional links in those notes that talk about this issue directly if you need further input.
<features>
<hyperv mode="custom">
<vendor_id state="on" value="someRandomBits"/>
</hyperv>
<kvm> <hidden state="on"/> </kvm>
</features>
Thank you for your reply. For some reason including the hidden tag and forwarding the gpu at the same time let’s the vm reboot every few seconds. That causes my system to crash since i am unable to let the host ignore the gpu. I tried
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio-pci.ids=1002:7480,1002:ab30 pci-stub.ids=1002:7480,1002:ab30 kvm_amd.npt=1 kvm_amd.avic=1 quiet splash "
but still get
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7480] (rev c1) (prog-if 00 [VGA controller])
Subsystem: Device [f111:0007]
Flags: bus master, fast devsel, latency 0, IRQ 129, IOMMU group 16
Memory at 8400000000 (64-bit, prefetchable) [size=8G]
Memory at 8600000000 (64-bit, prefetchable) [size=256M]
I/O ports at a000 [size=256]
Memory at 90d00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at 90e20000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: amdgpu
Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
Subsystem: Device [f111:0005]
Flags: bus master, fast devsel, latency 0, IRQ 141, IOMMU group 17
Memory at 90e00000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
after booting the host.
I guess that is the first problem I have to figure out before i can adress the VM rebooting issue. I am trying to adapt your solution to Ubuntu, so far with limited success
you need to make sure to load the vfio modules before the amdgpu module.
Got my crash problem solved. As it turns out i did not have anything to do with what I am currently working on. So i added the mentioned xml-tags, but that did not change anything. I managed to isolate the gpu. In addition to my changes in GRUB i changed the initramfs:
- Create the conf-File
sudo nano /etc/modprobe.d/vfio.conf
- Fill it with:
options vfio-pci ids=1002:7480,1002:ab30
softdep amdgpu pre: vfio-pci
- Update and reboot
sudo update-initramfs -c -k $(uname -r)
- Check with lspci -v
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 7480 (rev c1) (prog-if 00 [VGA controller])
Subsystem: Device f111:0007
Flags: fast devsel, IRQ 255, IOMMU group 16
Memory at 8400000000 (64-bit, prefetchable) [disabled] [size=8G]
Memory at 8600000000 (64-bit, prefetchable) [disabled] [size=256M]
I/O ports at a000 [disabled] [size=256]
Memory at 90d00000 (32-bit, non-prefetchable) [disabled] [size=1M]
Expansion ROM at 90e20000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device ab30
Subsystem: Device f111:0005
Flags: fast devsel, IRQ 255, IOMMU group 17
Memory at 90e00000 (32-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
GPU isolated!
VM-Config:
domain type="kvm">
<name>Windows11</name>
<uuid>681d0b64-7d1b-4b73-a499-55c851919c46</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/11"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">32498688</memory>
<currentMemory unit="KiB">32498688</currentMemory>
<vcpu placement="static">16</vcpu>
<os firmware="efi">
<type arch="x86_64" machine="pc-q35-6.2">hvm</type>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode="custom">
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vpindex state="on"/>
<runtime state="on"/>
<synic state="on"/>
<stimer state="on">
<direct state="on"/>
</stimer>
<reset state="on"/>
<vendor_id state="on" value="KVM Hv"/>
<frequencies state="on"/>
<reenlightenment state="on"/>
<tlbflush state="on"/>
<ipi state="on"/>
</hyperv>
<kvm>
<hidden state="on"/>
</kvm>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on"/>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/matthias/Schreibtisch/Win11_23H2_German_x64v2.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
<boot order="1"/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<disk type="block" device="disk">
<driver name="qemu" type="raw" cache="none" io="native" discard="unmap"/>
<source dev="/dev/nvme1n1"/>
<target dev="vda" bus="virtio"/>
<boot order="2"/>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:4c:e7:52"/>
<source network="default"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<channel type="unix">
<target type="virtio" name="org.qemu.guest_agent.0"/>
<address type="virtio-serial" controller="0" bus="0" port="2"/>
</channel>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<tpm model="tpm-crb">
<backend type="emulator" version="2.0"/>
</tpm>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="spice"/>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x03" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</hostdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="1"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>
Ideas?
2 Likes
I am still fighting with Error 43.
In the VM-logs I found:
2024-05-02T15:23:08.478565Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=hostdev0,bus=pci.6,addr=0x0: vfio_listener_region_add received unaligned region
Anyone else having this issue? Looking for answers in the web directs me towards a GPU-Rom / VBIOS-issue…
Since it is working for @CodeMichael , I am not convinced that I am following the right path here…
One thing that might have an impact is I do not see OVMF nvram firmware in your configuration. I’m guessing that is the critically missing piece. Here’s my os block, you’ll need the ovmf package installed and to update the paths for your system.
<os>
<type arch="x86_64" machine="pc-q35-8.2">hvm</type>
<loader readonly="yes" type="pflash">/run/libvirt/nix-ovmf/OVMF_CODE.fd</loader>
<nvram template="/run/libvirt/nix-ovmf/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
<boot dev="hd"/>
</os>