@CodeMichael, thank you for your research. It is truly helpful and I know I will finally have my perfect non-dual booting configuration with close to native performance inside VMs. I mostly use Linux but once in a while I have to work with Adobe or Microsoft products for a few projects and I hate having to keep a Windows partition for dual booting. I do have a Windows VM but without graphics pass-through, performance is poor. So when I can afford a FW16 I will be ready for that perfect setup. Thanks a lot!
Iāve got my notes about my nix setup published, and I discovered how to successfully detach and re-attach the the dgpu without (seemingly) any major issues.
4 Likes
Awesome! Thank you for the effort. Iām considering trying out nix with this laptop. Might steal your config for it 
Hey there,
I m running Ubuntu 22.04 with recommended OEM-Kernel on my Framework 16.
I am trying to forward the 7700S to my Windows VM. Therefore I forwarded the GPU to the VM using PCIE-forward in VirtManager.
Windows detects the card and i can install the AMD driver but im stuck with error 43 in device manager.
I checked lspci -nnv and it appears to be forwarded just fine. Do i need to forward another device to make it work?
Thanks for your help
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7480] (rev c1) (prog-if 00 [VGA controller])
Subsystem: Device [f111:0007]
Flags: bus master, fast devsel, latency 0, IRQ 49, IOMMU group 16
Memory at 8400000000 (64-bit, prefetchable) [size=8G]
Memory at 8600000000 (64-bit, prefetchable) [size=256M]
I/O ports at a000 [size=256]
Memory at 90d00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at 90e20000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
Subsystem: Device [f111:0005]
Flags: bus master, fast devsel, latency 0, IRQ 132, IOMMU group 17
Memory at 90e00000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
I address that in my notes (link above). There are additional links in those notes that talk about this issue directly if you need further input.
<features>
<hyperv mode="custom">
<vendor_id state="on" value="someRandomBits"/>
</hyperv>
<kvm> <hidden state="on"/> </kvm>
</features>
Thank you for your reply. For some reason including the hidden tag and forwarding the gpu at the same time letās the vm reboot every few seconds. That causes my system to crash since i am unable to let the host ignore the gpu. I tried
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio-pci.ids=1002:7480,1002:ab30 pci-stub.ids=1002:7480,1002:ab30 kvm_amd.npt=1 kvm_amd.avic=1 quiet splash "
but still get
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7480] (rev c1) (prog-if 00 [VGA controller])
Subsystem: Device [f111:0007]
Flags: bus master, fast devsel, latency 0, IRQ 129, IOMMU group 16
Memory at 8400000000 (64-bit, prefetchable) [size=8G]
Memory at 8600000000 (64-bit, prefetchable) [size=256M]
I/O ports at a000 [size=256]
Memory at 90d00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at 90e20000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: amdgpu
Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
Subsystem: Device [f111:0005]
Flags: bus master, fast devsel, latency 0, IRQ 141, IOMMU group 17
Memory at 90e00000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
after booting the host.
I guess that is the first problem I have to figure out before i can adress the VM rebooting issue. I am trying to adapt your solution to Ubuntu, so far with limited success
you need to make sure to load the vfio modules before the amdgpu module.
Got my crash problem solved. As it turns out i did not have anything to do with what I am currently working on. So i added the mentioned xml-tags, but that did not change anything. I managed to isolate the gpu. In addition to my changes in GRUB i changed the initramfs:
- Create the conf-File
sudo nano /etc/modprobe.d/vfio.conf
- Fill it with:
options vfio-pci ids=1002:7480,1002:ab30
softdep amdgpu pre: vfio-pci
- Update and reboot
sudo update-initramfs -c -k $(uname -r)
- Check with lspci -v
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 7480 (rev c1) (prog-if 00 [VGA controller])
Subsystem: Device f111:0007
Flags: fast devsel, IRQ 255, IOMMU group 16
Memory at 8400000000 (64-bit, prefetchable) [disabled] [size=8G]
Memory at 8600000000 (64-bit, prefetchable) [disabled] [size=256M]
I/O ports at a000 [disabled] [size=256]
Memory at 90d00000 (32-bit, non-prefetchable) [disabled] [size=1M]
Expansion ROM at 90e20000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device ab30
Subsystem: Device f111:0005
Flags: fast devsel, IRQ 255, IOMMU group 17
Memory at 90e00000 (32-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
GPU isolated!
VM-Config:
domain type="kvm">
<name>Windows11</name>
<uuid>681d0b64-7d1b-4b73-a499-55c851919c46</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/11"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">32498688</memory>
<currentMemory unit="KiB">32498688</currentMemory>
<vcpu placement="static">16</vcpu>
<os firmware="efi">
<type arch="x86_64" machine="pc-q35-6.2">hvm</type>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode="custom">
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vpindex state="on"/>
<runtime state="on"/>
<synic state="on"/>
<stimer state="on">
<direct state="on"/>
</stimer>
<reset state="on"/>
<vendor_id state="on" value="KVM Hv"/>
<frequencies state="on"/>
<reenlightenment state="on"/>
<tlbflush state="on"/>
<ipi state="on"/>
</hyperv>
<kvm>
<hidden state="on"/>
</kvm>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on"/>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/matthias/Schreibtisch/Win11_23H2_German_x64v2.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
<boot order="1"/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<disk type="block" device="disk">
<driver name="qemu" type="raw" cache="none" io="native" discard="unmap"/>
<source dev="/dev/nvme1n1"/>
<target dev="vda" bus="virtio"/>
<boot order="2"/>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:4c:e7:52"/>
<source network="default"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<channel type="unix">
<target type="virtio" name="org.qemu.guest_agent.0"/>
<address type="virtio-serial" controller="0" bus="0" port="2"/>
</channel>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<tpm model="tpm-crb">
<backend type="emulator" version="2.0"/>
</tpm>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="spice"/>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x03" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</hostdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="1"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>
Ideas?
2 Likes
I am still fighting with Error 43.
In the VM-logs I found:
2024-05-02T15:23:08.478565Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=hostdev0,bus=pci.6,addr=0x0: vfio_listener_region_add received unaligned region
Anyone else having this issue? Looking for answers in the web directs me towards a GPU-Rom / VBIOS-issueā¦
Since it is working for @CodeMichael , I am not convinced that I am following the right path hereā¦
One thing that might have an impact is I do not see OVMF nvram firmware in your configuration. Iām guessing that is the critically missing piece. Hereās my os block, youāll need the ovmf package installed and to update the paths for your system.
<os>
<type arch="x86_64" machine="pc-q35-8.2">hvm</type>
<loader readonly="yes" type="pflash">/run/libvirt/nix-ovmf/OVMF_CODE.fd</loader>
<nvram template="/run/libvirt/nix-ovmf/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
<boot dev="hd"/>
</os>
Thank you for your response!
Adapted your <os>-Block to my system.
Old:
<os firmware="efi">
<type arch="x86_64" machine="pc-q35-6.2">hvm</type>
<boot dev="hd"/>
</os>
New:
<os>
<type arch="x86_64" machine="pc-q35-6.2">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram template="/usr/share/OVMF/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/Windows11_VARS.fd</nvram>
<boot dev="hd"/>
</os>
virt-manager only accepts the new code, when i remove firmare="efi".
I cannot change machine to 8.2, i get an error:
'/usr/bin/qemu-system-x86_64' does not support machine type 'pc-q35-8.2'
With the new code I only get a black screen on the console and on the external display. Machine runs but appears to do nothing.
Log-Entries do not look any different. When I try to post them here I get an 403 Error form the forums pageā¦
Ok, got bei ānewā OS-Block working.
I had manually to delete the existing vars-File. After that I could boot with the new settings.
The problem however stays the same. I still get the error:
2024-05-09T08:53:45.602111Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=hostdev0,bus=pci.6,addr=0x0: vfio_listener_region_add received unaligned region
and Error 43 in device manager.
When I turn on the VM and have a display connected to the GPU via its USB-C-port, the display posts and then freezes while the VM continues to boot and work fine on the console.
I downloaded the rom-file with GPU-Z and tried:
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</source>
<rom bar="on" file="/usr/share/vgabios/amdrom"/>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0" multifunction="on"/>
</hostdev>
Settings applied, the display remains in Standby after starting the VM. Apart from that nothing changes.
Maybe the card detects that it is virtualized. The AMD Adrenaline software tells me that it only supports AMD-Systems.
Skip the Adrenaline software! Go straight for the drivers. The Adrenaline software does some extra stuff to detect the VM and never gets a far as actually installing the drivers. You can bypass it completely.
Got just the driver installed, still no difference. Something is very odd. I just tried to boot from a Ubuntu-Image to verify if it is a driver issue in Windows or an issue with the VM itself.
While navigating through GRUB I have an output on the console and the display which is connected to the GPUās USB-C. As soon as I boot to the live system, the output on the connected display freezes and in Display settings I can only find the virtual monitor. When i shutdown the live system display output works againā¦
I suspect that the card somehow fails during boot - either because it āknowsā that it is plugged in to a VM or something elseā¦
Iām out of ideas on how to investigate thisā¦
How do you bypass this? I donāt see a link for a direct driver download anywhere on that website, only Adrenalin. Iāve installed the drivers directly by extracting them from Adrenalin downloads but do you literally do this each time there is an update?
I had dGPU passthrough working just fine using proxmox, but when i tried using libvirt, i got stuck in the same place.
Switching to using bare qemu commands instead of going through libvirt solved it for me. I donāt know what libvirt is doing to mess this up, and also not sure how to address it there. I guess thereās a config option missing. Perhaps adding '-vga none -nographic" using libvirt would solve it, but I have not tried this myself. (using <qemu:commandline > and <qemu:arg ā¦/> options).
Do you mind sharing your qemu commandline that is working? i would like to compare the options that libvirt is usingā¦
From the logs I have the following command:
/usr/bin/qemu-system-x86_64 \
-name guest=Windows11,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-Windows11/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/Windows11_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-6.2,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=on,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vpindex=on,hv-runtime=on,hv-synic=on,hv-stimer=on,hv-stimer-direct=on,hv-reset=on,hv-vendor-id=1234>
-m 31737 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":33278656512}' \
-overcommit mem-lock=off \
-smp 16,sockets=16,cores=1,threads=1 \
-uuid jbasjdbbjbdf\
-smbios 'type=0,vendor=INSYDE Corp.,version=03.03,date=03/27/2024,release=3.3' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=33,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=21,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-root-port,port=22,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \
-device pcie-root-port,port=23,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 \
-device pcie-root-port,port=24,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 \
-device pcie-root-port,port=25,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \
-device pcie-root-port,port=26,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 \
-device pcie-root-port,port=27,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 \
-device pcie-root-port,port=28,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 \
-device pcie-root-port,port=29,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/home/matthias/Schreibtisch/ubuntu-22.04.4-desktop-amd64.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-2-format,id=sata0-0-1,bootindex=2 \
-blockdev '{"driver":"host_device","filename":"/dev/nvme0n1","aio":"native","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"u>
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on \
-netdev tap,fd=34,id=hostnet0,vhost=on,vhostfd=36 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:4c:e7:52,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-chardev socket,id=charchannel1,fd=31,server=on,wait=off \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 \
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/2-Windows11-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device tpm-crb,tpmdev=tpm-tpm0,id=tpm0 \
-audiodev '{"id":"audio1","driver":"spice"}' \
-spice port=5900,addr=127.0.0.1,disable-ticketing=on,image-compression=off,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0,audiodev=audio1 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 \
-device vfio-pci,host=0000:03:00.0,id=hostdev0,bus=pci.6,addr=0x0,rombar=1,romfile=/usr/share/vgabios/dummy.rom \
-device vfio-pci,host=0000:03:00.1,id=hostdev1,bus=pci.7,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
Looks like you are using the qxl-vga driver, I donāt know if that plays well with dGPU passthrough / Looking Glass. I suggest you replace the above with ā-vga none -nographicā.
My commandline looks like this:
-accel kvm
-daemonize
-nodefaults
-machine hpet=off,type=pc-q35-8.1
-rtc driftfix=slew,base=localtime
-global kvm-pit.lost_tick_policy=discard
-readconfig /usr/share/qemu-server/pve-q35-4.0.cfg
-device qemu-xhci,p2=15,p3=15,id=xhci,bus=pci.1,addr=0x1b
-iscsi initiator-name=iqn.1993-08.org.debian:01:39407ad058b
-device pvscsi,id=scsihw0,bus=pci.0,addr=0x5
-device ich9-intel-hda,id=audiodev0,bus=pci.2,addr=0xc
-device hda-micro,id=audiodev0-codec0,bus=audiodev0.0,cad=0,audiodev=spice-backend0
-device hda-duplex,id=audiodev0-codec1,bus=audiodev0.0,cad=1,audiodev=spice-backend0
-boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg
-smbios type=1,uuid=
-drive if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd
-drive if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/vm1/vm-108-efidisk,size=540672
-smp 8,sockets=1,cores=8,maxcpus=8
-cpu qemu64,+aes,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vendor_id=ezkvm,hv_vpindex,kvm=off,+kvm_pv_eoi,+kvm_pv_unhalt,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3
-m 16384
-vga none
-nographic
-spice port=5903,addr=0.0.0.0,disable-ticketing=on
-device virtio-serial-pci
-chardev spicevmc,id=vdagent,name=vdagent
-device virtserialport,chardev=vdagent,name=com.redhat.spice.0
-device virtio-mouse
-device virtio-keyboard
-audiodev spice,id=spice-backend0
-device ivshmem-plain,memdev=ivshmem0,bus=pcie.0
-object memory-backend-file,id=ivshmem0,share=on,mem-path=/dev/shm/lg03,size=128M
-device vfio-pci,host=0000:03:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on
-device vfio-pci,host=0000:03:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1
-device usb-host,bus=xhci.0,port=1,hostbus=1,hostport=2.2,id=usb0
-drive file=/dev/vm1/vm-108-boot,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap
-device scsi-hd,bus=scsihw0.0,scsi-id=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=0
-drive file=/dev/vm1/vm-108-tmp,if=none,id=drive-scsi1,discard=on,format=raw,cache=none,aio=io_uring,detect-zeroes=unmap
-device scsi-hd,bus=scsihw0.0,scsi-id=1,drive=drive-scsi1,id=scsi1,rotation_rate=1
-netdev id=hostnet0,type=tap,script=/root/bin/qemu-ifup.sh,downscript=no,vhost=on
-device id=net0,driver=virtio-net-pci,netdev=hostnet0,mac=,bus=pci.1,addr=0x0
Note that this does still read a proxmox config file & uses the proxmox ovmf firmware (as i run the same VM from proxmox as well).
Note also that prior to passing through the dGPU i installed all the Framework 16 drivers for Windows 11, i guess that includes the gpu driver.
Another way to diagnose this could be to make a Linux VM, and passthru the GPU to the Linux VM.
If that fails to work, you are likely to get more meaningful error messages in the Linux syslog.
Once you fix the problem on the Linux VM, hopefully that will fix the problem on the Windows VM also.
So there are some updates:
-
After disabling QXL I got a working display output in my display, but only with a 800x600 resolution. But cool!
-
I booted the VM into a Ubuntu-ISO and got my correct 1920x1080 resolution, even the displaytype was identified correctly. Even cooler!
-
I did a clean reinstall of Windows and extracted the driver files from the amd driver package using 7zip. I then updated the driver using device manager and after install Error 43 came right back. That sucks.
-
I created another VM and installed Ubuntu. The passtrough worked fine after i changed the BIOS to OVMF and added KVM=Hidden. I ran an GLmark2 Benchmark and got a score of 8665. So it seems to be working. I then changed the virtual harddrive to my windows hard drive - Error 43 disappeared but I was stuck at a resolution of 800x600 pixel. I then reinstalled the drivers and Error 43 came right back.