Virtualization on FW16's Ryzen 9 (7940HS)

Framework Laptop 16
41

One potential issue you may hit (that I didn’t see mentioned here yet) if you want to pass through the AMD graphics is that AMD’s consumer GPUs don’t support SR-IOV. This means the dGPU can only be passed through to a single VM.

Intel iGPUs (and I think Nvidia GPUs ones with a tweaked driver) support SR-IOV, which lets you use the GPU for things like hardware acclerated video encode/decode across multiple VMs (or both the VM and the host system) at the same time. I use it on my home server running Unraid, to transcode video (TV shows recorded with a HDHomeRun) in a Docker container while also using hardware accelerated transcoding in a Windows Server 2022 VM for Blue Iris (security camera software).

1 Like
42

Ah so there is progress, looks like you’re almost there :slight_smile:
Error 43 indicates driver errors, this could be because the driver doesn’t want to play nice in a VM (NVidia is notorious for that, but hey, this is AMD, and should work better). It could also indicate that the driver is expecting a different pci bus configuration (which is more likely i think).

There’s a few more things you could try. Looking at the command-line you posted earlier, i suggest the following:

  • in the ‘-device vfio-pci,host=0000:03:00.0’ line add ‘multifunction=on’, and remove the ‘romfile=…’ option.
  • also in the ‘-device vfio-pci’ lines, you now have different bus= assignment, each with address 0x0. Instead, try having them on the same bus, but then with address 0x0.0 and address 0x0.1 respectively, that way the video and audio devices are configured as subfunctions of the same master device, just like on the host.
  • in the ‘-cpu’ line, try adding the ‘kvm=off’ flag.

Not sure how to do that in libvirt, but surely it has an option for using ‘multifunction’ devices, which is the essence here.

Good luck!

43

Thank you for your input!

I followed your guide, but without success. PCI-Assignments should be in order now?

I reinstalled Windows again and rebooted the host before installing the framework driver package on the VM.

One strange thing i noticed is, that I cannot reboot the VM. If I do, I loose the disyplay output. I have to shutdown the VM and then start it again in order to get display output. Rebooting with Ubuntu on the VM on the other hand works just fine.

I only use the card in one VM at a time.

Here is my current log (I removed my serial number):

2024-05-18 18:43:21.075+0000: starting up libvirt version: 8.0.0, package: 1ubuntu7.10 (Marc Deslauriers <marc.deslauriers@ubuntu.com> Fri, 12 Apr 2024 13:48:21 >
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-3-ubuntu22.04 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-ubuntu22.04/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-ubuntu22.04/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-ubuntu22.04/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=ubuntu22.04,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-3-ubuntu22.04/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/Ubuntu_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap">
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-6.2,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on,kvm=off \
-m 16000 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":16777216000}' \
-overcommit mem-lock=off \
-smp 8,sockets=8,cores=1,threads=1 \
-uuid ................ \
-smbios 'type=0,vendor=INSYDE Corp.,version=03.03,date=03/27/2024,release=3.3' \
-smbios 'type=1,manufacturer=Framework,product=Laptop 16 (AMD Ryzen 7040 Series),version=A7,serial=FRAGACCPA740830006,sku=FRAGACCP07,family=16in Laptop' \
-smbios 'type=2,manufacturer=Framework,product=FRANMZCP07,version=A7,serial=leftout 00XS,asset=*,location=*' \
-smbios type=3,manufacturer=Framework,version=A7,serial=......,asset=...,sku=..... \ #serial left out
-smbios >
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=33,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=10,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=11,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \
-device pcie-root-port,port=12,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \
-device pcie-root-port,port=13,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \
-device pcie-root-port,port=14,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \
-device pcie-root-port,port=15,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 \
-device pcie-root-port,port=16,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=17,chassis=10,id=pci.10,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=18,chassis=11,id=pci.11,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=19,chassis=12,id=pci.12,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=20,chassis=13,id=pci.13,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=21,chassis=14,id=pci.14,bus=pcie.0,addr=0x2.0x5 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
-device virtio-scsi-pci,id=scsi0,bus=pci.5,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.6,addr=0x0 \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/ubuntu22.04.qcow2","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"discard":"unmap","driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-3-format,id=virtio-disk0 \
-blockdev '{"driver":"file","filename":"/home/matthias/Schreibtisch/Win11_23H2_German_x64v2.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":>
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-cd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0 \
-blockdev '{"driver":"file","filename":"/home/matthias/Downloads/virtio-win-0.1.240.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}>
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1 \
-netdev tap,fd=34,id=hostnet0,vhost=on,vhostfd=36 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:27:88:9e,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=31,server=on,wait=off \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-chardev spicevmc,id=charchannel1,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev '{"id":"audio1","driver":"spice"}' \
-spice port=5900,addr=127.0.0.1,disable-ticketing=on,image-compression=off,seamless-migration=on \
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0,audiodev=audio1 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 \
-device vfio-pci,host=0000:04:00.0,id=hostdev0,bootindex=1,bus=pci.9,addr=0x0 \
-device vfio-pci,host=0000:03:00.0,id=hostdev1,bus=pci.3,multifunction=on,addr=0x0 \
-device vfio-pci,host=0000:03:00.1,id=hostdev2,bus=pci.3,addr=0x0.0x1 \
-device virtio-balloon-pci,id=balloon0,bus=pci.7,addr=0x0 \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.8,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
2024-05-18T18:44:42.549183Z qemu-system-x86_64: terminating on signal 15 from pid 1553 (/usr/sbin/libvirtd)
2024-05-18 18:44:43.350+0000: shutting down, reason=destroyed
44

Not sure if this is the problem, but it seems you are still missing the

-vga none
-nographic

arguments in your command, meaning that qemu will still enable it’s default vga display, that could be a problem.

45

@CodeMichael did you ever figure out how to get dgpu to attach and detach without going unstable?

Has anyone else figured this out? It’d be awesome to be able to detach it when launching the VM to attach it to the VM, then have it reattach to the computer after the VM shuts down.

46

This is the script I’ve been using which seems to be stable, but I wouldn’t say I’ve gone through extensive testing either.

1 Like
47

Does this mean the comment about instability in your repo’s main README.md is possibly no longer valid?

48

I haven’t used it much, but so far its been stable. I’ve been refactoring my repo after rebuilding my desktop, so I haven’t update in a bit, but I’ll correct that statement when I do get around to updating.