I use virtual machines a lot for my work in software development. For all of what I do, KVM/qemu on Linux has been perfect as I have always used Intel based systems. Over the years as I learned and setup various virtual environments, I noticed that there seemed to be always some extra precautions and patches/workarounds that accompanied a lot of guides when addressing AMD processors and virtualization. I got the impression that AMD platforms seem to be an after-thought when it comes to virtualization. When it comes to AMD, I do not know as much about the virtualization instructions as much as I know about Intel processors.
My question is, for virtualization on the Ryzen 9 in FW16, can I safely expect KVM (or even VMware Workstation) to work on common distros, say Debian or Fedora, without a lot of workarounds or limitations? Basically are there any known issues with type-1 hypervisors on Ryzen 9 7940HS that would make me consider staying on an Intel platform?
What limitations and workarounds are you talking about?
I’ve always been using virtualization solutions (kvm, vmware and virtualbox) on AMD processors without any workarounds and didn’t notice any limitations.
I was referring to issues around IOMMU groups and hardware passthrough. If you use virtualized environments a lot on AMD processors and have not run into issues, then there are no issues I should be concerned about.
I have been using this with an AMD CPU and GPU for several years without any issue. I am typing this on an ASUS G513QY (AMD advantage edition from 2021) with AMD CPU and AMD GPU using Debian testing with a Windows 10 VM using GPU passthrough (6800M) and looking glass and everything works perfectly.
I am hoping to be able to do the same with a FW16 in the future.
I think you need to be clearerest () on what your expectations are from virtualization. I run a ton of VMs using KVM for testing purposes: lots of distributions, major OSes and even nested virtualization for other virtualization solutions. Unless you are doing something wild like passing physical hardware directly into a VM (I don’t do that), I don’t see anything getting in the way. KVM just worked and continues to work for me on the AMD FW13.
Mmmmaybe the only thing I have a complaint with is that the bridge created by libvirt sucks battery if enabled until restarted even if disabled (as I last recall). That’s easy to live with, work-around, and I expect somebody will pivot to stare at that at some point. I don’t care at the moment as I’m on power most of the time.
I’m planning/hoping to do the same, but we won’t know for sure until we can see the vfio groups for this motherboard layout. Ideally we’d have at least two groups for the USB interfaces so that you could pass through the entire interface, and the dGPU would be in its own group. I’m hopeful as the desktop motherboards in recent years have been good, with lots of discrete groups.
I’m still doing some work on getting my install to be reproducible, but as promised here’s the IOMMU groupings. As you can see just about everything is on its own group, should make pass through easy. Once I have a stable install I’ll do some testing of passing through individual USB controllers, but looks like most of them are in their own group.
FW16 IOMMU Groups with AMD 7700S
IOMMU Group 0:
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
IOMMU Group 1:
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ed]
IOMMU Group 2:
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ed]
IOMMU Group 3:
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
IOMMU Group 4:
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ee]
IOMMU Group 5:
00:02.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ee]
IOMMU Group 6:
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel [1022:14ef]
IOMMU Group 7:
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
00:04.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel [1022:14ef]
IOMMU Group 8:
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
IOMMU Group 9:
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14eb]
IOMMU Group 10:
00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14eb]
IOMMU Group 11:
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14eb]
IOMMU Group 12:
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f0]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f1]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f2]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f3]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f4]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f5]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f6]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f7]
IOMMU Group 14:
01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev 12)
IOMMU Group 15:
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479] (rev 12)
IOMMU Group 16:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 33 [Radeon RX 7700S/7600/7600S/7600M XT/PRO W7600] [1002:7480] (rev c1)
IOMMU Group 17:
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
IOMMU Group 18:
04:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5042] (rev 01)
IOMMU Group 19:
05:00.0 Network controller [0280]: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter [14c3:0616]
IOMMU Group 20:
06:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN770 / PC SN740 256GB / PC SN560 (DRAM-less) NVMe SSD [15b7:5017] (rev 01)
IOMMU Group 21:
c5:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 [1002:15bf] (rev c1)
IOMMU Group 22:
c5:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
IOMMU Group 23:
c5:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h (Model 74h) CCP/PSP 3.0 Device [1022:15c7]
IOMMU Group 24:
c5:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b9]
IOMMU Group 25:
c5:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ba]
IOMMU Group 26:
c5:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor [1022:15e2] (rev 63)
IOMMU Group 27:
c5:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
IOMMU Group 28:
c6:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ec]
IOMMU Group 29:
c6:00.1 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] AMD IPU Device [1022:1502]
IOMMU Group 30:
c7:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ec]
IOMMU Group 31:
c7:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15c0]
IOMMU Group 32:
c7:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15c1]
IOMMU Group 33:
c7:00.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Pink Sardine USB4/Thunderbolt NHI controller #1 [1022:1668]
IOMMU Group 34:
c7:00.6 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Pink Sardine USB4/Thunderbolt NHI controller #2 [1022:1669]
Thank you. That is exactly the kind of chipset in a laptop that has been like a thing of my dreams for a number of years haha. Unfortunately, I had to postpone my plans for getting the Framework 16 until later in the last quarter of 2024. But I do look forward to an improved workflow with virtualization.
This is great to see. I was just looking for this very info myself, because I also want to do GPU and peripheral passthrough. Hoping to create a win11 gaming VM under ubuntu/kvm and see how that goes. I want to run Star Citizen on it but I’m not going to be too upset if I can’t, I guess. I’ll just set up a dual boot with one of those external storage chips if that’s the case
@CodeMichael will you please continue to provide updates as you find successes and failures in what you’re doing?
FYI I have a desktop PC I built myself with AMD 5800x3d. I run ubuntu on it as host, and a windows KVM guest as my gaming machine, with GPU & ssd passthrough.
Most notably, nested virtualization seems to be either not working, not supported or very slow. I don’t remember what was the exactly conclusion back then when I did that experiment, but it seems to me that nested virtualization on amd is not as good as on intel.
Other than that I didn’t encounter any amd-specific issues. Qemu upgrade broke my setup once going from 5 to 6 but it’s not AMD related.
I’m a little distracted, but getting virtualization working in on the short list here. I’m going back through my notes around getting everything set up and doing what I can to move that whole process into my Nix configuration, so I’m being a little slow with getting things working. I’ll try to work on it this evening and this weekend and get back to you with results.
Thank you for your efforts! There’s no real rush so take your time. Please remember this is just a convenience for the rest of us, lol. I’ve ordered a framework 16 but it’s a couple months out still I think.
Figured I’d go ahead an post my current blocker and see if anyone had some ideas.
As you can from the above post the 7700S has the following devices: 1002:7480 and 1002:ab30 I’ve added those to the vfio-pci.ids and pci-sub.ids kernel parameter lists, but when I boot the vga device is using the amdgpu driver, rather than the vfio-pci driver. My understanding is that the vfio-pci.ids list should prevent the amdgpu from grabbing it, but that’s one of the earliest modules available during boot if I remove amdgpu from the initrd kernel modules its not booting at all.
In general things seem to be working just fine. I’ll pull together some documentation along with my nix configs so that you can reproduce if you like. I guess I’m taking requests if anyone has anything they’d like me to try or info they’d like me to report back on.
Nice! looking great! Seems you’re already doing what I want to do. What’s the base OS? Ubuntu? Mind sharing a guide on how you accomplished this when done? Or at least the general steps?
It sounds like he’s using NixOS, but I’m sure he can elaborate for us with the docs he’s writing. I’m planning on using QEMU in Kinoite or Solus, but Nix is all the rage.
Kinoite sounds neat. I’ve never gotten around to playing with Fedora (for more than a few min) but the whole Silverblue effort sounds like something I would be interested in (might have to load up sway atomic in a VM). Don’t judge me too harshly for using Nix.
Glad to do it. Lovely to be chatting with people with similar interests. If you have something you’d like to me to test for you let me know.
I’ll try to clean up my notes and make a legible document, but I suspect you all won’t need much of it, I really didn’t run into any significant challenges other than my own silliness. The board layout was the biggest unknown. I think the last checkbox on my own list for virtualization was to see if I could pass USB via PCI. I will try to knock that out this week.
) on what your expectations are from virtualization. I run a ton of VMs using KVM for testing purposes: lots of distributions, major OSes and even nested virtualization for other virtualization solutions. Unless you are doing something wild like passing physical hardware directly into a VM (I don’t do that), I don’t see anything getting in the way. KVM just worked and continues to work for me on the AMD FW13.
