QEMU/KVM on AMD Ryzen 9 Desktop with Dual-boot and Passthrough

From WikiMLT

I was in need to ac­cess Win­dows 10 from Kali Lin­ux on my dual boot­ed AMD Ryzen 9 based Desk­top PC.

Hard­ware Com­mands to de­tect
AMD Ryzen 9 5900X 12-Core Zen Proces­sor lscpu, sudo dmidecode -t processor, sudo lshw -class cpu
32 GB RAM free -h, sudo dmidecode -t mamory, sudo lshw -class memory
AS­Rock X570 Phan­tom Gam­ing 4 Moth­er­board sudo dmidecode -t baseboard, sudo lshw -class bus
NVIDIA TU117GL Quadro T600 and NVIDIA GF119 NVS 315 lsp­ci | grep ‑i 'vga', su­do lshw ‑class dis­play
SSD Cru­cial CT250MX 250GB and NVMe Sam­sung 980 1TB sudo fdisk -l, lsblk -d, lsblk -dD, ls -l /dev/disk/by-id

Ac­tu­al­ly I will passthrough one of the phys­i­cal stor­age de­vices where Win­dows 10 is al­ready in­stalled (SSD Cru­cial 250GB) and one of the video cards (NVS 315). So here are the things I've done to achieve that. The SSD will be passed as block de­vice.

Test the Vir­tu­al­iza­tion Ca­pa­bil­i­ties of the Sys­tem

Check weath­er the sys­tem sup­ports vir­tu­al­iza­tion and weath­er it is en­abled via the UEFI/BIOS. The fol­low­ing com­mand must re­turn at least 1:

egrep -c '(vmx|svm)' /proc/cpuinfo

In­stall QE­MU, KVM, LIB­VIRT

With­in the old­er ver­sions of De­bian based OS, like as Ubun­tu 20.04, we was in need to in­stall the pack­ages qe­mu qe­mu-kvm, but in mot re­cent op­er­at­ing sys­tems as Kali 2022 we need to in­stall qe­mu-sys­tem-x86 in­stead.

sudo apt install qemu-system-x86 libvirt-daemon bridge-utils
sudo apt install libvirt-clients virtinst libosinfo-bin ovmf
sudo apt install virt-manager virt-viewer remmina # For desktop user

In or­der to get rid of the pass­word di­a­logue for virt-man­ag­er"Sys­tem pol­i­cy pre­vents man­age­ment of lo­cal vir­tu­al­iza­tion sys­tems" – I've added my Lin­ux user to the lib­virt group.

sudo usermod -aG libvirt $USER
sudo usermod -aG kvm $USER
grep "$USER" /etc/group

Set­ting-up the PCI Passthrough

This sec­tion is ded­i­cat­ed to AMD based sys­tems. For In­tel based sys­tems check the ar­ti­cle QEMU/KVM and GPU Passthrough in De­tails.

En­abling IOM­MU

In or­der to en­abling the IOM­MU fea­ture we must ed­it the con­fig­u­ra­tion file /​​​etc/​​​default/​​​grub, as fol­low:

sudo nano /etc/default/grub # cat /etc/default/grub | grep 'GRUB_CMDLINE_LINUX_DEFAULT'
# For AMD CPU
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
  • Note the the host hard­ware is qui­et mod­ern, so with­in the UE­FI (BIOS) set­tings there are IOM­MU op­tion and when it is en­abled the above op­tions are not need­ed.

Up­date the boot man­ag­er con­fig­u­ra­tion and re­boot the sys­tem.

sudo update-grub
sudo systemctl reboot

Af­ter the re­boot ver­i­fy does IOM­MU is en­abled:

sudo dmesg | grep -i 'IOMMU'
#Out­put
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-5.18.0-kali5-amd64 root=/dev/mapper/kali--x--vg-root ro quiet amd_iommu=on iommu=pt splash
[    0.009753] Kernel command line: BOOT_IMAGE=/vmlinuz-5.18.0-kali5-amd64 root=/dev/mapper/kali--x--vg-root ro quiet amd_iommu=on iommu=pt splash
[    0.590958] iommu: Default domain type: Passthrough (set via kernel command line)
[    0.607751] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.607771] pci 0000:00:01.0: Adding to iommu group 0
[    0.607776] pci 0000:00:01.2: Adding to iommu group 1
[    0.607781] pci 0000:00:01.3: Adding to iommu group 2
[    0.607788] pci 0000:00:02.0: Adding to iommu group 3
[    0.607795] pci 0000:00:03.0: Adding to iommu group 4
[    0.607800] pci 0000:00:03.1: Adding to iommu group 5
[    0.607806] pci 0000:00:04.0: Adding to iommu group 6
[    0.607812] pci 0000:00:05.0: Adding to iommu group 7
[    0.607818] pci 0000:00:07.0: Adding to iommu group 8
[    0.607823] pci 0000:00:07.1: Adding to iommu group 9
[    0.607829] pci 0000:00:08.0: Adding to iommu group 10
[    0.607834] pci 0000:00:08.1: Adding to iommu group 11
[    0.607842] pci 0000:00:14.0: Adding to iommu group 12
[    0.607846] pci 0000:00:14.3: Adding to iommu group 12
[    0.607863] pci 0000:00:18.0: Adding to iommu group 13
[    0.607867] pci 0000:00:18.1: Adding to iommu group 13
[    0.607872] pci 0000:00:18.2: Adding to iommu group 13
[    0.607875] pci 0000:00:18.3: Adding to iommu group 13
[    0.607879] pci 0000:00:18.4: Adding to iommu group 13
[    0.607883] pci 0000:00:18.5: Adding to iommu group 13
[    0.607887] pci 0000:00:18.6: Adding to iommu group 13
[    0.607891] pci 0000:00:18.7: Adding to iommu group 13
[    0.607895] pci 0000:01:00.0: Adding to iommu group 14
[    0.607944] pci 0000:02:02.0: Adding to iommu group 15
[    0.607972] pci 0000:02:06.0: Adding to iommu group 16
[    0.607977] pci 0000:02:08.0: Adding to iommu group 17
[    0.607981] pci 0000:02:09.0: Adding to iommu group 18
[    0.607987] pci 0000:02:0a.0: Adding to iommu group 19
[    0.608018] pci 0000:03:00.0: Adding to iommu group 20
[    0.608045] pci 0000:03:00.1: Adding to iommu group 20
[    0.608073] pci 0000:04:00.0: Adding to iommu group 21
[    0.608075] pci 0000:05:00.0: Adding to iommu group 17
[    0.608077] pci 0000:05:00.1: Adding to iommu group 17
[    0.608079] pci 0000:05:00.3: Adding to iommu group 17
[    0.608081] pci 0000:06:00.0: Adding to iommu group 18
[    0.608083] pci 0000:07:00.0: Adding to iommu group 19
[    0.608088] pci 0000:08:00.0: Adding to iommu group 22
[    0.608096] pci 0000:09:00.0: Adding to iommu group 23
[    0.608102] pci 0000:09:00.1: Adding to iommu group 23
[    0.608108] pci 0000:0a:00.0: Adding to iommu group 24
[    0.608114] pci 0000:0b:00.0: Adding to iommu group 25
[    0.608121] pci 0000:0b:00.1: Adding to iommu group 26
[    0.608127] pci 0000:0b:00.3: Adding to iommu group 27
[    0.608133] pci 0000:0b:00.4: Adding to iommu group 28
[    0.608569] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.609034] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    0.890747] AMD-Vi: AMD IOMMUv2 loaded and initialized
sudo dmesg | grep 'AMD-Vi'
#Out­put
[    0.607751] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.608569] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.608569] AMD-Vi: Extended features (0x58f77ef22294a5a): PPR NX GT IA PC GA_vAPIC
[    0.608572] AMD-Vi: Interrupt remapping enabled
[    0.890747] AMD-Vi: AMD IOMMUv2 loaded and initialized

Iden­ti­fi­ca­tion of the Group Con­trollers

In or­der to gen­er­ate a tidy list of your grouped de­vices cre­ate a script as the fol­low.

sudo nano /usr/local/bin/get_iommu_groups.sh && sudo chmod +x /usr/local/bin/get_iommu_groups.sh
#Script
#!/bin/bash
# https://mathiashueber.com/pci-passthrough-ubuntu-2004-virtual-machine/
# change the 9999 if needed
shopt -s nullglob
for d in /sys/kernel/iommu_groups/{0..9999}/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done

Run the script and fil­ter the out­put.

get_iommu_groups.sh | grep -iP 'VGA compatible controller|SATA controller|USB controller|NVIDIA'
#Out­put
IOMMU Group 17 05:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 17 05:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 18 06:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 19 07:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 20 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [NVS 315] [10de:107c] (rev a1)
IOMMU Group 20 03:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1)
IOMMU Group 23 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117GL [T600] [10de:1fb1] (rev a1)
IOMMU Group 23 09:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
IOMMU Group 27 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

In this case I need to iso­late the NVS 315 video con­troller: IOM­MU Group 20 that con­tains PCI-bus 03:00.0 [de­vice ID 10de:107c] and 03:00.1 [de­vice ID 10de:0e08].

En­abling VFIO Ker­nel mod­ules

To load VFIO and oth­er re­quired mod­ules at boot, ed­it the /etc/initramfs-tool­s/­mod­ules file. If you run Ubun­tu 20.04, Lin­ux Mint 20 or sim­i­lar, then the fol­low­ing mod­ules have been in­te­grat­ed in­to the ker­nel and you do not need to load them (ref­er­ence).

sudo nano /etc/initramfs-tools/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net
sudo update-initramfs -u -k all

Black­list the de­fault nou­veau dri­ver. Note in my case it is dis­abled ear­li­er be­cause of the oth­er NVIDIA dri­ver.

sudo nano /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
sudo update-initramfs -u

Iso­la­tion of the Guest GPU

In or­der to iso­late the GPU we have two op­tions. Se­lect the de­vices by PCI bus ad­dress or by de­vice ID. Both op­tions have pros and cons. Here we will iso­late VFIO-pci dri­ver by de­vice id. This op­tion should on­ly be used, in case the graph­ic cards (or oth­er de­vices that will be iso­lat­ed) in the sys­tem are not ex­act­ly the same mod­el, oth­er­wise we need to use iso­la­tion by PCI bus, be­cause the de­vices will have an iden­ti­cal IDs.

sudo nano /etc/default/grub # cat /etc/default/grub | grep 'GRUB_CMDLINE_LINUX_DEFAULT'
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt kvm.ignore_msrs=1 kvm.report_ignored_msrs=0 irqpoll vfio-pci.ids=10de:107c,10de:0e08 vfio-pci.disable_vga=1"
  • Ex­pla­na­tion about the used op­tions could be found in the ar­ti­cle list­ed with­in the ref­er­ences.

Up­date the boot man­ag­er con­fig­u­ra­tion and re­boot the sys­tem.

sudo update-grub
sudo systemctl reboot

Af­ter this re­boot the iso­lat­ed GPU will be ig­nored by the host OS. Now, you have to use the oth­er GPU for the host OS. Af­ter the re­boot, ver­i­fy the Iso­la­tion of the guest GPU by an­a­lyze the out­put of the fol­low­ing com­mand:

sudo lspci -nnv -s 03:00
#Out­put
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [NVS 315] [10de:107c] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation GF119 [NVS 315] [10de:102f]
	Flags: bus master, fast devsel, latency 0, IRQ 255, IOMMU group 20
	Memory at f9000000 (32-bit, non-prefetchable) [size=16M]
	Memory at d8000000 (64-bit, prefetchable) [size=128M]
	Memory at e0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at e000 [size=128]
	Expansion ROM at fa000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Endpoint, MSI 00
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100] Virtual Channel
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau, nvidia_drm, nvidia

03:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1)
	Subsystem: NVIDIA Corporation GF119 HDMI Audio Controller [10de:102f]
	Flags: fast devsel, IRQ 255, IOMMU group 20
	Memory at fa080000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Endpoint, MSI 00
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

Con­grat­u­la­tions, the hard­est part is done!

Set­up the Vir­tu­al Ma­chines

My cur­rent set­up has two vir­tu­al ma­chines. Both vir­tu­al ma­chines have ac­cess to the same phys­i­cal SSD dri­ve, where Win­dows 10 is pre­vi­ous­ly in­stalled and ful­ly op­er­a­tional via the dual boot op­tion and I want to keep this way of ac­cess­ing Win­dows 10 too.

The first "spe­cial" thing ac­cord­ing to my set­up is that both op­er­at­ing sys­tems are in­stalled in UE­FI mode, so the vir­tu­al ma­chine should have UE­FI firmware and with chipset Q35.

Iden­ti­fy the SSD

There are two ways to pass the SSD as block de­vice as it is de­scribed in the ar­ti­cle QEMU/KVM on ThinkPad X230T Lap­top with Dual-boot and and by pass­ing the SA­TA con­troller as it is de­scribed in this sec­tion.

At first glance there is not any sig­nif­i­cant per­for­mance dif­fer­ence. How­ev­er when you use the block de­vice ap­proach you can use the write cache op­tion for the de­vice, which will in­crease the speed of han­dling of the large files. On the oth­er hand, when pass­ing the SA­TA con­troller ap­proach is in use, Win­dows 10 will use the same dri­ver with­in the VM en­vi­ron­ment and with­in the na­tive boot. In both cas­es the SSD shouldn't be mount­ed at the host's side.

The SA­TA ports are iden­ti­fied as con­trollers and each of them has its own IOM­MU group, so we don't need any ad­di­tion­al set­up ex­cept to check at which port is at­tached the de­vice. This can be done by the com­mands be­low. Note in this case we are search­ing for the con­troller where is at­tached /​​​dev/​​​sda.

lspci | grep SATA   # Just for check - which controllers (SATA ports) have attached drives
06:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
udevadm info -q path -n /dev/sda
/devices/pci0000:00/0000:00:01.2/0000:01:00.0/0000:02:09.0/0000:06:00.0/ata2/host1/target1:0:0/1:0:0:0/block/sda
udevadm info -q path -n /dev/sda | awk -F'/' '{print $7}'
0000:06:00.0

Cre­ate and Con­fig­ure the Vir­tu­al ma­chines

Cre­ate a vir­tu­al ma­chine via wiz­ard of the virt-man­ag­er GUI tool:

  • Step 1/5: Choice "Man­u­al In­stall".
  • Step 2/5: Type win10 and choose the en­try "Mi­crosoft Win­dows 10".
  • Step 3/5: Pro­vide suit­able to your sys­tem amount of Mem­o­ry and num­ber of CPUs.
  • Step 4/5: Don't en­able any stor­age de­vices.
  • Step 5/5: Tick the check­box Cus­tomize con­fig­u­ra­tion be­fore in­stall and choose the the type of the net­work con­nec­tion – I pre­fer to use bridged de­vice.

Be­fore pro­ceed by click­ing the but­ton Be­gin in­stall (up­per left cor­ner – see Video 1), choose the fol­low­ing op­tions:

  • Chipset: Q35,
  • Firmware: UE­FI x86_​​​64: /usr/share/OVMF/OVMF_CODE_4M.ms.fd
  • The use the but­ton Add Hard­ware and add the passthrough de­vices – see Video 1.
Video 1. Virt-manager VM with configuration example: [right side] GPU, USB and SATA passthrough, [right side] SATA only passthrough.
Video 1. Virt-man­ag­er VM with con­fig­u­ra­tion ex­am­ple: [right side] GPU, USB and SA­TA passthrough, [right side] SA­TA on­ly passthrough.

On Video 1, at the right side is shown the VM Win10PT which us­es GPU, USB and SA­TA passthrough. At the left side – the VM Win10 which us­es on­ly SA­TA passthrough. Both can­not be start­ed at the same time, be­cause they share the same SA­TA con­troller and phys­i­cal SSD.

I'm ac­cess­ing the left one Win10 via virt-view­er. And, the right one Win10PT by a ded­i­cat­ed mon­i­tor, key­board and mouse (or via RDP).

For the VM Win10PT which us­es GPU, USB and SA­TA passthrough, I wasn't able to re­move all SPICE de­vices (as it is rec­om­mend­ed for best per­for­mance with­in GPU passthrough) via virt-man­ag­er. So I've re­moved them by edit­ing the con­fig­u­ra­tion af­ter the VM was cre­at­ed via virsh.

virsh --connect qemu:///system edit Win10PT

The Fi­nal Con­fig­u­ra­tion of the Vir­tu­al ma­chines

The fi­nal con­fig­u­ra­tion of the both vir­tu­al ma­chines Win10 and Win10PT is avail­able at github​.com/​m​e​t​a​l​e​v​e​l​–​t​e​c​h​/​q​e​m​u​–​k​v​m​–​s​c​r​ipts/vm-dumpxml, where the files that start with kali-x_­date are these re­lat­ed to the cur­rent ar­ti­cle.

The repos­i­to­ry github​.com/met­alev­el-tech/qe­mu-kvm-scripts con­tains al­so a script that al­lows me to menage the two vir­tu­al ma­chines via .desk­top short­cuts. More in­for­ma­tion about the script is pro­vid­ed with­in the repos­i­to­ry it­self.

In­stall the Guest Tools and Trou­bleshoot­ing

Once the guest OS is run­ning suc­cess­ful­ly, the fi­nal step of the set­up is in­stalling the QEMU/KVM Guest tools for Win­dows, thus the screen will be au­to­mat­i­cal­ly re­sized with­in the SPICE client of virt-man­ag­er and virt-view­er. And al­so al­lows you to grace­ful­ly shut­down the guest.

There was ex­pe­ri­enced the prob­lem Guest HD­MI Au­dio Crack­ling and was able to fix it by en­abling the MSISup­port­ed op­tion for the de­vice via RegEd­it in the Win­dows 10 guest.

It is not men­tioned any­where above but you need to En­able the QE­MU Guest Agent with­in the vir­tu­al ma­chine con­fig­u­ra­tion, oth­er­wise the agent will not work and, for ex­am­ple, you will not be able to shut­down the VM from the host's side.

Ad­di­tion­al Guides

Ker­nel 6.0.x Trou­bleshoot­ing

Af­ter up­grade to Ker­nel 6.0.x the sys­tem boot­ed but there was no screen out­put. Ini­tial­ly I was think­ing it is NVIDIA dri­ver is­sue, but fi­nal­ly I've found it is is­sue with VFIO. The same trou­ble is re­port­ed in the fol­low­ing blog and fo­rum posts:

There as workaround is pro­posed: Us­ing  the “driver_​​​override” fea­ture. And al­so it is re­port­ed that with Ker­nel 6.1.x every­thing work well with the GRUB method.

In my par­tic­u­lar case the PT video card is NVS 315 which is not com­pat­i­ble with the NVIDIA dri­ver for T600 and it is ig­nored at the boot time. So I just mod­i­fied GRUB_CMDLINE_LINUX_DEFAULT in the fol­low­ing way to get the sys­tem op­er­a­tional.

#GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt kvm.ignore_msrs=1 kvm.report_ignored_msrs=0 irqpoll vfio-pci.ids=10de:107c,10de:0e08"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt kvm.ignore_msrs=1 kvm.report_ignored_msrs=0 irqpoll"

At this point when I run the vir­tu­al ma­chine vfio-pci takes con­trol over NVS 315. And the guest os works as it is ex­pect­ed – strange :) So I will not ap­ply the “driver_​​​override” fea­ture and will wait for Ker­nel 6.1.x to be­came de­fault on my dis­tro.

Ref­er­ences