VM using VFIO PCIe passthrough experiences freezing and crashes.
VM using VFIO PCIe passthrough experiences freezing and crashes.
I am using KVM with Virtual machine manager on manjaro linux510. Ive been trying to get a windows 10 virtual machine with pcie passthrough working all day. I want to pass through my GPU, an Ethernet controller, a USB controller, and a SATA controller. The VM seemingly randomly freezes. It does not shut itself off, it just shows a still frozen image on the screen. The cpu activity indicator levels out and does not change at all when this happens, like so: I am a bit of a VM noob so I am not sure where to even start troubleshooting this. I have tried running the VM with different devices disabled and enabled, I dont think its any of the pcie devices, but i could be wrong. I have also tried swapping the USB controller because i noticed that somtimes when I flipped my KVM switch that is when It froze, but this might be correlation without causation, and the same thing happened with the new USB controller. I appreciate any help. Below I have included some related information. Please let me know if you need anything else to help me. Thank you very much Specs: MSI z490-A Pro 32GB DDR4 (16 to vm) 10700k (4 cores to vm, igpu to host) 2 SSDs 1 ssd and 1 hdd (connected to the passthrough sata controller) 1660 super (to vm) ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller (to vm) ASMedia Technology Inc. ASM1062 Serial ATA Controller (to vm) Realtek 1Gb ethernet controller (to vm) USB Switch box (1 connected to asmedia usb controller and other connected to onboard usb controller) Here is my XML: Spoiler <domain type="kvm"> <name>win10</name> <uuid>12529c4b-8f04-45a7-ad9d-fcef3f4d298b</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://microsoft.com/win/10"/> </libosinfo:libosinfo> </metadata> <memory unit="KiB">16777216</memory> <currentMemory unit="KiB">16777216</currentMemory> <vcpu placement="static">4</vcpu> <os> <type arch="x86_64" machine="pc-q35-6.0">hvm</type> <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram> <bootmenu enable="yes"/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state="on"/> <vapic state="on"/> <spinlocks state="on" retries="8191"/> </hyperv> <vmport state="off"/> </features> <cpu mode="host-model" check="partial"> <topology sockets="1" dies="1" cores="4" threads="1"/> </cpu> <clock offset="localtime"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled="no"/> <suspend-to-disk enabled="no"/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type="file" device="cdrom"> <driver name="qemu" type="raw"/> <source file="/home/ruby/Desktop/AME_2004_(2020-09-16).iso"/> <target dev="sda" bus="sata"/> <readonly/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk> <controller type="usb" index="0" model="qemu-xhci" ports="15"> <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/> </controller> <controller type="sata" index="0"> <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/> </controller> <controller type="pci" index="0" model="pcie-root"/> <controller type="pci" index="1" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="1" port="0x8"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/> </controller> <controller type="pci" index="2" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="2" port="0x9"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/> </controller> <controller type="pci" index="3" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="3" port="0xa"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/> </controller> <controller type="pci" index="4" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="4" port="0xb"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/> </controller> <controller type="pci" index="5" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="5" port="0xc"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/> </controller> <controller type="pci" index="6" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="6" port="0xd"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/> </controller> <controller type="pci" index="7" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="7" port="0xe"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x6"/> </controller> <controller type="pci" index="8" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="8" port="0xf"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x7"/> </controller> <controller type="pci" index="9" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="9" port="0x10"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/> </controller> <controller type="pci" index="10" model="pcie-root-port"> <model name="pcie-root-port"/> <target chassis="10" port="0x11"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/> </controller> <input type="mouse" bus="ps2"/> <input type="keyboard" bus="ps2"/> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/> </source> <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x01" slot="0x00" function="0x1"/> </source> <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x01" slot="0x00" function="0x2"/> </source> <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x01" slot="0x00" function="0x3"/> </source> <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/> </source> <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x06" slot="0x00" function="0x0"/> </source> <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/> </hostdev> <hostdev mode="subsystem" type="pci" managed="yes"> <source> <address domain="0x0000" bus="0x07" slot="0x00" function="0x0"/> </source> <boot order="1"/> <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/> </hostdev> <memballoon model="virtio"> <address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/> </memballoon> </devices> </domain> Contents of " /etc/modprobe.d/vfio.conf": Spoiler options vfio-pci ids=8086:1901,10de:21c4,10de:1aeb,10de:1aec,10de:1aed,10ec:8161,1b21:1142,1b21:0612 "lspci -nnk" returns: Spoiler [ruby@Tempest ~]$ lspci -nnk 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:9b43] (rev 05) DeviceName: Onboard - Other Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] Kernel driver in use: skl_uncore 00:01.0 PCI bridge [0604]: Intel Corporation 6th-9th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05) Kernel driver in use: pcieport 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:9bc5] (rev 05) DeviceName: Onboard - Video Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] Kernel driver in use: i915 Kernel modules: i915 00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911] DeviceName: Onboard - Other Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] 00:12.0 Signal processing controller [1180]: Intel Corporation Comet Lake PCH Thermal Controller [8086:06f9] DeviceName: Onboard - Other Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] Kernel driver in use: intel_pch_thermal Kernel modules: intel_pch_thermal 00:14.0 USB controller [0c03]: Intel Corporation Comet Lake USB 3.1 xHCI Host Controller [8086:06ed] DeviceName: Onboard - Other Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 00:14.2 RAM memory [0500]: Intel Corporation Comet Lake PCH Shared SRAM [8086:06ef] DeviceName: Onboard - Other Subsystem: Intel Corporation Device [8086:7270] 00:16.0 Communication controller [0780]: Intel Corporation Comet Lake HECI Controller [8086:06e0] DeviceName: Onboard - Other Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] Kernel driver in use: mei_me Kernel modules: mei_me 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:06d2] DeviceName: Onboard - SATA Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] Kernel driver in use: ahci 00:1b.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #17 [8086:06c0] (rev f0) Kernel driver in use: pcieport 00:1b.4 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #21 [8086:06ac] (rev f0) Kernel driver in use: pcieport 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:06b8] (rev f0) Kernel driver in use: pcieport 00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:06bc] (rev f0) Kernel driver in use: pcieport 00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:06be] (rev f0) Kernel driver in use: pcieport 00:1c.7 PCI bridge [0604]: Intel Corporation Device [8086:06bf] (rev f0) Kernel driver in use: pcieport 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:0685] DeviceName: Onboard - Other Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] 00:1f.3 Audio device [0403]: Intel Corporation Comet Lake PCH cAVS [8086:06c8] DeviceName: Onboard - Sound Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:9c75] Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel, snd_soc_skl, snd_sof_pci 00:1f.4 SMBus [0c05]: Intel Corporation Comet Lake PCH SMBus Controller [8086:06a3] DeviceName: Onboard - Other Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH SPI Controller [8086:06a4] DeviceName: Onboard - Other Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 SUPER] [10de:21c4] (rev a1) Subsystem: eVga.com. Corp. Device [3842:1068] Kernel driver in use: vfio-pci Kernel modules: nouveau 01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1) Subsystem: eVga.com. Corp. Device [3842:1068] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1) Subsystem: eVga.com. Corp. Device [3842:1068] Kernel driver in use: vfio-pci Kernel modules: xhci_pci 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1) Subsystem: eVga.com. Corp. Device [3842:1068] Kernel driver in use: vfio-pci Kernel modules: i2c_nvidia_gpu 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Device [10ec:8161] (rev 15) Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:8168] Kernel driver in use: vfio-pci Kernel modules: r8169 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 04) Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7c75] Kernel driver in use: r8169 Kernel modules: r8169 06:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] Subsystem: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] Kernel driver in use: vfio-pci Kernel modules: xhci_pci 07:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02) Subsystem: ASMedia Technology Inc. Device [1b21:1060] Kernel driver in use: vfio-pci UPDATE: I tried without the sata controller or USB controller, only the ethernet controller. I used a vfio virtual disk. It worked really really well. I then tried adding the sata controller and it crashed while installing windows. I went back to no SATA controller and used a virtual disk, but this time added the USB controller. I installed windows fine but it locked up when I tried to copy files through the usb controller from an external drive. I read online that some USB controllers cause problems for PCIe pass through, and both of the controllers I have in my possession have offending chips on them. I tried to look for information on known iffy SATA controllers but I could not find much. For now I will install barrier and deal with a vfio disk but the point of me setting this up was to have a low latency virtual machine and I would still very much like to fix this soon. Thank you.
Have you thought about using Looking Glass instead of routing USB controllers through the VM? VirtIO aims to deliver close-to-the-metal performance for network and storage devices. You don’t always need to handle a SATA controller manually. This is especially useful if the device is integrated into your chipset and not in its own IOMMU group. Additionally, you might want to disable the host kernel driver at startup.
Thanks for your response. The glass reflection looks really good. I’ll take a closer look. I’m using a SATA controller with a PCIe card, along with USB and Ethernet connections. Each had its own IOMMU configuration, and I’ve checked for reset problems—everything seems fine. I’ve also reserved the devices in GRUB so the host can’t access them directly. This is because I want my virtual machine drives to be physical rather than relying on a VirtIO drive. Additionally, I need a separate network link for my VM from the host computer since I use a VPN there. I want to avoid latency when playing online games in my VM. The USB controller lets me use a physical switch for keyboard and mouse.
Looking Glass has no audio support. Consider using Scream Audio if you need sound in this VM. I've got a complete guide on VFIO, QEMU/KVM, and Scream Audio for an older Debian-based Linux system. If you're on Arch, your instructions might differ. Check Step #9 for Scream Audio. This can also assist with Looking Glass. Please update me on your progress. If the VM freezes due to limited hardware, other problems might be at play. You're likely using a NVIDIA beta driver, right? Otherwise, you're employing the Code 43 workaround, correct?
On a X399 setup, Scream performed well. You send the audio through a network protocol from your local machine and then share it across the LAN. Anyone connected can access the stream. You can automate everything, making it simpler to start Looking Glass and Scream together.