Configure PCI passthrough for KVM on Devuan Linux 5.0
Configure PCI passthrough for KVM on Devuan Linux 5.0
I just installed an AMD/ATI FirePro W4100 in my secondary PCIe 16x port on my MOBO, alongside the 6900XT in the primary slot. I'm attempting to route the 6900XT to a Windows 11 VM via KVM and use basic Linux displays for system management. The main specs are: CPU 5950X, MOBO Asus TUF Gaming X570 Plus, GPU Radeon RX 6900XT, and FirePro W4100.
So far, I've adjusted the GRUB settings in /etc/default/grub and added a custom entry for the FirePro. The GRUB config looks like this:
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=$(lsb_release -i)
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on vfio-pci.ids=1002:73bf,1002:ab28"
I regenerated the GRUB file and created a vfio.conf to include both devices:
options vfio_pci ids=1002:73bf, 1002:ab28
After updating the initramfs, I didn’t remove any driver blacklists. My main worry is ensuring everything works smoothly without disrupting the displays. Since both GPUs share the same kernel drivers, I’m unsure how to exclude the 6900XT’s driver from using VFIO-PCI. I don’t want to disable its display driver entirely.
I appreciate any guidance, especially since I rely on Timeshift for backups and haven’t found a clear solution on the Debian wiki. No issues so far, but this setup is quite complex.
Verify your equipment works with PCI passthrough. To route a PCI device, you require a motherboard supporting IOMMU and a CPU that backs VT-d. Ensure your Linux kernel includes IOMMU and VT-d features. Set up KVM. If not installed, add it via your distribution's package manager. Set up QEMU (Quick Emulator) for virtualization tasks. Install QEMU and adjust its settings. Build a Windows VM with QEMU, defining RAM, storage, and the target hardware (AMD Radeon 6900XT). Route the GPU to the Windows VM using the appropriate PCI address. Configure the VM to use the AMD Radeon 6900XT as its display. Then, set your Linux system to utilize the AMD FirePro W4100 for output display. Apply the necessary drivers and select it as the main screen in your Linux settings.
Don't need to exclude the driver. My configuration matches yours, and I didn't block the Radeon driver. You haven't mentioned enabling iommu... Probably you did this already. Check the IOMMU groups using `find /sys/kernel/iommu_groups/ -type l`. Ensure the GPU is in its own group; otherwise, it won't pass. If it's not in a dedicated group, use acs_override to isolate it. When deploying in a VM, include both video and audio streams, plus set multifunction=on for video. Also, make sure a monitor is connected if you're using it with the GPU. I've faced similar problems when the monitor isn't attached.
I've turned on IOMMU in my BIOS and updated the GRUB configuration. I'm not sure if everything in the original post is accurate or if I should also set iommu=pt. From what I understand, IOMMU is fully activated. The main issue is having a driver blacklisted, even though I know it's unnecessary. The 6900XT only uses amdgpu, unlike the W4100 which relies on Radeon. Another part of this setup needs the 6900XT to launch with the vfio-pci driver instead of amdgpu. Despite following all possible steps, lspci -nnk still lists amdgpu as the kernel driver under the 6900XT. I'm getting closer now that I see what's happening—when I checked both devices via lspci -nnk, they were grouped together in the same IOMMU group. Here are the details from each:
- 0e:00.0 VGA compatible controller
[AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT]
Kernel driver in use: amdgpu
- 1002:73bf
Audio device
Kernel modules: amdgpu
- 1462:3952
Kernel driver in use: snd_hda_intel
- 1002:ab28
Kernel modules: snd_hda_intel
I tried a script earlier that showed both devices belong to the same IOMMU group, which should resolve the problem. It seems easier if I had an Nvidia card, though that feels odd given Nvidia's ties with Linux.
EDIT: The list of devices for the 6900XT when running lspci -nnk includes both, indicating they're part of the same IOMMU group.
Kernel driver in use: vfio-pci Kernel modules: amdgpu I have a very similar configuration you're attempting, but my kernel remains static. That's the setup I rely on (and I don't install amd_iommu since it's not needed for a static kernel). However, it stopped working after upgrading to kernel 6.x (it still functions on the 5.x lineup), which likely stems from my brief focus on `make oldconfig`. Since you operate with various kernel modules—both mine and yours use amdgpu—I suggest you follow this tutorial. More importantly, review the scripts he provides to safely "disassemble" your current Linux system and enable GPU access for your Windows VM.
The main reason I recommend this approach is that having passthrough on a single GPU is the most complex configuration available. While it isn't your ultimate goal, shifting the passthrough initialization into userspace lets you evaluate the kernel's backend support, BIOS settings, and IOMMU groupings without needing to modify grub or reboot just to test alternate configurations (which could involve udev issues).
For experienced Linux users, some of his choices may seem a bit awkward, but he acknowledges them and completes the task. He demonstrates significant research and effort to reach this outcome, which deserves recognition. I'm now examining my linux-6.config; I hope these resources assist you in progressing...
EDIT: I resolved the 6.x issue by adding "guest VM drivers," which allowed the kernel to pass the spare GPU to itself—an unusual situation.
I considered a few points recently. Are you receiving video output at any point during startup on the 6900XT? You should adjust your BIOS to point the primary GPU to the WS4100, or else the drivers won’t work properly. For testing, you might try moving the 6900XT into a secondary slot to check if the VFIO driver loads. I connected to my workstations... I’m about to share some GRUB configurations I’m using in the hope they assist. On my AMD server, I set GRUB_CMDLINE_LINUX_DEFAULT to include iommu settings and disabled certain options. I’m excluding NVIDIA drivers in pve-blacklist.conf for that machine (to allow the Tesla card). I’m running Ubuntu on an X299, passing through a 3060Ti to Windows 11, and a 6600XT to Mac OS. My older W7000 is used for the Ubuntu display. My current GRUB line is: quiet intel_iommu=on iommu=pt video=vesa:off vfio-pci.ids=.... I only have vfio.conf with a specific line, but it seems incomplete.
So, with all of this, would you all be able to help me create and consolidate the step by step process? As stated earlier, I want to get what I'm doing sort of "fact checked" to see if it's all correct. 1. Update and configure GRUB to include the following: GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio-pci.ids=1002:73bf,1002:ab28" (The vfio-pci.ids are the 6900XT and it's associated sound card. Are these the correct ID's I should be using when running lspci -nnk?) 2. Update GRUB with: sudo grub-mkconfig -o /boot/grub/grub.cfg Reboot. 3. Edit the vfio.conf file to add the 6900XT and associated sound card to launch with the vfio-pci driver instead of their normal one: sudo nano /etc/modprobe.d/vfio.conf. Following line(s) are added: options vfio_pci ids=1002:73bf, 1002:ab28 4. Update the initramfs with: sudo update-initramfs -u Reboot. 5. Verify that the vfio-pci driver is the kernel driver in use for the 6900XT and associated sound card. lspci -k 6. In KVM Windows 11 VM config, select the 6900XT and associated sound card to add to the list of hardware installed to the VM. 7. Start the VM and verify in Device Manager, under display adapters that an RX 6900XT is installed and functioning. Does anyone see any possible issues with this? I really do appreciate all of the help I've gotten with this.
I began it by simply installing the 6900XT without proper PCI passthrough configuration, which caused my entire system to fail. I had to use CTRL+ALT+F3 to terminate KDE and access the command line to restart. Even after that, the Windows 11 VM kept shutting down. I think if I try to launch it again now, it would behave the same way.