VT-d with KVM

General support questions
Post Reply
AndyBurns
Posts: 5
Joined: 2014/08/25 20:48:55

VT-d with KVM

Post by AndyBurns » 2014/08/25 21:27:07

I'm fairly well versed in PCI passthrough using CentOS 6 and the gitco.de Xen repo with software IOMMU.

Having just upgraded my hardware, I am investigating the options using CentOS 7 and KVM with VT-d IOMMU

Hardware is an ASrock H97 Pro4 motherboard (latest P1.40 firmware), an i5-4690T processor, 16GB memory, no separate GPU, VT-x and VT-d both enabled in BIOS

# uname -a

Linux localhost.localdomain 3.10.0-123.6.3.el7.x86_64 #1 SMP Wed Aug 6 21:12:36 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

# cat /proc/cpuinfo

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i5-4690T CPU @ 2.50GHz
stepping : 3
microcode : 0x1a
cpu MHz : 3406.933
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm
bogomips : 4988.38
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

# lspci -tv

-[0000:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller
+-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
+-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
+-14.0 Intel Corporation Device 8cb1
+-16.0 Intel Corporation Device 8cba
+-19.0 Intel Corporation Ethernet Connection (2) I218-V
+-1a.0 Intel Corporation Device 8cad
+-1b.0 Intel Corporation Device 8ca0
+-1c.0-[01]--
+-1c.3-[02-05]----00.0-[03-05]--+-01.0-[04]----00.0 Philips Semiconductors SAA7231
| \-02.0-[05]----00.0 Philips Semiconductors SAA7231
+-1c.6-[06-08]----00.0-[07-08]----01.0-[08]--+-04.0 Brooktree Corporation Bt878 Video Capture
| +-04.1 Brooktree Corporation Bt878 Audio Capture
| +-05.0 Brooktree Corporation Bt878 Video Capture
| \-05.1 Brooktree Corporation Bt878 Audio Capture
+-1d.0 Intel Corporation Device 8ca6
+-1f.0 Intel Corporation Device 8cc6
+-1f.2 Intel Corporation Device 8c82
\-1f.3 Intel Corporation Device 8ca2

It will be the BT878 camera capture and SAA7231 tuner cards to be passed to a Fedora 20 virtual machine, though the specifics are not especially relevant, the VM was first installed without any PCI devices, and they were added to the config later

# virsh dumpxml f20

<domain type='kvm'>
<name>f20</name>
<uuid>ca7023d6-e9e1-49d4-aea4-2c0c80e09f4b</uuid>
<memory unit='KiB'>3018752</memory>
<currentMemory unit='KiB'>3018752</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/var/lib/libvirt/images/f20.img'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:a1:7c:d2'/>
<source network='default'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<graphics type='spice' autoport='yes'/>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x00' slot='0x1c' function='0x3'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</memballoon>
</devices>
</domain>

But now the PCI devices have been added, the VM will not start

# virsh start f20

error: Failed to start domain f20
error: unsupported configuration: host doesn't support passthrough of host PCI devices

I thought the IOMMU and DMAR messages indicated that VT-d had been recognised and PCI passthrough should be supported

# dmesg | grep -iE "dmar|iommu"

[ 0.000000] ACPI: DMAR 00000000cd63dbc0 000B8 (v01 INTEL BDW 00000001 INTL 00000001)
[ 0.044440] dmar: Host address width 39
[ 0.044442] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.044450] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[ 0.044451] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.044456] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[ 0.044457] dmar: RMRR base: 0x000000cdea8000 end: 0x000000cdeb6fff
[ 0.044458] dmar: RMRR base: 0x000000cf000000 end: 0x000000df1fffff
[ 0.044524] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1

Anyone have any experience or suggestions?
Last edited by AndyBurns on 2014/08/26 00:57:17, edited 1 time in total.

AndyBurns
Posts: 5
Joined: 2014/08/25 20:48:55

Re: VT-d with KVM

Post by AndyBurns » 2014/08/25 23:50:36

OK so I figured out I needed to pass intel_iommu=on to the kernel

then figured out that grub.conf lives in /boot/efi/EFI/centos with a UEFI system

That partly helped, but abrt alerted me to a kernel warning

# dmesg | grep -iE "dmar|iommu"

[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-123.6.3.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/swap crashkernel=auto rd.lvm.lv=centos/root vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk rhgb quiet LANG=en_GB.UTF-8 intel_iommu=on
[ 0.000000] ACPI: DMAR 00000000cd63dbc0 000B8 (v01 INTEL BDW 00000001 INTL 00000001)
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-123.6.3.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/swap crashkernel=auto rd.lvm.lv=centos/root vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk rhgb quiet LANG=en_GB.UTF-8 intel_iommu=on
[ 0.000000] Intel-IOMMU: enabled
[ 0.044805] dmar: Host address width 39
[ 0.044807] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.044815] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[ 0.044816] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.044821] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[ 0.044822] dmar: RMRR base: 0x000000cdea8000 end: 0x000000cdeb6fff
[ 0.044823] dmar: RMRR base: 0x000000cf000000 end: 0x000000df1fffff
[ 0.044889] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1
[ 0.449605] DMAR: No ATSR found
[ 0.449642] IOMMU 0 0xfed90000: using Queued invalidation
[ 0.449643] IOMMU 1 0xfed91000: using Queued invalidation
[ 0.449646] IOMMU: Setting RMRR:
[ 0.449657] IOMMU: Setting identity map for device 0000:00:02.0 [0xcf000000 - 0xdf1fffff]
[ 0.451323] IOMMU: Setting identity map for device 0000:00:1d.0 [0xcdea8000 - 0xcdeb6fff]
[ 0.451352] IOMMU: Setting identity map for device 0000:00:1a.0 [0xcdea8000 - 0xcdeb6fff]
[ 0.451376] IOMMU: Setting identity map for device 0000:00:14.0 [0xcdea8000 - 0xcdeb6fff]
[ 0.451393] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 0.451400] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.451582] [<ffffffff814a9d8d>] intel_iommu_add_device+0x4d/0x230
[ 0.451585] [<ffffffff814a0e30>] ? bus_set_iommu+0x50/0x50
[ 0.451588] [<ffffffff814a0e5a>] add_iommu_group+0x2a/0x50
[ 0.451593] [<ffffffff814a0e28>] bus_set_iommu+0x48/0x50
[ 0.451597] [<ffffffff81a58273>] intel_iommu_init+0x3ea/0x413

KVM is at least willing to consider starting the VM with PCI passthrough now, the 0000:00:1f.0 device is the ISA bridge, which seems to fit with the kernel warning happening while mapping the LPC

# virsh start f20

error: Failed to start domain f20
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device vfio-pci,host=00:1c.3,id=hostdev0,bus=pci.0,addr=0x8: vfio: error, group 8 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-kvm: -device vfio-pci,host=00:1c.3,id=hostdev0,bus=pci.0,addr=0x8: vfio: failed to get group 8
qemu-kvm: -device vfio-pci,host=00:1c.3,id=hostdev0,bus=pci.0,addr=0x8: Device initialization failed.
qemu-kvm: -device vfio-pci,host=00:1c.3,id=hostdev0,bus=pci.0,addr=0x8: Device 'vfio-pci' could not be initialized

So I probably need to unbind some devices from the host PCI to let the VM get at them, and maybe bind them to pci-stub, I can experiment with that.

A quick search for the kernel warning message

[ 0.451393] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 0.451400] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.451455] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[ 0.451546] ------------[ cut here ]------------
[ 0.451552] WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x87/0x90()
[ 0.451553] Modules linked in:
[ 0.451557] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 3.10.0-123.6.3.el7.x86_64 #1
[ 0.451558] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H97 Pro4, BIOS P1.40 07/29/2014
[ 0.451559] 0000000000000000 000000003a87b4a1 ffff880408885cf0 ffffffff815e20bb
[ 0.451561] ffff880408885d28 ffffffff8105dee1 ffff880408212000 ffff88040821c000
[ 0.451563] ffff88040821c098 0000000000000000 0000000000000000 ffff880408885d38
[ 0.451565] Call Trace:
[ 0.451570] [<ffffffff815e20bb>] dump_stack+0x19/0x1b
[ 0.451574] [<ffffffff8105dee1>] warn_slowpath_common+0x61/0x80
[ 0.451576] [<ffffffff8105e00a>] warn_slowpath_null+0x1a/0x20
[ 0.451579] [<ffffffff812ec727>] pci_find_upstream_pcie_bridge+0x87/0x90
[ 0.451582] [<ffffffff814a9d8d>] intel_iommu_add_device+0x4d/0x230
[ 0.451585] [<ffffffff814a0e30>] ? bus_set_iommu+0x50/0x50
[ 0.451588] [<ffffffff814a0e5a>] add_iommu_group+0x2a/0x50
[ 0.451591] [<ffffffff813b4563>] bus_for_each_dev+0x73/0xc0
[ 0.451593] [<ffffffff814a0e28>] bus_set_iommu+0x48/0x50
[ 0.451597] [<ffffffff81a58273>] intel_iommu_init+0x3ea/0x413
[ 0.451601] [<ffffffff81a11c93>] ? memblock_find_dma_reserve+0x147/0x147
[ 0.451603] [<ffffffff81a11ca5>] pci_iommu_init+0x12/0x3c
[ 0.451606] [<ffffffff810020e2>] do_one_initcall+0xe2/0x190
[ 0.451608] [<ffffffff81a09153>] kernel_init_freeable+0x18b/0x22a
[ 0.451610] [<ffffffff81a0892b>] ? do_early_param+0x88/0x88
[ 0.451614] [<ffffffff815c3960>] ? rest_init+0x80/0x80
[ 0.451617] [<ffffffff815c396e>] kernel_init+0xe/0x180
[ 0.451619] [<ffffffff815f26ec>] ret_from_fork+0x7c/0xb0
[ 0.451621] [<ffffffff815c3960>] ? rest_init+0x80/0x80
[ 0.451626] ---[ end trace e5ab4299053e7cdf ]---

Led me to try forcing ASPM on

# dmesg | grep -iE "aspm"

[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-123.6.3.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/swap crashkernel=auto rd.lvm.lv=centos/root vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk rhgb quiet LANG=en_GB.UTF-8 intel_iommu=on pcie_aspm=force
[ 0.000000] PCIe ASPM is forcibly enabled
[ 0.105428] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[ 0.154890] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[ 0.166205] acpi PNP0A08:00: Disabling ASPM (FADT indicates it is unsupported)

But that didn't help, I'll keep trying ...
Last edited by AndyBurns on 2014/08/26 09:18:08, edited 1 time in total.

AndyBurns
Posts: 5
Joined: 2014/08/25 20:48:55

Re: VT-d with KVM

Post by AndyBurns » 2014/08/26 06:27:21

I thought I'd try the 3.16.1-1 kernel from elrepo, I realise that's probably off-topic here but it didn't help with the LPC/IOMMU warning, perhaps this patch might?

http://lists.linuxfoundation.org/piperm ... 06085.html
Last edited by AndyBurns on 2014/08/26 09:16:23, edited 1 time in total.

AndyBurns
Posts: 5
Joined: 2014/08/25 20:48:55

Re: VT-d with KVM

Post by AndyBurns » 2014/08/26 08:39:16

OK, I thought that patch was a month old instead of a year old :oops:

so I've experimented with 3.17 rc1 kernel from
http://elrepo.org/people/ajb/devel/kern ... 6_64/RPMS/

That has got rid of the kernel warning, looks like I need to read about VFIO now ...

AndyBurns
Posts: 5
Joined: 2014/08/25 20:48:55

Re: VT-d with KVM

Post by AndyBurns » 2014/08/27 20:50:52

Just to finish-off this thread in case anyone stumbles across it in a search.

The 3.17-rc1 kernel did the trick, but it is only a development version so far, once I learned how to find the other PCI devices and bridges that are grouped with the PCI devices I want to assign and unbound them, the VM started and the devices were visible inside it.

Unfortunately due to the way PCI bridges and functions are grouped on my motherboard, looks like I can't pass the two PCI cards I was planning to into two separate VMs, they will both have to be assigned to the same VM.

ILMostro
Posts: 3
Joined: 2015/04/25 05:47:31

Re: VT-d with KVM

Post by ILMostro » 2015/08/06 23:34:53

According to this related post on linux-kvm.org:
  • VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
  • If the device doesn't support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.

tthtlc
Posts: 1
Joined: 2016/02/16 23:32:15

Re: VT-d with KVM

Post by tthtlc » 2016/02/16 23:34:40

I suspect you are hitting at a deadend after all:

http://www.intel.com/content/www/us/en/ ... 05758.html

And you can see that your board H97 is not listed as the supported board for VT-d.

Post Reply

Return to “CentOS 7 - General Support”