[RESOLVED] PCI Passthrough in KVM

Issues related to hardware problems
simon_lefisch
Posts: 91
Joined: 2017/07/12 21:02:02

[RESOLVED] PCI Passthrough in KVM

Post by simon_lefisch » 2020/05/27 02:24:08

Hello everyone,

I have been scouring the web for an answer but could not find one that was clear.

I am trying to setup a FreeNAS VM on KVM and to say that I should passthrough the PCIe bus that the HDDs are on. I have a Marvell PCIe card installed but do not how to pass it thru. tried adding it in virt-manager but the VM would not boot do to an error "unsupported configuration: host doesn't support passthrough of host PCI devices." Do I need to set iommu in grub or can CentOS not passthrough PCI devices at all?
Last edited by simon_lefisch on 2020/05/28 22:39:10, edited 1 time in total.
Hardware:
Supermicro X10SRi-F mobo
E5-2683v4 CPU
112GB RAM
2x 250GB SSD RAID1 (OS)
2x 500GB SSD RAID1 (VM Storage)
2x 4TB HDD RAID 1 (Data Storage)

User avatar
jlehtone
Posts: 2870
Joined: 2007/12/11 08:17:33
Location: Finland

Re: PCI Passthrough in KVM

Post by jlehtone » 2020/05/27 11:02:35

I have network card's SR-IOV virtual functions passthrough to libvirt/KVM.
I think I've had NIC passthrough without SR-IOV too.
SR-IOV in itself requires iommu kernel option.

I had some reason not to passthrough NIC on CentOS 5, nine years ago.

For disks, I've used only VirtIO. (I have host connecting to iSCSI targets in SAN and VM's using those volumes, even though a VM could use iSCSI directly.)
https://passthroughpo.st/disk-passthrough-explained/

simon_lefisch
Posts: 91
Joined: 2017/07/12 21:02:02

Re: PCI Passthrough in KVM

Post by simon_lefisch » 2020/05/27 17:18:20

I did come across that article, was quite helpful. I also came across this red hat article as well which talks about iommu. I'm assuming this would be the correct way to pas the disk thru? Forgive the noob questions as this is the first time I am attempting this and want to make sure I do it correctly. I have found so many other articles on how to pass a disk through to a VM that it's difficult to know which way is right.

This is part of my /boot/grub2/grub.cfg file

Code: Select all

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/00_tuned ###
set tuned_params=""
set tuned_initrd=""
### END /etc/grub.d/00_tuned ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'CentOS Linux (3.10.0-1127.8.2.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.9.1.el7.x86_64-advanced-4311ffb0-df38-490e-9547-ef4ac056c005' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod part_gpt
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/93b43d046fd14a8b58d38244d62a280d'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/93b43d046fd14a8b58d38244d62a280d'  26e2826f-a84e-4f3f-b883-768d2b5a14eb
	else
	  search --no-floppy --fs-uuid --set=root 26e2826f-a84e-4f3f-b883-768d2b5a14eb
	fi
	linuxefi /vmlinuz-3.10.0-1127.8.2.el7.x86_64 root=UUID=4311ffb0-df38-490e-9547-ef4ac056c005 ro crashkernel=auto rd.md.uuid=271f5c13:114bba41:fdc9def6:51883821 rd.md.uuid=93b43d04:6fd14a8b:58d38244:d62a280d rd.md.uuid=a3d488a7:666a4314:200b5ed4:db061329 LANG=en_US.UTF-8
	initrdefi /initramfs-3.10.0-1127.8.2.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.9.1.el7.x86_64-advanced-4311ffb0-df38-490e-9547-ef4ac056c005' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod part_gpt
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/93b43d046fd14a8b58d38244d62a280d'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/93b43d046fd14a8b58d38244d62a280d'  26e2826f-a84e-4f3f-b883-768d2b5a14eb
	else
	  search --no-floppy --fs-uuid --set=root 26e2826f-a84e-4f3f-b883-768d2b5a14eb
	fi
	linuxefi /vmlinuz-3.10.0-1127.el7.x86_64 root=UUID=4311ffb0-df38-490e-9547-ef4ac056c005 ro crashkernel=auto rd.md.uuid=271f5c13:114bba41:fdc9def6:51883821 rd.md.uuid=93b43d04:6fd14a8b:58d38244:d62a280d rd.md.uuid=a3d488a7:666a4314:200b5ed4:db061329 LANG=en_US.UTF-8
	initrdefi /initramfs-3.10.0-1127.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.9.1.el7.x86_64-advanced-4311ffb0-df38-490e-9547-ef4ac056c005' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod part_gpt
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/93b43d046fd14a8b58d38244d62a280d'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/93b43d046fd14a8b58d38244d62a280d'  26e2826f-a84e-4f3f-b883-768d2b5a14eb
	else
	  search --no-floppy --fs-uuid --set=root 26e2826f-a84e-4f3f-b883-768d2b5a14eb
	fi
	linuxefi /vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=4311ffb0-df38-490e-9547-ef4ac056c005 ro crashkernel=auto rd.md.uuid=271f5c13:114bba41:fdc9def6:51883821 rd.md.uuid=93b43d04:6fd14a8b:58d38244:d62a280d rd.md.uuid=a3d488a7:666a4314:200b5ed4:db061329 LANG=en_US.UTF-8
	initrdefi /initramfs-3.10.0-1062.18.1.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.9.1.el7.x86_64-advanced-4311ffb0-df38-490e-9547-ef4ac056c005' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod part_gpt
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/93b43d046fd14a8b58d38244d62a280d'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/93b43d046fd14a8b58d38244d62a280d'  26e2826f-a84e-4f3f-b883-768d2b5a14eb
	else
	  search --no-floppy --fs-uuid --set=root 26e2826f-a84e-4f3f-b883-768d2b5a14eb
	fi
	linuxefi /vmlinuz-3.10.0-1062.12.1.el7.x86_64 root=UUID=4311ffb0-df38-490e-9547-ef4ac056c005 ro crashkernel=auto rd.md.uuid=271f5c13:114bba41:fdc9def6:51883821 rd.md.uuid=93b43d04:6fd14a8b:58d38244:d62a280d rd.md.uuid=a3d488a7:666a4314:200b5ed4:db061329 LANG=en_US.UTF-8
	initrdefi /initramfs-3.10.0-1062.12.1.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-1062.9.1.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.9.1.el7.x86_64-advanced-4311ffb0-df38-490e-9547-ef4ac056c005' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod part_gpt
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/93b43d046fd14a8b58d38244d62a280d'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/93b43d046fd14a8b58d38244d62a280d'  26e2826f-a84e-4f3f-b883-768d2b5a14eb
	else
	  search --no-floppy --fs-uuid --set=root 26e2826f-a84e-4f3f-b883-768d2b5a14eb
	fi
	linuxefi /vmlinuz-3.10.0-1062.9.1.el7.x86_64 root=UUID=4311ffb0-df38-490e-9547-ef4ac056c005 ro crashkernel=auto rd.md.uuid=271f5c13:114bba41:fdc9def6:51883821 rd.md.uuid=93b43d04:6fd14a8b:58d38244:d62a280d rd.md.uuid=a3d488a7:666a4314:200b5ed4:db061329
	initrdefi /initramfs-3.10.0-1062.9.1.el7.x86_64.img
}
menuentry 'CentOS Linux (0-rescue-cfc6701036054b7284e570f25f8db89d) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-0-rescue-cfc6701036054b7284e570f25f8db89d-advanced-4311ffb0-df38-490e-9547-ef4ac056c005' {
	load_video
	insmod gzio
	insmod part_gpt
	insmod part_gpt
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/93b43d046fd14a8b58d38244d62a280d'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/93b43d046fd14a8b58d38244d62a280d'  26e2826f-a84e-4f3f-b883-768d2b5a14eb
	else
	  search --no-floppy --fs-uuid --set=root 26e2826f-a84e-4f3f-b883-768d2b5a14eb
	fi
	linuxefi /vmlinuz-0-rescue-cfc6701036054b7284e570f25f8db89d root=UUID=4311ffb0-df38-490e-9547-ef4ac056c005 ro crashkernel=auto rd.md.uuid=271f5c13:114bba41:fdc9def6:51883821 rd.md.uuid=93b43d04:6fd14a8b:58d38244:d62a280d rd.md.uuid=a3d488a7:666a4314:200b5ed4:db061329
	initrdefi /initramfs-0-rescue-cfc6701036054b7284e570f25f8db89d.img
}
if [ "x$default" = 'CentOS Linux (3.10.0-1062.9.1.el7.x86_64) 7 (Core)' ]; then default='Advanced options for CentOS Linux>CentOS Linux (3.10.0-1062.9.1.el7.x86_64) 7 (Core)'; fi;
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###
According to the article, I have to append the intel_iommu line in the kernel line, however I do not see the anywhere. Does that mean it's not possible to pass a PCI bus thru to a VM? Or is it something I need to activate in BIOS? If the latter, I thought it was already activated but I will have to check. Or is there a way in CLI for me to know whether it is possible to pass the disk thru?
Hardware:
Supermicro X10SRi-F mobo
E5-2683v4 CPU
112GB RAM
2x 250GB SSD RAID1 (OS)
2x 500GB SSD RAID1 (VM Storage)
2x 4TB HDD RAID 1 (Data Storage)

User avatar
TrevorH
Forum Moderator
Posts: 28831
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: PCI Passthrough in KVM

Post by TrevorH » 2020/05/27 17:20:19

Are you really sure you need to pass the entire thing through? Is the 1 or 2% performance gain involved really worth the hassle?
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

simon_lefisch
Posts: 91
Joined: 2017/07/12 21:02:02

Re: PCI Passthrough in KVM

Post by simon_lefisch » 2020/05/27 17:25:23

Apparently it is needed in order to get FreeNAS to run optimally, especially with ZFS. I have extra disks available to use and would like to get it going if possible. Otherwise I may have to look at another host alternative (ie; ESXi, Proxmox). the biggest reason is that I would like to experiment with it before making the jump so if its possible to pass the PCI bus thru then cool. I have the disks on a separate PCI card which is what I want to passthrough.

https://www.ixsystems.com/blog/yes-you- ... e-freenas/
Hardware:
Supermicro X10SRi-F mobo
E5-2683v4 CPU
112GB RAM
2x 250GB SSD RAID1 (OS)
2x 500GB SSD RAID1 (VM Storage)
2x 4TB HDD RAID 1 (Data Storage)

User avatar
jlehtone
Posts: 2870
Joined: 2007/12/11 08:17:33
Location: Finland

Re: PCI Passthrough in KVM

Post by jlehtone » 2020/05/27 19:41:58

simon_lefisch wrote:
2020/05/27 17:18:20
This is part of my /boot/grub2/grub.cfg file
And it says quite clearly:

Code: Select all

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
What you should do is to edit the /etc/default/grub and then run

Code: Select all

grub2-mkconfig -o /boot/grub2/grub.cfg
The /etc/default/grub has the command line options in a variable GRUB_CMDLINE_LINUX
See https://access.redhat.com/documentation ... Menu_Entry

simon_lefisch
Posts: 91
Joined: 2017/07/12 21:02:02

Re: PCI Passthrough in KVM

Post by simon_lefisch » 2020/05/27 21:15:33

jlehtone wrote:
2020/05/27 19:41:58
And it says quite clearly:

Code: Select all

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
What you should do is to edit the /etc/default/grub and then run

Code: Select all

grub2-mkconfig -o /boot/grub2/grub.cfg
The /etc/default/grub has the command line options in a variable GRUB_CMDLINE_LINUX
See https://access.redhat.com/documentation ... Menu_Entry
*derp* yes of course I will not edit that lol. Thank you tho for your help with this. I will give this a try and let you know what happens.
Hardware:
Supermicro X10SRi-F mobo
E5-2683v4 CPU
112GB RAM
2x 250GB SSD RAID1 (OS)
2x 500GB SSD RAID1 (VM Storage)
2x 4TB HDD RAID 1 (Data Storage)

simon_lefisch
Posts: 91
Joined: 2017/07/12 21:02:02

Re: PCI Passthrough in KVM

Post by simon_lefisch » 2020/05/28 00:05:20

jlehtone wrote:
2020/05/27 19:41:58
What you should do is to edit the /etc/default/grub and then run

Code: Select all

grub2-mkconfig -o /boot/grub2/grub.cfg
The /etc/default/grub has the command line options in a variable GRUB_CMDLINE_LINUX
See https://access.redhat.com/documentation ... Menu_Entry
Ok so I ran edited /etc/default/grub and added intel_iommu=on, rebooted, added the PCI card in virt-manager, powered on the VM but still the same error. I looked up requirements for pass through and I found a post where I can run dmesg | grep 'IOMMU enabled' to see if it is on but it came back with nothing....which sounds like I can't pass through anything. I am contacting my mobo manufacturer to verify. Is there southing else I need to do to get this to work or am I just SOL?
Hardware:
Supermicro X10SRi-F mobo
E5-2683v4 CPU
112GB RAM
2x 250GB SSD RAID1 (OS)
2x 500GB SSD RAID1 (VM Storage)
2x 4TB HDD RAID 1 (Data Storage)

User avatar
TrevorH
Forum Moderator
Posts: 28831
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: PCI Passthrough in KVM

Post by TrevorH » 2020/05/28 09:04:06

Try dmesg | grep -i iommu
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

simon_lefisch
Posts: 91
Joined: 2017/07/12 21:02:02

Re: PCI Passthrough in KVM

Post by simon_lefisch » 2020/05/28 16:15:42

TrevorH wrote:
2020/05/28 09:04:06
Try dmesg | grep -i iommu

Code: Select all

user@server:~$ dmesg|grep -i iommu
[    0.118827] DMAR-IR: IOAPIC id 1 under DRHD base  0xfbffc000 IOMMU 0
[    0.118831] DMAR-IR: IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 0
Let me ask you this @TrevorH, does the base package allow for PCI pass-through or would I need to use the qemu-kvm-ev package provided by SIG? If the latter, can I install qemu-kvm-ev over my existing qemu-kvm or would I have to remove the qemu-kvm package first?
Hardware:
Supermicro X10SRi-F mobo
E5-2683v4 CPU
112GB RAM
2x 250GB SSD RAID1 (OS)
2x 500GB SSD RAID1 (VM Storage)
2x 4TB HDD RAID 1 (Data Storage)

Post Reply

Return to “CentOS 7 - Hardware Support”