In the course of testing our puppet manifests with beaker I came across the necessity to run KVM virtual machines inside KVM guests. Normally this has a severe performance penalty as the CPUs need to be fully emulated and cannot take advantage of performance enhancing CPU instructions. Several years ago Intel and AMD added functionality which basically enables CPU instruction passthrough to the guests to get around this limitation. You can see these extensions on a host CPU in /proc/cpuinfo
egrep --color " svm | vmx " /proc/cpuinfo
Qemu has a switch to enable nested virtualization but this has been disabled in libvirt by Redhat and has only just been renabled as a “Technology Preview” in RHEL 7.3 https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/7.3_Release_Notes/technology_previews_virtualization.html
The fun I have is that our development hosts machines are still running Centos 6.8. Reading the docs and multiple internet posts I could see that all the parts were there for this to work but I just needed to work around the limitations put in by Redhat.
First up I am using an AMD machine and could see that it had the SVM extensions in /proc/cpuinfo. Redhat/Centos comes with a command to validate a virtualization host virt-host-validate A quick yum search showed that it was provided by the libvirt-client package.
Running that command showed that my physical host was ready to go.
[root@testhost05 ~]# virt-host-validate
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking for device /dev/kvm : PASS
QEMU: Checking for device /dev/vhost-net : PASS
QEMU: Checking for device /dev/net/tun : PASS
LXC: Checking for Linux >= 2.6.26 : PASS
To test if you have nested virtualization enabled you can check under /proc for both of these you need the value to be 1.
For Intel systems you can check here.
[root@testhost05 ~]# cat /sys/module/kvm_intel/parameters/nested
1
On an AMD physical machine you can do pretty much the same thing.
[root@testhost05 ~]# cat /sys/module/kvm_amd/parameters/nested
1
If you have a value of 0 then you can pass a parameter to the driver.
Intel CPU:
echo "options kvm-intel nested=1" >> /etc/modprobe.d/kvm_intel.conf
AMD CPU:
echo "options kvm-amd nested=1" >> /etc/modprobe.d/kvm_amd.conf
You can either reboot or remove and add the module to get the host to pick up the changes..
Once I knew that the KVM driver supported the nested functionality I created a KVM guest machine using the standard virt-install tool.
/usr/bin/virt-install --name isrunner01 \
--ram 8192 \
--vcpus 4 \
--noautoconsole \
--force \
--accelerate \
--hvm \
--nographics \
--disk path=/var/lib/libvirt/images/testrunner01.root.img,size=10,bus=virtio,sparse=true \
--location http://mirror.centos.org/centos/6/os/x86_64/ \
--network network=default \
--extra-args "noipv6 console=ttyS0,115200 SERVERNAME=testrunner01
After the virtual host was finished building I logged in as root and checked where we were at with virt-install and got the following results.
[root@testrunner01 ~]# virt-host-validate
QEMU: Checking for hardware virtualization : WARN (Only emulated CPUs are available, performance will be significantly limited)
QEMU: Checking for device /dev/vhost-net : PASS
QEMU: Checking for device /dev/net/tun : PASS
LXC: Checking for Linux >= 2.6.26 : PASS
This is where I ran into problems. Multiple blog posts said that should work but looking in /dev/ I found there was no /dev/kvm and grepping /proc/cpuinfo for the virtualization extensions returned no results.
My first thought was to pass through the cpu to the guest by editing the virtual
domains xml config and adding <cpu mode='host-passthrough'/>
<domain type='kvm'>
<name>test01</name>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>4</vcpu>
<cpu mode='host-passthrough'></cpu>
It turns out that this is half the work but I still has the WARN (Only emulated CPUs are available issue.
In my search I came across the following email https://www.redhat.com/archives/libvir-list/2012-October/msg01168.html and thought that was old enough to be applicable to Centos 6. Looking around in the source I realised that libvirt was not passing the option -enable-nesting to qemu-kvm.
Not wanting to recompile libvirt or change anything from standard I went looking for options to trick libvirt into passing a option to the emulator. Finally after a lot of searching I found that you could pass arbitrary commands through to qemu and that it was built into libvirt through a schema extension. https://libvirt.org/drvqemu.html#qemucommand
The essential changes therefore are to extend the schema using the XML namespace for qemu.
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
and then to add the foloowing <qemu: tags to the bottom of the XML file.
</devices>
<qemu:commandline>
<qemu:arg value='-enable-nesting'/>
</qemu:commandline>
</domain>
After restarting the guest machine and running virt-host-validate I get the following output.
[root@testrunner01 ~]# virt-host-validate
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking for device /dev/kvm : PASS
QEMU: Checking for device /dev/vhost-net : PASS
QEMU: Checking for device /dev/net/tun : PASS
LXC: Checking for Linux >= 2.6.26 : PASS