Attention

You are viewing an older version of the documentation. The latest version is v3.3.

KVM Hypervisor

Kernel-based Virtual Machine (KVM) is an open source virtualization technology for Linux* on x86 hardware containing virtualization extensions (for instance, Intel® Virtualization Technology (Intel® VT)).

KVM consists of loadable kernel modules, for example, kvm.ko that provides the core virtualization infrastructure. KVM turns Linux into a hypervisor that allows host machine to run multiple, isolated virtual environments called virtual machine, running unmodified Linux or Windows images.

Each virtual machine can have private virtual BIOS (basic input/output system) and private virtualized hardware: processor, memory, network cards, disk, graphics adapter, and so on.

KVM Terminology

Term	Description
VM	Hypervised operating systems (OS) are referred to as virtual machines or VMs.
GVT-d	Graphics Virtualization Technology - VT-d
IGD	Integrated Graphics Device
BDF	Bus:Device.Function notation used to succinctly describe PCI and PCIe devices.

KVM Prerequisites

The following section is applicable to:

KVM and relevant modules are pre-installed in Linux kernel.

Do the following to set up the system:

1. Add intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=8 to the kernel command line. Make sure that the value of hugepagesz*hugepages does not exceed the size of the physical memory.

2. Setup the ECI APT repository.

3. Run the following command to install this component:

   $ sudo apt install qemu-system-x86
   

4. Disable graphical desktop:

   $ sudo systemctl set-default multi-user.target
   

Set up QEMU Command to Boot Windows VM

The following section is applicable to:

This section describes the procedure to start a Windows VM with KVM hypervisor on Intel® Edge Controls for Industrial (Intel® ECI or ECI).

Basic/Minimal Configuration of KVM

The following is a basic or minimal configuration of KVM:

$ sudo qemu-system-x86_64 -nodefaults -serial mon:stdio \
-name ${vm_name},debug-threads=on \
-machine kernel_irqchip=on \
-enable-kvm \
-bios OVMF.fd \
-cpu host,hv_time \
-smp ${vcpu_num} -m ${mem_size} -mem-prealloc -mem-path /dev/hugepages \

This configuration creates a VM named as ${vm_name} with ${vcpu_num} cores, ${mem_size} memory, full acceleration with KVM in-kernel irqchip support, empowered by hugepage memory. The -nodefaults does not specify a default device, so only one host PCI bridge and one ISA bridge is attached to this VM. debug-threads=on is useful when setting vCPU affinity. For details on setting the vCPU thread affinity, refer to KVM vCPU Affinity. To build OVMF.fd, which is a virtual UEFI binary file, refer to Build OVMF.fd for KVM.

Passthrough Generic PCI Device on KVM

To passthrough a PCI device via VT-d, make sure that the intel_iommu=on kernel command line parameter is enabled.

Device passthrough requires that the vfio-pci kernel module is loaded, and will bind to the target device. If vfio-pci is not a built-in kernel module, modprobe it after the kernel completes booting.

$ sudo modprobe vfio-pci vfio_iommu_type1

Use the following function to passthrough specific devices and all other devices in the same IOMMU group. The following example shows how to passthrough a USB controller fixed at 0000:00:14.0. You can determine the value ${pci_BDF} using the command lspci.

function vfio_bind()
{
   local bdf dev vendor device
   for bdf in "$@"; do
      echo "vfio bind: $bdf ..."
      for dev in $(ls /sys/bus/pci/devices/$bdf/iommu_group/devices); do
            vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
            device=$(cat /sys/bus/pci/devices/$dev/device)
            if [ -e /sys/bus/pci/devices/$dev/driver ]; then
               echo "    $dev : unbind driver ..."
               sudo sh -c "echo $dev > /sys/bus/pci/devices/$dev/driver/unbind"
            fi
            echo "    $dev : bind to vfio ..."
            sudo sh -c "echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id"
      done
   done
}

$ vfio_bind 0000:00:14.0

Now, the device can be assigned to QEMU. The following parameter will attach the host ${pci_BDF} device, for example the parameter attaches 0000:00:14.0 to the VM PCI bus. Multiple devices can be attached to a VM with different IDs.

-device vfio-pci,host=${pci_BDF}

If the vfio-pci kernel module is not needed for the target device, unbind it so that the VM can bind the original driver back. See the following example of a USB controller fixed at 0000:00:14.0:

$ usb_device=$(cat /sys/bus/pci/devices/0000\:00\:14.0/device)
$ sudo sh -c "echo 0000:00:14.0 > /sys/bus/pci/drivers/vfio-pci/unbind"
$ sudo sh -c "echo 8086 $usb_device >  /sys/bus/pci/drivers/vfio-pci/remove_id"
$ sudo sh -c "echo 0000:00:14.0 > /sys/bus/pci/drivers/xhci_hcd/bind"

Passthrough KVM Graphics Device

GVT-d is special case for Passthrough Generic PCI Device on KVM. For the integrated graphics device (IGD), the first step is to rebind the vfio-pci kernel module as a device driver. Usually, IGD is fixed at 0000:00:02.0.

$ vfio_bind 0000:00:02.0

Passthrough of the IGD is similar to that of a generic PCI device, except for some extra parameters to QEMU.

• The x-igd-gms property sets a value multiplied by 32 as the amount of pre-allocated memory (in units of MB) to support IGD in VGA modes.

• The x-igd-opregion property exposes opregion (VBT included) to guest driver so that the guest driver could parse display connector information from. This property mandatory for the Windows VM to enable display output.

The following command shows the IGD device parameter for QEMU:

-device vfio-pci,host=00:02.0,x-igd-gms=2,id=hostdev0,x-igd-opregion=on \

Set up KVM Storage Device

It is recommended to use iothread and virtio-blk drive (Red Hat VirtIO SCSI Disk) since the disk read and write performance is much better than using default QEMU IDE drive.

-object iothread,id=io1 \
-device virtio-blk-pci,drive=disk0,iothread=io1 \
-drive if=none,id=disk0,format=raw,aio=threads,file=${vm_image} \

${vm_image} is the Windows image file. Follow this section to create a new Windows VM image.

Set up KVM Network Device

An Ethernet or wireless adapter can be assigned to a VM either with emulated Ethernet and network backend or via a generic PCI device passthrough. Refer to Passthrough Generic PCI Device on KVM for exclusive assignment. Attach a virtio Ethernet adapter using TAP network backend with id tap0 and configure the ${BRIDGE_SCRIPT} script .

-netdev tap,id=net0,ifname=tap0,script=${BRIDGE_SCRIPT} -device virtio-net-pci,netdev=net0 \

The following is an example of ${BRIDGE_SCRIPT}, which can support either public bridge or private virtual bridge by setting the switch to the target bridge name.

#!/bin/sh
set -x
switch=br0
ip link set "$1" up
if [ -d /sys/class/net/$switch/bridge/. ]; then
   ip link set "$1" master "$switch"
   exit 0
else
   echo "Error: No bridge for guest interface found."
   exit 1
fi

Example commands to create a public bridge:

$ ifconfig eno1 0.0.0.0
$ brctl addbr br0
$ brctl addif br0 eno1
$ dhclient br0

KVM Miscellaneous Set up

Enable/Disable ACPI S3/S4 State

-global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 \

KVM vCPU Affinity

To achieve better real-time performance, isolated CPUs can be bound to a VM vCPU so that the QEMU vCPU thread will always run on the CPU. Enabling debug-threads=on on a VM name as described in Basic/Minimal Configuration of KVM helps to identify whether a QEMU thread is a vCPU thread or a working thread.

First, find the vCPU thread. Run the following command to show the corresponding PID of each vCPU:

$ sudo ps -p ${pid_of_qemu-system-x86_64} -T

Then, use taskset to change the affinity of vCPU threads to a physical CPU.

$ sudo taskset -cp ${vm_vcpu_affinity} ${vm_vcpu_pid}

KVM Recommended Configurations

The following command is recommended for booting a Windows VM with IDV GVT-d. It is recommended to login from another machine and execute the commands from a remote terminal since GVT-d will block any output from the host to the screen.

$ sudo qemu-system-x86_64 -nodefaults -serial mon:stdio \
-name ${vm_name},debug-threads=on \
-machine kernel_irqchip=on \
-enable-kvm \
-bios OVMF.fd \
-cpu host,hv_time \
-smp ${vcpu_num} -m ${mem_size} -mem-prealloc -mem-path /dev/hugepages \
-device vfio-pci,host=${pci_BDF_1} \
-device vfio-pci,host=${pci_BDF_2} \
-device vfio-pci,host=00:02.0,x-igd-gms=2,id=hostdev0,x-igd-opregion=on \
-object iothread,id=io1 \
-device virtio-blk-pci,drive=disk0,iothread=io1 \
-drive if=none,id=disk0,format=raw,aio=threads,file=${vm_image} \
-netdev tap,id=net0,ifname=tap0,script=${BRIDGE_SCRIPT} -device virtio-net-pci,netdev=net0 \
-global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1

Prepare Image for KVM

To create a Windows VM image on the KVM hypervisor, refer to Create Windows Image.

Build OVMF.fd for KVM

Do the following to build OVMF.fd.

1. Clone the EDK2 source from upstream and checkout to a specific release.

   $ git clone https://github.com/tianocore/edk2.git
   $ cd edk2
   $ git checkout edk2-stable202008
   

2. Apply the following patches on EDK2 using the git apply command.

   • 0001-OvmfPkg-Add-OpRegion-and-VBT-header-definition.patch

   • 0002-OvmfPkg-add-IgdAssignmentDxe.patch

   • 0003-OvmfPkg-add-Platform-GOP-Policy.patch

   • 0004-OvmfPkg-PlatformGopPolicy-Add-OpRegion-2.1-support.patch

   • 0005-OvmfPkg-PlatformPei-Reserve-IGD-Stolen-in-E820.patch

3. Build OVMF.fd:

   Refer to this document to build OVMF.fd.

   After the build is complete, OVMF.fd will be in edk2/Build/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd.