Attention

You are viewing an older version of the documentation. The latest version is v3.3.

KVM Hypervisor

Kernel-based Virtual Machine (KVM) is an open source virtualization technology for Linux* on x86 hardware containing virtualization extensions (for instance, Intel® Virtualization Technology (Intel® VT)).

KVM consists of loadable kernel modules, for example, kvm.ko that provides the core virtualization infrastructure. KVM turns Linux* into a hypervisor that allows host machine to run multiple, isolated virtual environments called virtual machine, running unmodified Linux* or Microsoft Windows* images.

Each virtual machine can have private virtual BIOS (basic input/output system) and private virtualized hardware: processor, memory, network cards, disk, graphics adapter, and so on.

KVM Terminology

Term	Description
VM	Hypervised operating systems (OS) are referred to as virtual machines or VMs.
GVT-d	Graphics Virtualization Technology - VT-d
IGD	Integrated Graphics Device
BDF	Bus:Device.Function notation used to succinctly describe PCI and PCIe devices.

KVM Prerequisites

The following section is applicable to:

KVM and relevant modules are pre-installed in Linux* kernel.

Do the following to set up the system:

1. Add intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=8 to the kernel command line. Make sure that the value of hugepagesz*hugepages does not exceed the size of the physical memory.

2. Setup the ECI APT repository.

3. Run the following command to install this component:

   $ sudo apt install qemu-system-x86
   

4. Disable graphical desktop:

   $ sudo systemctl set-default multi-user.target
   

Set up QEMU Command to Boot Windows VM

The following section is applicable to:

This section describes the procedure to start a Microsoft Windows* VM with KVM hypervisor on Intel® Edge Controls for Industrial (Intel® ECI or ECI).

Basic/Minimal Configuration of KVM

The following is a basic or minimal configuration of KVM:

$ sudo qemu-system-x86_64 -nodefaults -serial mon:stdio \
-name ${vm_name},debug-threads=on \
-machine kernel_irqchip=on \
-enable-kvm \
-bios OVMF.fd \
-cpu host,hv_time \
-smp ${vcpu_num} -m ${mem_size} -mem-prealloc -mem-path /dev/hugepages \

This configuration creates a VM named as ${vm_name} with ${vcpu_num} cores, ${mem_size} memory, full acceleration with KVM in-kernel irqchip support, empowered by hugepage memory. The -nodefaults does not specify a default device, so only one host PCI bridge and one ISA bridge is attached to this VM. debug-threads=on is useful when setting vCPU affinity. For details on setting the vCPU thread affinity, refer to KVM vCPU Affinity. To build OVMF.fd, which is a virtual UEFI binary file, refer to Build OVMF.fd for KVM.

Passthrough Generic PCI Device on KVM

To passthrough a PCI device via VT-d, make sure that the intel_iommu=on kernel command line parameter is enabled.

Device passthrough requires that the vfio-pci kernel module is loaded, and will bind to the target device. If vfio-pci is not a built-in kernel module, modprobe it after the kernel completes booting.

$ sudo modprobe vfio-pci vfio_iommu_type1

Use the following function to passthrough specific devices and all other devices in the same IOMMU group. The following example shows how to passthrough a USB controller fixed at 0000:00:14.0. You can determine the value ${pci_BDF} using the command lspci.

function vfio_bind()
{
   local bdf dev vendor device
   for bdf in "$@"; do
      echo "vfio bind: $bdf ..."
      for dev in $(ls /sys/bus/pci/devices/$bdf/iommu_group/devices); do
            vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
            device=$(cat /sys/bus/pci/devices/$dev/device)
            if [ -e /sys/bus/pci/devices/$dev/driver ]; then
               echo "    $dev : unbind driver ..."
               sudo sh -c "echo $dev > /sys/bus/pci/devices/$dev/driver/unbind"
            fi
            echo "    $dev : bind to vfio ..."
            sudo sh -c "echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id"
      done
   done
}

$ vfio_bind 0000:00:14.0

Now, the device can be assigned to QEMU. The following parameter will attach the host ${pci_BDF} device, for example the parameter attaches 0000:00:14.0 to the VM PCI bus. Multiple devices can be attached to a VM with different IDs.

-device vfio-pci,host=${pci_BDF}

If the vfio-pci kernel module is not needed for the target device, unbind it so that the VM can bind the original driver back. See the following example of a USB controller fixed at 0000:00:14.0:

$ usb_device=$(cat /sys/bus/pci/devices/0000\:00\:14.0/device)
$ sudo sh -c "echo 0000:00:14.0 > /sys/bus/pci/drivers/vfio-pci/unbind"
$ sudo sh -c "echo 8086 $usb_device >  /sys/bus/pci/drivers/vfio-pci/remove_id"
$ sudo sh -c "echo 0000:00:14.0 > /sys/bus/pci/drivers/xhci_hcd/bind"

Passthrough KVM Graphics Device

GVT-d is special case for Passthrough Generic PCI Device on KVM. For the integrated graphics device (IGD), the first step is to rebind the vfio-pci kernel module as a device driver. Usually, IGD is fixed at 0000:00:02.0.

$ vfio_bind 0000:00:02.0

Passthrough of the IGD is similar to that of a generic PCI device, except for some extra parameters to QEMU.

• The x-igd-gms property sets a value multiplied by 32 as the amount of pre-allocated memory (in units of MB) to support IGD in VGA modes.

• The x-igd-opregion property exposes opregion (VBT included) to guest driver so that the guest driver could parse display connector information from. This property mandatory for the Microsoft Windows* VM to enable display output.

The following command shows the IGD device parameter for QEMU:

-device vfio-pci,host=00:02.0,x-igd-gms=2,id=hostdev0,x-igd-opregion=on \

Set up KVM Storage Device

It is recommended to use iothread and virtio-blk drive (Red Hat Enterprise Linux* VirtIO SCSI Disk) since the disk read and write performance is much better than using default QEMU IDE drive.

-object iothread,id=io1 \
-device virtio-blk-pci,drive=disk0,iothread=io1 \
-drive if=none,id=disk0,format=raw,aio=threads,file=${vm_image} \

${vm_image} is the Microsoft Windows* image file. Follow this section to create a new Microsoft Windows* VM image.

Set up KVM Network Device

An Ethernet or wireless adapter can be assigned to a VM either with emulated Ethernet and network backend or via a generic PCI device passthrough. Refer to Passthrough Generic PCI Device on KVM for exclusive assignment. Attach a virtio Ethernet adapter using TAP network backend with id tap0 and configure the ${BRIDGE_SCRIPT} script .

-netdev tap,id=net0,ifname=tap0,script=${BRIDGE_SCRIPT} -device virtio-net-pci,netdev=net0 \

The following is an example of ${BRIDGE_SCRIPT}, which can support either public bridge or private virtual bridge by setting the switch to the target bridge name.

#!/bin/sh
set -x
switch=br0
ip link set "$1" up
if [ -d /sys/class/net/$switch/bridge/. ]; then
   ip link set "$1" master "$switch"
   exit 0
else
   echo "Error: No bridge for guest interface found."
   exit 1
fi

Example commands to create a public bridge:

$ ifconfig eno1 0.0.0.0
$ brctl addbr br0
$ brctl addif br0 eno1
$ dhclient br0

KVM Miscellaneous Set up

Enable/Disable ACPI S3/S4 State

-global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 \

KVM vCPU Affinity

To achieve better real-time performance, isolated CPUs can be bound to a VM vCPU so that the QEMU vCPU thread will always run on the CPU. Enabling debug-threads=on on a VM name as described in Basic/Minimal Configuration of KVM helps to identify whether a QEMU thread is a vCPU thread or a working thread.

First, find the vCPU thread. Run the following command to show the corresponding PID of each vCPU:

$ sudo ps -p ${pid_of_qemu-system-x86_64} -T

Then, use taskset to change the affinity of vCPU threads to a physical CPU.

$ sudo taskset -cp ${vm_vcpu_affinity} ${vm_vcpu_pid}

KVM Recommended Configurations

The following command is recommended for booting a Microsoft Windows* VM with IDV GVT-d. It is recommended to login from another machine and execute the commands from a remote terminal since GVT-d will block any output from the host to the screen.

$ sudo qemu-system-x86_64 -nodefaults -serial mon:stdio \
-name ${vm_name},debug-threads=on \
-machine kernel_irqchip=on \
-enable-kvm \
-bios OVMF.fd \
-cpu host,hv_time \
-smp ${vcpu_num} -m ${mem_size} -mem-prealloc -mem-path /dev/hugepages \
-device vfio-pci,host=${pci_BDF_1} \
-device vfio-pci,host=${pci_BDF_2} \
-device vfio-pci,host=00:02.0,x-igd-gms=2,id=hostdev0,x-igd-opregion=on \
-object iothread,id=io1 \
-device virtio-blk-pci,drive=disk0,iothread=io1 \
-drive if=none,id=disk0,format=raw,aio=threads,file=${vm_image} \
-netdev tap,id=net0,ifname=tap0,script=${BRIDGE_SCRIPT} -device virtio-net-pci,netdev=net0 \
-global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1

Prepare Image for KVM

To create a Microsoft Windows* VM image on the KVM hypervisor, refer to Create Windows Image.

Build OVMF.fd for KVM

Do the following to build OVMF.fd.

1. Clone the EDK2 source from upstream and checkout to a specific release.

   $ git clone https://github.com/tianocore/edk2.git
   $ cd edk2
   $ git checkout edk2-stable202008
   

2. Apply the following patches on EDK2 using the git apply command.

   • 0001-OvmfPkg-Add-OpRegion-and-VBT-header-definition.patch

   • 0002-OvmfPkg-add-IgdAssignmentDxe.patch

   • 0003-OvmfPkg-add-Platform-GOP-Policy.patch

   • 0004-OvmfPkg-PlatformGopPolicy-Add-OpRegion-2.1-support.patch

   • 0005-OvmfPkg-PlatformPei-Reserve-IGD-Stolen-in-E820.patch

3. Build OVMF.fd:

   Refer to this document to build OVMF.fd.

   After the build is complete, OVMF.fd will be in edk2/Build/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd.