Attention

You are viewing an older version of the documentation. The latest version is v3.3.

Real-Time Systems Hypervisor (RTH)¶

Real-Time Systems Hypervisor* (RTH) is a Type I (bare-metal) Hypervisor commercially available for 10+ years to meet the most stringent Industrial real-time performance and Workload consolidated usecase across all Intel processors families (Atom, Core, Xeon).

When focusing on Determinism the RTH Privileged-Mode allow 0-us added latencies and a tight resource partitioning over a wide range of Real-time operating systems supports :

Direct Hardware Access

Use standard drivers

Secure separation

Temporal Isolation of Workloads

When focusing on Desktop rendering & composition UX the RTH Virtualization-mode brings Simple and flexible GFx passthrough over a wide range of Desktop operating systems supports:

Not tied to any OS or Hardware

No customization required

Easy installation

The Intel Edge Controls for Industrial on Real-Time Systems Hypervisor (ECI-R) responds Industry-graded stackholder needs for Automation on high-density PLC, consolidate multi-axis robot controller,…

RTH Terminology¶

RTH: Real-Time Systems Hypervisor refers to as Type I (bare-metal) rthx86 Hypervisor runtime, “RTH” for short.
GPOS: Hypervised operating systems (OS) are referred to as Virtualized-Mode operating systems (OS), or “GPOS” for short. This is equivalent to “VM” or “Guest OS” in other hypervisor terminology.
POS: Since RTH is a Type 1 hypervisor, the OS runtime in which the RTH allows “Direct Hardware Access” is referred to as Privileged-Mode operating systems (OS) using the terms “POS” for short.

RTH Prerequisites¶

RTH supported a very large set of Intel platforms across processors families (Atom, Core, Xeon). The full compatibility list can be found :

https://www.real-time-systems.com/

https://www.intel.com/content/www/us/en/developer/articles/technical/real-time-systems-and-intel-take-industrial-embedded-systems-to-the-next-level.html

Important

To gain access to Real-Time Systems Hypervisor (RTH) 2-months renewable Evaluation RTH-Release_R5.4.01.xxxxx.zip for [INTEL EMPLOYEES ONLY] License please contact matthias.hahn@intel.com or for [INTEL PARTNERS/CUSTOMERS] License please contact info@real-time-systems.com

Follow section rts-buster to build an ECI target with the Real-Time Systems Hypervisor (RTH).

Tip

The RTH hypervisor can be used to run ECI-B and ECI-X as Privileged-Mode operating systems (OS) respectively called POS-RT and POS-XE. rts-buster Target allows embedding multiple POS as Debian package pre-installed onto target image.
Follow section Installing ECI-B/R/X to create a bootable USB flash drive with rts-buster image and install to a target system.
Follow section Configure Target System BIOS to configure the target system BIOS correctly for use with RTH.
For ECI technical support : FAQ about latest RTH release 5.4.01.27752 support, Virtualized or Privileged-mode Linux BSP updates or patches, or advice and guidance please contact your local ECI support engineer.

Using RTH to Virtualize your Desktop Environment (GPOS)¶

The following section is applicable to:

Edge Controls For Industrial (ECI) enables GPU-rendering of Debian GNU/Linux LXDE desktop environment out-of-the-box experience. It can be incrementally replace by user-preferred Ubuntu, Fedora or Microsoft Windows installation.

RTH Virtualized-Mode mode incurs a performance overhead due to Hypervisor runtime background execution still CPU-cycles (ie. vmm entry/exit IA instruction) to GPOS. However, it allows Privileged-OS runtimes to be isolated from the GPOS, while each are running with different hw-resources (CPU cores, PCI devices , Interrupt,…) In general, hw-resources not allocated to the Privileged-RTOS are given to the Virtualized-GPOS, except GPU/Graphics resource ONLY accessible by Virtualized-GPOS :

RTH Virtual Ethernet Network - Virtual Network controller (veth) driver is emulated by the Real-Time Hypervisor.
This driver allows the RTOS and Ubuntu operating systems to communicate over standard Ethernet network protocols (UPD, TCP, ICMP,…) shall be installed manually on Virtualized operating systems runtimes.
Standard Ethernet Network Driver – Standard TCP/IP networking allow runtimes access from another machine to the GPOS (Ubuntu).
It is recommended to configure port-forwarding 8080, 1217, 4840, 11740, and 9022 (i.e. iptables) GPOS with to allow remote connection to various ECI RT-OS runtimes (HTTP, CODESYS runtime/HMI/OPC UA server, SSH, …)

The following section is applicable to:

Default Debian GNU/Linux LXDE Desktop GPOS¶

Out-of-box installed rts-buster image make generic assumptions about the desired Debian GNU/Linux LXDE desktop default Grub menu.

Select a desired RTH pre-canned GPOS + POS configurations Advanced options for ECI-R (...) Linux Intel 5.4-rt/lts ...:

Select subsequent grub multiboot configuration ECI-R (RTS Hypervisor) GPOS GNU/Debian Desktop and POS

Successfully RTH boot log are displayed

Successfully GPOS kernel boot log are displayed

Enter Login in LXDE Lightdm UI menu eci-user (passwd=eci-user) or as root

Select UI various applications for example htop and glxgears

use Debian package update¶

rts-buster provide the build environment for ECI-R to create and maintain a consistent set of Real-Time Systems Hypervisor* (RTH) Debian packages and resolve runtime dependencies.

Note

Modification to the ECI-R Grub template file /etc/grub.d/42-rth-pos-rt-linux, /etc/grub.d/43-rth-pos-xe-linux, /etc/grub.d/44-rth-pos-qn-linux and /etc/grub.d/45-rth-pos-vx-linux may be necessary for customizing Grub entries.

Incremental Microsoft® Windows® Desktop GPOS installation¶

Section	Description
Microsoft® Windows® Desktop RTH virtualized-mode GPOS Preconditions for RTH Windows® Edit RTH Windows® settings manually Install RTH Windows® Virtualized-mode BSP	Microsoft Windows installation and virtualized-mode RTH configuration for OS/0

Replacing GPOS with user-preferred Canonical® Ubuntu® Desktop¶

Section	Description
Canonical® Ubuntu® Desktop RTH virtualized-mode GPOS Instantiate the Ubuntu® Desktop GPOS alternative Install RTH Linux Virtualized-mode BSP	Ubuntu Desktop installation and virtualized-mode RTH configuration for OS/0

Using RTH to consolidate Privileged RTOS (POS)¶

Edge Controls For Industrial (ECI) provides an out-of-the-box experience Debian GNU/Linux PLCopen-ready Control environment.

Under RTH Privileged-Mode the Linux Real-Time runtime to have direct hardware access and without adding extra latency caused by Hypervisor runtime (ie. no vmm entry/exit IA instruction)

rts-buster image make generic assumptions about the desired POS PREEMPT-RT (pos-rt) or Xenomai/Cobalt 3.1 (pos-xe) Linux OS. Modification to the ECI-R templates /boot/rth/pos-rt/Linux_Linux64_<nvme0n1pY|sdY>.txt or /boot/rth/pos-xe/Linux_Linux64_<nvme0n1pY|sdY>.txt may be necessary for customizing the POS.

Linux PREEMPT-RT Linux kernel with patch allows for a fully preemptible kernel.
In ECI privileged-mode default configuration, the Real-Time OS (RTOS) 5.4-rt runtime is mounted in RAM as ramdisk (ie. both kernel and rootfs) to improve execution latency performance and software resiliency. For more information see https://rt.wiki.kernel.org/index.php/Main_Page (Note: the wiki is in the process of a migration to https://wiki.linuxfoundation.org/realtime/start )
Xenomai/Cobalt 3.1 Linux kernel with ipipe-x86 patch allowing for a fully preemptible and low-latency IRQ co-kernel approach
In ECI privileged-mode user configuration, the Xenomai/Ipipe-x86 Real-Time extension to Linux 5.4-rt runtime is mounted in RAM as ramdisk (ie. both kernel and rootfs) to improve execution latency performance and software resiliency.
PLCopen IEC 611131-3 Linux x86_64 control runtime system – Allows any IA platform to operate as an IEC 61131-3-compliant industrial controller.
In ECI privileged-mode default configuration, the PLC control runtime is built in to the Linux Real-Time OS (RTOS) runtime.
Time Sensitive Networking (TSN) Reference Software for Linux on Intel Ethernet Controller supporting IEEE 802.1AS, IEEE 802.1Q-2018 (Qbv, Qbu,…) hardware offloading.
In ECI privileged-mode default configuration, RTH assigned i225-LM i210-IT discrete or integrated TSN GbE devices to Linux Real-Time OS (RTOS) privileged OS runtime to enabled TSN endpoint hardware capability.
RTH Virtual Ethernet Network - Virtual Network controller (veth) driver is emulated by the Real-Time Hypervisor.
In ECI privileged-mode default configuration, The virtual-Ethernet controller (vnet.ko) Linux kernel module is enabled automatically at boot in privileged Linux Real-Time OS (RT-OS) runtime and shall be installed manually on Virtualized operating systems runtimes to communicate over standard Ethernet network protocols (UPD, TCP, ICMP,…) between both runtime OSes.

The following section is applicable to:

Upgrading POS images¶

rts-buster provide the build environment for ECI-R to create and maintain pos-xe or/and pos-rt or/and pos-vx or/and pos-qn Debian packaging and resolve runtime dependencies.

use Debian pos-rt and pos-xe package update¶

For example, the ECI-R created pos-xe Debian package contains grub configuration entries and RTH multiboot images files:

$ dpkg -c rth-pos-xe_5.4.01.27752_amd64.deb
drwxr-xr-x root/root         0 2021-09-17 17:09 ./
drwxr-xr-x root/root         0 2021-09-17 17:09 ./boot/
drwxr-xr-x root/root         0 2021-09-17 17:09 ./boot/rth/
drwxr-xr-x root/root         0 2021-09-17 17:09 ./boot/rth/pos-xe/
-rw-r--r-- root/root 504138773 2021-09-17 17:09 ./boot/rth/pos-xe/initrd-mbLinux64.gz
-rw-r--r-- root/root   9724456 2021-09-17 17:09 ./boot/rth/pos-xe/mbLinuz
drwxr-xr-x root/root         0 2021-09-17 17:09 ./etc/
drwxr-xr-x root/root         0 2021-09-17 17:09 ./etc/grub.d/
-rwxr-xr-x root/root      1311 2021-09-17 17:09 ./etc/grub.d/43-rth-pos-xe-linux
drwxr-xr-x root/root         0 2021-09-17 17:09 ./root/
-rwxr-xr-x root/root        29 2021-09-17 17:09 ./root/hello-rth-pos-xe.sh
drwxr-xr-x root/root         0 2021-09-17 17:09 ./usr/
drwxr-xr-x root/root         0 2021-09-17 17:09 ./usr/share/
drwxr-xr-x root/root         0 2021-09-17 17:09 ./usr/share/doc/
drwxr-xr-x root/root         0 2021-09-17 17:09 ./usr/share/doc/rth-pos-xe/
-rw-r--r-- root/root       128 2021-09-17 14:46 ./usr/share/doc/rth-pos-xe/changelog.gz

      
    

When Default Debian GNU/Linux LXDE Desktop GPOS is used upgrading Xenomai/Cobalt 3.1 Linux (e.g ECI-X) RTH privileged-mode pos-xe images and grub menu can be done as followed :

$ dpkg -i rth-pos-xe_5.4.01.27752_amd64.deb
(Reading database ... 51507 files and directories currently installed.)
Preparing to unpack rth-pos-xe_5.4.01.27752_amd64.deb ...
Unpacking rth-pos-xe (5.4.01.27752) over (5.4.01.27752) ...
Setting up rth-pos-xe (5.4.01.27752) ...

Configuration file '/etc/grub.d/43-rth-pos-xe-linux'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** 43-rth-pos-xe-linux (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /etc/grub.d/43-rth-pos-xe-linux ...
Start updating pos-xe RTS Hypervisor configuration files
/dev/sda5: LABEL="pos-xe" UUID="53fa36d5-7933-478c-931d-6a34aee4d803" TYPE="ext4" PARTLABEL="primary" PARTUUID="e7a5e19f-f15a-4035-a161-be8e0d6cf450"
Found pos-xe ext4 partition /dev/sda5...
Extracting system.sfs from /boot/rth/pos-xe/initrd-mbLinux64.gz...
996963 blocks
5842 blocks
 pos-xe/system.sfs to partition /dev/sda5. Done.
Start updating pos-xe RTS Hypervisor configuration files
/dev/sda2: LABEL="/" UUID="d29a4ea9-f36c-48ba-82f3-dd4462eef540" TYPE="ext4" PARTLABEL="root" PARTUUID="2447b400-d280-4880-a469-60e4ed5de67c"
/dev/sda5: LABEL="pos-xe" UUID="53fa36d5-7933-478c-931d-6a34aee4d803" TYPE="ext4" PARTLABEL="primary" PARTUUID="e7a5e19f-f15a-4035-a161-be8e0d6cf450"
Grub config  switch from RAMFS to OVERLAYFS. Done.
update-grub config on RTH GPOS. done.
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.115-intel-pk-standard+
Found initrd image: /boot/initrd.img-5.4.115-intel-pk-standard+
done
Grub entries updates. Done. (ONLY visible on the next RTH system REBOOT/SHUTDOWN (e.g. sudo rth -sysreboot or sudo rth -sysshutdown)
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.115-intel-pk-standard+
Found initrd image: /boot/initrd.img-5.4.115-intel-pk-standard+
done

      
    

Important

To be able to upgrade both pos-xe or/and pos-rt image that have mount POS overlayFS rootfs partition, it may be necessary to reboot rth -sysreboot && reboot and select non-Virtualized Debian GNU/Linux grub configuration.

Run ECI update script manually¶

For rts-poky Debian package installation, the ECI-R provides convenience scripts at directory /usr/libexec/rth-pos-updates/ which simplify the process of upgrading POS images under Default Debian GNU/Linux LXDE Desktop GPOS console. These scripts and their usage are described in the table below:

Note

These scripts make generic assumptions about the desired POS configuration. Modification to a script may be necessary to customize the POS. See section Change POS rootfs mount assignment for more information.

Launch Script Name

Use Case

Default image location

rth-pos-overlayfs-setup.sh <pos-xe|pos-rt>

This script install a Privileged-Mode OS rootfs . It is best suited for ECI targets ECI-B and ECI-X.

Note

pos-rt and pos-xe are dedicated partitions namespace to mount POS rootfs as overlayfs

/boot/rth/pos-rt/initrd-mbLinux64.gz /boot/rth/pos-xe/initrd-mbLinux64.gz

rth-grub-setup.sh <pos-xe|pos-rt|pos-vx|pos-vx>

This script launches grub updates and RTH config. It is best suited Privileged-OS targets ECI-B and ECI-X

/etc/grub.d/42-rth-pos-rt-linux /etc/grub.d/43-rth-pos-xe-linux /etc/grub.d/44-rth-pos-qn-linux /etc/grub.d/45-rth-pos-vx-linux

Augment POS runtime memory size¶

The following section is applicable to:

rts-buster image make generic assumptions about the desired POS Memory setting. Modification to the ECI-R templates /boot/rth/Linux_Linux64.txt or /boot/rth/WindowsEFI_Linux64.txt may be necessary for customizing the POS on [/SYSTEM] "IOMMU" = uint32: 1, [/OS/1] "virtual_MMU" = uint32: 1 and "memory_size" = uint64: 0x80000000 RTH Init parameters for different device memory protection, physical memory size and relocation disabled.

[/SYSTEM]
     "IOMMU" = uint32: 1 # Activate IOMMU for all OSs
...
[/OS/1]

    "name"          = "Privileged-RTOS"
    "boot_priority" = uint32: 1
    "memory_size"   = uint64: 0x80000000      # 2048 MB
    "CPU"           = bytelist: 3,4           # e.g. two CPUs: bytelist: 3, 4
    "virtual_MMU"             = uint32: 1
    "restricted_IO"           = uint32: 1
...

      
    

Note

The RTH activate IOMMU for all OSs may be impossible due to a BIOS setting VT-d = Disabled by default. Please make sure VT-d = Enabled on the machine.

$ readtrace -o 1
...
Init IOMMU
IOMMU configured but not present or disabled in BIOS.
Support for device memory protection and physical memory relocation disabled.
...

      
    

Change POS rootfs mount assignment¶

At installation step rts-buster image make generic assumptions about the desired NVME or SATA disk drive partition assignment considering pos-rt and pos-xe namespaces. Modification to the ECI-R templates /boot/rth/pos-rt/Linux_Linux64_<nvme0n1pY|sdY>.txt or /boot/rth/pos-rt/WindowsEFI_Linux64_<nvme0n1pY|sdY>.txt may be necessary for customizing the POS on NVME or SATA disk drive partition assignment

Important

NVMe Drive assignment feature is not supported if your hardware configures NVMe controller bar above 4GB. Sometimes server grade hardware does not fulfill this requirement in its default configuration. Typically this behavior is configurable within BIOS settings, e.g. under:

[on commercial UEFI BIOS vendor] under Advanced -> PCI configuration -> Memory Mapped I/O above 4GB [Disabled]
[on TGL UP3 RVP Intel UEFI BIOS] under Intel Advanced Menu -> System Agent Configruation -> Above 4G -> [disabled]

eMMC Integrated controller partition assignment is NOT supported by RTH.

$ readtrace -o 1
...
NVMe (1/0/0) (Vendor:0x8086 Device:0xF1A5) - no drive sharing configured
...

Following configuration steps we will assume OT admin is running a POS installation on NVME disk created distinct EXT4 dedicated partition respectively. GPOS rootfs / tree referred with /dev/nvme0n1p4 and /dev/nvme0n1p3 basename :

$ blkid -o list
device                    fs_type         label            mount point        UUID
------------------------------------------------------------------------------------------------------------------
/dev/nvme0n1p1            vfat                             /boot/efi          D473-9F49
/dev/nvme0n1p2            ext4            /                /                  ae1a66a5-ee9c-4573-9249-dd207ffa6248
/dev/nvme0n1p4            ext4            pos-xe           (not mounted)      b3c0531b-2cec-47da-ab21-6a0a928e55c9
/dev/nvme0n1p5            ext4            pos-rt           (not mounted)      a4b56194-8688-4b73-b8e1-58f6bc88b72b
/dev/nvme0n1                                               (in use)
/dev/nvme0n1p3                                             (not mounted)

      
    

or their corresponding Disk :

$ ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 13 Sep  7 14:58 pci-0000:01:00.0-nvme-1 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Sep  7 14:58 pci-0000:01:00.0-nvme-1-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Sep  7 14:58 pci-0000:01:00.0-nvme-1-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 Sep  7 14:58 pci-0000:01:00.0-nvme-1-part3 -> ../../nvme0n1p3
lrwxrwxrwx 1 root root 15 Sep  7 14:58 pci-0000:01:00.0-nvme-1-part4 -> ../../nvme0n1p4
lrwxrwxrwx 1 root root 15 Sep  7 14:58 pci-0000:01:00.0-nvme-1-part5 -> ../../nvme0n1p5

      
    

A first step often needed to assign all NVME disk partitions to OS/0, then gather the required information by generating a readtrace hypervisor logs :

$ cp /boot/rth/Linux_Linux64.txt /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
[/DRIVE/0]

"default"   = uint32: 0   # default is OS/0
"bus"       = uint32: 1   # 1 0 0   0x8086 0xF1A5 NVMe controller
"device"    = uint32: 0   #
"function"  = uint32: 0   #
...
$ rth -sysreboot && reboot

      
    

$ readtrace -o 0
...
NVMe (1/0/0) (Vendor:0x8086 Device:0xF1A5) - Write Cache: enabled
    Drive assignment for namespace 1 (GPT):
      Partition | OS | Start Sector |   End Sector | Type
              0 |  0 |         2048 |        38911 | 0xEF EFI System
              1 |  0 |        59392 |    152413430 | 0x83 Linux filesystem
              2 |  0 |    152414208 |    230539263 | 0x07 Microsoft Basic Data
              3 |  1 |    234444800 |    238350335 | 0x83 Linux filesystem
              4 |  0 |    238350336 |    246163455 | 0x83 Linux filesystem
...

      
    

As The second step user shall decide among two possible to mount and boot POS rootfs

mount POS overlayFS rootfs partition¶

This approach uses overlayfs kernel capability to allow a minimalist RTH initramfs that mount a system.sfs located [/DRIVE/0/PARTITION/3] as POS [/OS/1] diskroot filesystems diskroot=/dev/hda4 allowing Linux RAM initialization with fully system RAM size.

Important

The privileged-mode kernel is built with CONFIG_OVERLAYFS=y for the ECI-R initramfs to create privileged-mode OS root filesystems (ie rootfs) as single tree.

$ vi /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
[/OS/1]

    "name"          = "Privileged RTOS"
    "boot_priority" = uint32: 2             # start second
    "memory_size"   = uint64: 0x80000000    # 2000 MB
    "virtual_MMU"   = uint32: 1             # set to 1 to restrict memory access
    "restricted_IO" = uint32: 1             # set to 1 to restrict I/O access
    "CPU"           = bytelist: 3,4           # e.g. two CPUs: bytelist: 3, 4

    "virtual_COM-to-log_port" = uint32 : 0x3F8
    "trace_partition_number" = uint32: 1    # write log to /SHM/1

    [/OS/1/RUNTIME/0]

        "bootline" = "diskroot=/dev/hda4 ip=192.168.2.2:::::vnet0  debug=all verbose=all random.trust_cpu=on console=tty0 console=ttyS0,115200n8"
        "image_0"  = "mbLinuz64"
        "image_1"  = "initrd-mbLinux64.gz"
...
[/DRIVE/0]

"default"   = uint32: 0   # default is OS/0
"bus"       = uint32: 1   # 1 0 0   0x8086 0xF1A5 NVMe controller
"device"    = uint32: 0   #
"function"  = uint32: 0   #

    [/DRIVE/0/PARTITION/3]
    "OS" = uint32: 1   # assigned to OS/1
...

      
    

Note

At installation step rts-buster image make generic assumptions about the desired NVME or SATA disk drive partition assignment considering pos-rt and pos-xe namespaces. Modification to the ECI-R templates POS rootfs system.sfs or the predefined pos-rt EXT4 partition (for example /dev/nvme0n1p4) or manual update from /boot/rth/pos-rt/initrd-mbLinux64.gz to /boot/rth/pos-rt/initrd-mbLinux64-no-sfs.gz (approximately 850 KB in size) should proceed with the following steps :

$ cd /boot/rth/pos-rt/
$ gunzip initrd-mbLinux64.gz
$ mkdir initramfs-tmp
$ cd initramfs-tmp/
$ cpio -i -d -H newc --no-absolute-filenames < ../initrd-mbLinux64
$ cd ..
$ diskroot=nvme0n1p4
$ mkdir -p /mnt/tmp-`basename $diskroot`
$ mount /dev/$diskroot /mnt/tmp-`basename $diskroot`
$ dd if=initramfs-tmp/system.sfs of=/mnt/tmp-`basename $diskroot`/system.sfs  status=none
$ rm initramfs-tmp/system.sfs
$ gzip initrd-mbLinux64
$ cd initramfs-tmp/
$ find . | cpio -H newc -o > ../initrd-mbLinux64-no-sfs
$ cd ..
$ cat initrd-mbLinux64-no-sfs | gzip > initrd-mbLinux64-no-sfs.gz

      
    

Modification to the ECI-R templates /etc/grub.d/42-rth-pos-rt-linux or /etc/grub.d/43-rth-pos-xe-linux may be necessary for customizing GRUB multiboot entries (e.g. module2) naming convention .e.g initrd-mbLinux64-no-sfs.gz , Linux_Linux64_<diskroot>.txt and mbLinuz

$ vi /etc/grub.d/42-rth-pos-rt-linux
...
menuentry "ECI-R (RTS Hypervisor) GPOS GNU/Debian Desktop and POS Preempt-RT Linux" {
    search --no-floppy --fs-uuid --set=root ae1a66a5-ee9c-4573-9249-dd207ffa6248
    multiboot2 /boot/rth/rthx86
    module2 /boot/rth/license.txt
    module2 /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
    echo "Loading POS multi-boot Linux Preempt-RT bootable [OVERLAYFS] images..."
    module2 /boot/rth/pos-rt/mbLinuz mbLinuz64
    module2 /boot/rth/pos-rt/initrd-mbLinux64-no-sfs.gz initrd-mbLinux64.gz
    echo "Loading [ROOTFS] GPOS GNU/Debian Desktop bootable images ..."
    module2 /vmlinuz vmlinuz.lnk
    module2 /initrd.img initramfs.lnk
    ...
$ update-grub
$ rth -sysreboot && reboot

      
    

mount POS flattened rootfs partition¶

This approach does NOT require any RTH initramfs (ie. no “image_1” in RTH config) but a flattened rootfs contents extracted onto a dedicated disk GPT partition and mounted as the root=/dev/nvme0n1p4 partition to mount POS flat rootfs for [/OS/1/RUNTIME/0].

$ vi /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
[/OS/1]

    "name"          = "Privileged RTOS"
    "boot_priority" = uint32: 2             # start second
    "memory_size"   = uint64: 0x80000000    # 2000 MB
    "virtual_MMU"   = uint32: 1             # set to 1 to restrict memory access
    "restricted_IO" = uint32: 1             # set to 1 to restrict I/O access
    "CPU"           = bytelist: 3,4           # e.g. two CPUs: bytelist: 3, 4

    "virtual_COM-to-log_port" = uint32 : 0x3F8
    "trace_partition_number" = uint32: 1    # write log to /SHM/1

    [/OS/1/RUNTIME/0]

        "bootline" = "root=/dev/nvme0n1p4 ip=192.168.2.2:::::vnet0  debug=all verbose=all random.trust_cpu=on console=tty0 console=ttyS0,115200n8"
        "image_0"  = "mbLinuz64"
...
[/DRIVE/0]

"default"   = uint32: 0   # default is OS/0
"bus"       = uint32: 1   # 1 0 0   0x8086 0xF1A5 NVMe controller
"device"    = uint32: 0   #
"function"  = uint32: 0   #

    [/DRIVE/0/PARTITION/3]
    "OS" = uint32: 1   # assigned to OS/1
...

      
    

Note

At installation step rts-buster image make generic assumptions about the desired NVME or SATA disk drive partition assignment considering pos-rt and pos-xe namespaces. Modification to the ECI-R templates POS rootfs system.sfs or the predefined pos-rt and pos-xe LABEL partition or manual create a minimal mknod should proceed with the following steps :

$ cd /boot/rth/
$ gunzip initrd-mbLinux64.gz
$ mkdir initramfs-tmp
$ cd initramfs-tmp/
$ cpio -i -d -H newc --no-absolute-filenames < ../initrd-mbLinux64
$ cd ..
$ gzip initrd-mbLinux64
$ diskroot=nvme0n1p4
$ mkdir -p /mnt/tmp-`basename $diskroot`
$ mount /dev/$diskroot /mnt/tmp-`basename $diskroot`
$ unsquashfs -d /mnt/tmp-`basename $diskroot` -f initramfs-tmp/system.sfs
$ cd /mnt/tmp-`basename $diskroot`/dev/
$ mknod null    c 1 3
$ mknod zero    c 1 5
$ mknod random  c 1 8
$ mknod urandom c 1 9
$ mknod tty     c 5 0
$ mknod console c 5 1
$ mknod ptmx    c 5 2
$ mkdir pts
$ mknod ttyS0   c 4 64
$ umount /mnt/tmp-`basename $diskroot`
$ rm -rf initramfs-tmp

      
    

Modification to the ECI-R templates /etc/grub.d/42-rth-pos-rt-linux or /etc/grub.d/43-rth-pos-xe-linux may be necessary for customizing GRUB multiboot entries (e.g. module2) naming convention .e.g initrd-mbLinux64-no-sfs.gz , Linux_Linux64_<diskroot>.txt and mbLinuz

$ vi /etc/grub.d/42-rth-pos-rt-linux
...
menuentry "ECI-R (RTS Hypervisor) GPOS GNU/Debian Desktop and POS Preempt-RT Linux" {
    search --no-floppy --fs-uuid --set=root ae1a66a5-ee9c-4573-9249-dd207ffa6248
    multiboot2 /boot/rth/rthx86
    module2 /boot/rth/license.txt
    module2 /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
    echo "Loading POS multi-boot Linux Preempt-RT bootable [FLATROOTFS] images..."
    module2 /boot/rth/pos-rt/mbLinuz mbLinuz64
    echo "Loading [ROOTFS] GPOS GNU/Debian Desktop bootable images ..."
    module2 /vmlinuz vmlinuz.lnk
    module2 /initrd.img initramfs.lnk
    ...
$ update-grub
$ rth -sysreboot && reboot

      
    

mount POS RAM disk rootfs (no partition mapped)¶

This approach does NOT require any dedicated disk GPT partition, mounted as the root=/dev/ram to mount POS system.sfs rootfs for [/OS/1/RUNTIME/0].

$ vi /etc/grub.d/42-rth-pos-rt-linux
...
menuentry "ECI-R (RTS Hypervisor) GPOS GNU/Debian Desktop and POS Preempt-RT Linux" {
    search --no-floppy --fs-uuid --set=root ae1a66a5-ee9c-4573-9249-dd207ffa6248
    multiboot2 /boot/rth/rthx86
    module2 /boot/rth/license.txt
    module2 /boot/rth/Linux_Linux64.txt
    echo "Loading POS multi-boot Linux Preempt-RT bootable [RAMFS] images..."
    module2 /boot/rth/pos-rt/mbLinuz mbLinuz64
    module2 /boot/rth/pos-rt/initrd-mbLinux64.gz initrd-mbLinux64.gz
    echo "Loading [ROOTFS] GPOS GNU/Debian Desktop bootable images ..."
    module2 /vmlinuz vmlinuz.lnk
    module2 /initrd.img initramfs.lnk
    ...
$ update-grub
$ rth -sysreboot && reboot

      
    

$ vi /boot/rth/Linux_Linux64.txt
...
[/OS/1]

    "name"          = "Privileged RTOS"
    "boot_priority" = uint32: 2             # start second
    "memory_size"   = uint64: 0x80000000    # 2000 MB
    "virtual_MMU"   = uint32: 1             # set to 1 to restrict memory access
    "restricted_IO" = uint32: 1             # set to 1 to restrict I/O access
    "CPU"           = bytelist: 3,4           # e.g. two CPUs: bytelist: 3, 4

    "virtual_COM-to-log_port" = uint32 : 0x3F8
    "trace_partition_number" = uint32: 1    # write log to /SHM/1

    [/OS/1/RUNTIME/0]

        "bootline" = "diskroot=/dev/ram ip=192.168.2.2:::::vnet0  debug=all verbose=all random.trust_cpu=on console=tty0 console=ttyS0,115200n8"
        "image_0"  = "mbLinuz64"
        "image_1"  = "initrd-mbLinux64.gz"
...

      
    

For RTH configuration file /boot/rth/Linux_Linux.txt grub multiboot load the initrd-mbLinux64.gz image to mount POS as ramdisk ONLY in contiguous DRAM space below 4 GB memory space.

For example, Intel(R) Core(TM) i5-6500TE system has 2 GB RAM space available between 0x000000100000 - 0x000077DD0FFF allowing to mount "image_1" = "initrd-mbLinux64.gz" (md5sum 1B8C8B8059DC02C8E1EEDB8766602A1FA5A4510F) rootfs prior /OS/1 runtime to boot.

$ readtrace -o 1
...
Init memory
   start          - end            (size          ) type
 0 0x000000000000 - 0x000000057FFF (0x000000058000) 0x01
 1 0x000000058000 - 0x000000058FFF (0x000000001000) 0x02
 2 0x000000059000 - 0x00000009EFFF (0x000000046000) 0x01
 3 0x00000009F000 - 0x0000000FFFFF (0x000000061000) 0x02
 4 0x000000100000 - 0x000077DD0FFF (0x000077CD1000) 0x01
 5 0x000077DD1000 - 0x000077DD1FFF (0x000000001000) 0x04
 6 0x000077DD2000 - 0x000077DD2FFF (0x000000001000) 0x02
 7 0x000077DD3000 - 0x00007E44BFFF (0x000006679000) 0x01
 8 0x00007E44C000 - 0x00007E78AFFF (0x00000033F000) 0x02
 9 0x00007E78B000 - 0x00007E8FAFFF (0x000000170000) 0x01
10 0x00007E8FB000 - 0x00007EFE5FFF (0x0000006EB000) 0x04
11 0x00007EFE6000 - 0x00007F3A6FFF (0x0000003C1000) 0x02
12 0x00007F3A7000 - 0x00007F3FEFFF (0x000000058000) 0x14
13 0x00007F3FF000 - 0x00007F3FFFFF (0x000000001000) 0x01
14 0x00007F400000 - 0x00008FFFFFFF (0x000010C00000) 0x02
15 0x0000E0000000 - 0x0000EFFFFFFF (0x000010000000) 0x02
16 0x0000FE000000 - 0x0000FE010FFF (0x000000011000) 0x02
17 0x0000FEC00000 - 0x0000FEC00FFF (0x000000001000) 0x02
18 0x0000FED00000 - 0x0000FED00FFF (0x000000001000) 0x02
19 0x0000FEE00000 - 0x0000FEE00FFF (0x000000001000) 0x02
20 0x0000FF000000 - 0x0000FFFFFFFF (0x000001000000) 0x02
21 0x000100000000 - 0x00026DFFFFFF (0x00016E000000) 0x01
Available RAM total:      7877 MB
Available RAM below 4 GB: 2021 MB
...
4589745D2A52272BB4F5BCFC4AE45309B872F469
A04BB9AED48127C8BC794B2737A2E241B0053C85
9AB3786B9DDF28AC5287125D9D53850527BB4C9F
944F860A5FDF5EDD3B1D2B363D578F22BF8E42D1 vmlinuz
45BB561269A6232C9C4776F4FA994904EE51B37F initramfs.img
DA2019732E096B1692C59A0B54F57C79CA38FC48 mbLinuz
1B8C8B8059DC02C8E1EEDB8766602A1FA5A4510F initrd-mbLinux64.gz

      
    

Important

IA64 UEFI Firmware for Intel Atom® x6000E Series processors (code name: Elkhart Lake) and 11th Gen Intel® Core™ processors (code name: Tiger Lake UP3) have a very restricted amount of RAM available below 4 GB for RTH to successfully mount a privileged-mode ECI-B (POS-RT) or privileged-mode ECI-X (POS-XE) images initrd-mbLinux64.gz ramdisk even with a minimalist assign 512 MB [/OS/1] "memory_size" = uint64: 0x20000000 .

Alias	Description	size	Bootable
rts-buster	POS initramfs file `initrd-mbLinux64.gz` (incl. `system.sfs`)	168 MB	OK
		201 MB	OK
		234 MB	NOK

[/SYSTEM]
     "IOMMU" = uint32: 1 # Activate IOMMU for all OSs
...
[/OS/1]

    "name"          = "Privileged RTOS"
    "boot_priority" = uint32: 1
    "memory_size"   = uint64: 0x20000000      # 512 MB
    "CPU"           = bytelist: 3,4           # e.g. two CPUs: bytelist: 3, 4
    "virtual_MMU"             = uint32: 1
    "restricted_IO"           = uint32: 1
...

      
    

$ readtrace -o 1
...
Init memory
   start          - end            (size          ) type
0x000000000000 - 0x00000009EFFF (0x00000009F000) 0x01
0x00000009F000 - 0x0000000FFFFF (0x000000061000) 0x02
0x000000100000 - 0x000030A5DFFF (0x00003095E000) 0x01
0x000030A5E000 - 0x000030A5EFFF (0x000000001000) 0x02
0x000030A5F000 - 0x0000338BCFFF (0x000002E5E000) 0x01
0x0000338BD000 - 0x0000338BDFFF (0x000000001000) 0x02
0x0000338BE000 - 0x0000400C7FFF (0x00000C80A000) 0x01
0x0000400C8000 - 0x00004012BFFF (0x000000064000) 0x14
0x00004012C000 - 0x000043B30FFF (0x000003A05000) 0x02
0x000043B31000 - 0x000043BB3FFF (0x000000083000) 0x04
0x000043BB4000 - 0x000043C4EFFF (0x00000009B000) 0x03
0x000043C4F000 - 0x000043C4FFFF (0x000000001000) 0x01
0x000043C50000 - 0x000047FFFFFF (0x0000043B0000) 0x02
0x000049A00000 - 0x00004F3FFFFF (0x000005A00000) 0x02
0x0000FED20000 - 0x0000FED7FFFF (0x000000060000) 0x02
0x0000FF400000 - 0x0000FFFFFFFF (0x000000C00000) 0x02
0x000100000000 - 0x0002B0BFFFFF (0x0001B0C00000) 0x01
Available RAM total:      7948 MB
Available RAM below 4 GB: 1024 MB
...

      
    

Intel® Time Coordinated Computing (TCC) enabled RTH configuration¶

RTH provides built-in support of latest Processors with Intel® Time Coordinated Computing (Intel® TCC) and other Intel® Resource Director Technology (Intel® RDT)

Important

Intel® Time Coordinated Computing Tools are NOT required. The RTH provides already an easy, abstracted and backward compatible feature configuration entries (for example /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt)

Partition L2/L3 cache memory between CPU cores and GPU¶

L2 or L3 Last Level of Cache (LLC) memory is shared between multiple IA64 CPUs by default. Intel® TCC or prior Intel® Cache Allocation Technology (Intel® CAT) is IA technology available on Intel Processors offering to divide and assign cache to individual CPUs.

../_images/TCC_CAT_L3_partitionned_seq-diag.png

To detects if Intel IA64 processor supporting Intel® CAT or Intel® TCC, an admin user will simply look into the Hypervisor log messages.

For example, the Intel(R) Core(TM) i5-6500TE CPU processor is featuring 4 logical CPUs with a total of 4 so called Classes of Service (COS). Further it can be seen that there are 12 cache segments available that can be assigned to individual COS.

$ less /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
[/SYSTEM]
"CPU_resource_partitioning" = uint32: 1

$ readtrace -o 1
...
Cache topology:
    |---------------|
CPU |  1|  2|  3|  4|
    |---------------|
L1d |   |   |   |   |
    |---------------|
L1i |   |   |   |   |
    |---------------|
L2  |   |   |   |   |
    |---------------|
L3  |               |  inclusive
    |---------------|
...
CAT supported on L3 Cache 1 (CPUs 1 - 4):
    Max. Class of Service (COS): 4
    Cache segments             : 12
...

      
    

If Intel® TCC or Intel® CAT is detected at RTH Intel in LLC/L3 or LLC/L2 IA64 CPU, an admin user can set extra [/OS/1] key L3_cache_X_segments or L2_cache_X_segments in RTH configuration file to change LLC segments partitioning of next consecutive system reboot.

$ vi /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...

[/OS/1]

    "name"          = "Privileged RTOS"
    "boot_priority" = uint32: 1
    "memory_size"   = uint64: 0x20000000      # 512 MB
    "CPU"           = bytelist: 3,4           # e.g. two CPUs: bytelist: 3, 4
    "virtual_MMU"             = uint32: 1
    "restricted_IO"           = uint32: 1
    "L3_cache_1_segments" = bytelist: 1,2,3,4,5,6

    "virtual_COM-to-log_port" = uint32: 0x3F8
    "trace_partition_number" = uint32: 1    # write log to /SHM/1

    [/OS/1/RUNTIME/0]

        "bootline" = "diskroot=/dev/ram ip=192.168.2.2:::::vnet0  debug=all verbose=all random.trust_cpu=on console=tty0 console=ttyS0,115200n8"
        "image_0"  = "mbLinuz64"
        "image_1"  = "initrd-mbLinux64.gz"

$ readtrace -o 1
...
Cache allocation:
  Package 1
    L3 #1  (12 segments)  6 MB
      OS  0 |      ******|  6 segments = 3 MB
      OS  1 |******      |  6 segments = 3 MB

      
    

Intel® TCC or Intel® CAT allow sub-feature called GT_CLOS to restrict the GPU usage of the certain LLC segments by the GPU (available on RTH development_mode only on certain Intel® processors).

$ vi /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
[/SYSTEM]
     "IOMMU" = uint32: 1 # Activate IOMMU for all OSs
     "development_mode" = uint32: 1
     "GPU_cache_restriction" = uint32: 1

$ readtrace -o 1
...
Supported GPU cache restriction settings:
  0: Cache segments: 1 2 3 4 5 6 7 8 9 10 11 12
  1: Cache segments: 7 8 9 10 11 12
  2: Cache segments: 11 12
  3: Cache segments: 12

Cache allocation:
  Package 1
    L3 #1  (12 segments)  6 MB
      OS  0 |      ******|  6 segments = 3 MB
      OS  1 |******      |  6 segments = 3 MB
      GPU   |      ******|  6 segments

      
    

Set SW SRAM as /dev/shm¶

RTH provides built-in Intel® Time Coordinated Computing (Intel® TCC) SW SRAM feature support on recent 11th Gen Intel® Core™ and Intel Atom® x6000E Series for a low-latency L2 or L3 cache data locality using RTH Application Programming Interface (API) for Shared-Memory device /dev/shm

$ vi /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
[/SHM/0]

    "name" = "trace_0"
    "size" = uint64: 0xF000

[/SHM/1]

    "name" = "trace_1"
    "size" = uint64: 0xF000

[/SHM/2]

    "name" = "low-latency"
    "size" = uint64: 0x100000   # one megabyte
    "cache_locked" = uint32: 3 # lock data in the fastest shared cache
...

$ readtrace -o 1
...
    L2 #1  (20 segments)  1.25 MB
      OS  0 |********************|  20 segments = 1.25 MB
    L2 #2  (20 segments)  1.25 MB
      OS  0 |********************|  20 segments = 1.25 MB
    L2 #3  (20 segments)  1.25 MB
      OS  0 |********************|  20 segments = 1.25 MB
    L2 #4  (20 segments)  1.25 MB
      OS  0 |********************|  20 segments = 1.25 MB
    L2 #8  (20 segments)  1.25 MB
      OS  1 |********************|  20 segments = 1.25 MB
    L3 #1  (12 segments)  24 MB
      OS  0   |      ***** | 5 segments = 10 MB
      OS  1   |******      | 6 segments = 12 MB
      SW SRAM |           *| 1 segment = 2 MB

Init time synchronization
Found OS "GPOS"
Found OS "Privileged RTOS"
Init Registry
Setup Operating Systems
Loading "Privileged RTOS"
Shared Memory Partitions
  0 0x00004086D000 - 0x00004087BFFF (0x00000000F000) "trace_0"
  1 0x00004085E000 - 0x00004086CFFF (0x00000000F000) "trace_1“
  2 0x000040080000 - 0x0000400FFFFF (0x000000080000) "low-latency"
/OS/1 Booting runtime 0
/OS/1 Activating Virtual MMU
/OS/1 COM: 000: Linux version 5.4.115-rt57-rts2.2.02.15407-intel-pk-standard+ (builder@f228

      
    

Note

For more references about RTS Shared-Memory (SHM) and RTH Application Programming Interface (API) for Linux, Windows™, QNX® Neutrino® , … please refer documentation or contact Real-Time Systems GmbH via E-mail info@real-time-systems.com

Set #AC on Split Lock¶

RTH provides Intel® Time Coordinated Computing (Intel® TCC) #AC (Alignment Check Fault) on Split Lock feature support on recent 11th Gen Intel® Core™ and Intel Atom® x6000E Series, To help detect CPU instructions with the LOCK prefix on an unaligned memory location that spans two cache lines, creating timing overhead than those CPU instructions operating on aligned memory locations.

Executing an instruction with the LOCK prefix on an unaligned location results in an #AC fault. It can be selectively turned on or off from RTH config file for an hypervised OS to enable fine-grained control of the feature.

$ vi /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
[/OS/0]
...
"AC_on_split_lock" = uint32: 0
...
[/OS/1]
...
"AC_on_split_lock" = uint32: 1
...

      
    

Enable Intel® Ethernet TSN Controllers¶

RTH enables latest Intel® Ethernet integrated or discrete controllers featuring IEEE 802.1 Time Sensitive Networking (TSN)

Change PCI device memory¶

At installation step rts-buster image make generic assumptions about the desired Ethernet devices PCI assignment to pos-rt and pos-xe runtimes. Modification to the ECI-R templates /boot/rth/<pos-rt|pos-xe>/Linux_Linux64_<nvme0n1pY|sdY>.txt or /boot/rth/<pos-rt|pos-xe>/WindowsEFI_Linux64__<nvme0n1pY|sdY>.txt or /boot/rth/WindowsEFI_Linux64.txt or /boot/rth/Linux_Linux64.txt may be necessary for customizing the TSN Ethernet Device assignment.

RTH boot logs are generally used to identify default PCI device assignments.

$ less /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
[/PCI]

    "default" = uint32: 0   # default is GPOS
...
$ readtrace -o 1
...
PCI devices:
bus dev func vendor device pin IRQ MSI mode | OS | description
  0   0  0   0x8086 0x191F  -    -  n   --  |  0 | Host/PCI bridge
  0   1  0   0x8086 0x1901  A    -  y   --  |    | PCIe Root Port to bus 1
  0   2  0   0x8086 0x1912  A   16  y  INTx |  0 | Display controller
  0   8  0   0x8086 0x1911  A   16  y   --  |    | Base system peripheral
  0  20  0   0x8086 0xA12F  A   16  y  INTx |  0 | USB xHCI controller
  0  20  2   0x8086 0xA131  C   18  y  INTx |  0 | Data acquisition controller
  0  22  0   0x8086 0xA13A  A   16  y  INTx |  0 | Simple comm. controller
  0  22  3   0x8086 0xA13D  D   19  y  INTx |  0 | Serial Controller
  0  23  0   0x8086 0xA102  A   16  y  INTx |  0 | AHCI SATA controller
  0  28  0   0x8086 0xA114  A    -  y   --  |    | PCIe Root Port to bus 2
  0  31  0   0x8086 0xA146  -    -  n   --  |  0 | PCI/ISA bridge
  0  31  2   0x8086 0xA121  -    -  n   --  |  0 | Memory controller
  0  31  3   0x8086 0xA170  A   16  y  INTx |  0 | Audio device
  0  31  4   0x8086 0xA123  A   16  n  INTx |  0 | SMBus controller
  0  31  6   0x8086 0x15B7  A   16  y  INTx |  0 | Network controller
  1   0  0   0x8086 0x15F2  A   16  y  MSI  |  0 | Network controller
  2   0  0   0x8086 0x1533  A   16  y  MSI  |  0 | Network controller
...

      
    

RTH configuration file provides entries assign exclusive access to a PCI devices for POS, based of PCI bus topology position (bus and device number) or simply by the vendor_ID:device_ID key pair.

For example, an admin user can see report that Intel i225-LM Ethernet Controller PCI devices (Device ID=0x15F2 and Vendor ID=0x8086) and Intel i210 Ethernet Controller PCI devices (Device ID=0x1533 and Vendor ID=0x8086) are both being assigned to the [/OS/1] Linux-rt runtimes on consecutive system reboot.

$ less /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
[/PCI]

    "default" = uint32: 0   # default is GPOS

# Device assignment with vendor ID / device ID and MSI Mode
[/PCI/0]
    "OS"         = uint32: 1
    "vendor_ID"  = uint32: 0x8086
    "device_ID"  = uint32: 0x15f2 # i225-LM Ethernet TSN endpoint
    "interrupt_mode" = uint32: 2

[/PCI/1]
    "OS"         = uint32: 1
    "vendor_ID"  = uint32: 0x8086
    "device_ID"  = uint32: 0x1533 # i210-T1 Ethernet TSN endpoint
    "interrupt_mode" = uint32: 2
...
[/PCI/6]
    "OS" = uint32: 1
    "vendor_ID" = uint32: 0x8086
    "device_ID" = uint32: 0x0D9F # i225-IT Ethernet TSN endpoint
    "interrupt_mode" = uint32: 2
...
$ rth -sysreboot && reboot
$ readtrace -o 1
...
PCI devices:
bus dev func vendor device pin IRQ MSI mode | OS | description
  0   0  0   0x8086 0x191F  -    -  n   --  |  0 | Host/PCI bridge
  0   1  0   0x8086 0x1901  A    -  y   --  |    | PCIe Root Port to bus 1
  0   2  0   0x8086 0x1912  A   16  y  INTx |  0 | Display controller
  0   8  0   0x8086 0x1911  A   16  y   --  |    | Base system peripheral
  0  20  0   0x8086 0xA12F  A   16  y  INTx |  0 | USB xHCI controller
  0  20  2   0x8086 0xA131  C   18  y  INTx |  0 | Data acquisition controller
  0  22  0   0x8086 0xA13A  A   16  y  INTx |  0 | Simple comm. controller
  0  22  3   0x8086 0xA13D  D   19  y  INTx |  0 | Serial Controller
  0  23  0   0x8086 0xA102  A   16  y  INTx |  0 | AHCI SATA controller
  0  28  0   0x8086 0xA114  A    -  y   --  |    | PCIe Root Port to bus 2
  0  31  0   0x8086 0xA146  -    -  n   --  |  0 | PCI/ISA bridge
  0  31  2   0x8086 0xA121  -    -  n   --  |  0 | Memory controller
  0  31  3   0x8086 0xA170  A   16  y  INTx |  0 | Audio device
  0  31  4   0x8086 0xA123  A   16  n  INTx |  0 | SMBus controller
  0  31  6   0x8086 0x15B7  A   16  y  INTx |  0 | Network controller
  1   0  0   0x8086 0x15F2  A   16  y  MSI  |  1 | Network controller
  2   0  0   0x8086 0x1533  A   16  y  MSI  |  1 | Network controller
PCI memory:
  0   2  0   BAR 0 0x0000DE000000 - 0x0000DEFFFFFF
  0   2  0   BAR 2 0x0000C0000000 - 0x0000CFFFFFFF
  0   8  0   BAR 0 0x0000DF450000 - 0x0000DF450FFF
  0  20  0   BAR 0 0x0000DF430000 - 0x0000DF43FFFF
  0  20  2   BAR 0 0x0000DF44F000 - 0x0000DF44FFFF
  0  22  0   BAR 0 0x0000DF44E000 - 0x0000DF44EFFF
  0  22  3   BAR 1 0x0000DF44D000 - 0x0000DF44DFFF
  0  23  0   BAR 0 0x0000DF448000 - 0x0000DF449FFF
  0  23  0   BAR 1 0x0000DF44C000 - 0x0000DF44C0FF
  0  23  0   BAR 5 0x0000DF44B000 - 0x0000DF44B7FF
  0  31  2   BAR 0 0x0000DF444000 - 0x0000DF447FFF
  0  31  3   BAR 0 0x0000DF440000 - 0x0000DF443FFF
  0  31  3   BAR 4 0x0000DF420000 - 0x0000DF42FFFF
  0  31  4   BAR 0 0x0000DF44A000 - 0x0000DF44A0FF
  0  31  6   BAR 0 0x0000DF400000 - 0x0000DF41FFFF
  1   0  0   BAR 0 0x0000DF200000 - 0x0000DF2FFFFF
  1   0  0   BAR 3 0x0000DF300000 - 0x0000DF303FFF
  2   0  0   BAR 0 0x0000DF000000 - 0x0000DF0FFFFF
  2   0  0   BAR 3 0x0000DF100000 - 0x0000DF103FFF
PCI I/O space:
  0   2  0   BAR 4 0xF000 - 0xF03F
  0  22  3   BAR 0 0xF0A0 - 0xF0A7
  0  23  0   BAR 2 0xF090 - 0xF097
  0  23  0   BAR 3 0xF080 - 0xF083
  0  23  0   BAR 4 0xF060 - 0xF07F
  0  31  4   BAR 4 0xF040 - 0xF05F
  2   0  0   BAR 2 0xE000 - 0xE01F

      
    

An admin user connecting to /OS/1 Linux-rt runtimes over ssh would confirm that Linux igc.ko and igb.ko kernel modules have successfully loaded onto the respective Intel i225-LM Ethernet PCIe Controller (PCI memory bars region 0 0x0000DF200000 - 0x0000DF2FFFFF and region 3 0x0000DF300000 - 0x0000DF303FF) and Intel i210 Ethernet PCIe Controller (PCI memory bar region 0 0x0000DF000000 - 0x0000DF0FFFFF, bar region 3 0x0000DF100000 - 0x0000DF103FFFF and PCI I/O space bar region 2 0xE000 - 0xE01) and that two new Linux net device are mounted successfully.

$ ssh root@192.168.2.2
$ lspci --vv
...
01:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15f2] (rev 01)
    Subsystem: Intel Corporation Device [8086:0000]
    Flags: bus master, fast devsel, latency 0
    Memory at df200000 (32-bit, non-prefetchable) [size=1M]
    Memory at df300000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Kernel driver in use: igc
    Kernel modules: igc

02:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
    Flags: bus master, fast devsel, latency 0
    Memory at df000000 (32-bit, non-prefetchable) [size=1M]
    I/O ports at e000 [size=32]
    Memory at df100000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Kernel driver in use: igb
    Kernel modules: igb
...
$ ip link show
...
4: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc taprio state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:18:7d:be:11:e7 brd ff:ff:ff:ff:ff:ff
5: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:a0:c9:00:00:00 brd ff:ff:ff:ff:ff:ff
...

      
    

Set MSI-X interrupt emulation¶

The RTH Release R5.4.01.27752 introduce a new Interrupt Mode "interrupt_mode" = uint32: 3 enables MSI-X interrupt emulation for devices only supporting a single MSI interrupts for multiple Ethernet TX & RX Hw-queues.

This interrupt mode is required for PCI Device memory mapping non MSI-X capable controller Intel Atom® x6000E Series processors (code name: Elkhart Lake) and 11th Gen Intel® Core™ processors (code name: Tiger Lake UP3) Ethernet MSI-only PCI Devices to enable Intel multiple-IRQ hooks available on stmmac-pci.ko kernel module. otherwise it will fail to load Linux with error stmmaceth 0000:00:1e.4 enp0s30f4: stmmac_request_irq: alloc mac MSI -22 (error: -22).

$ less /boot/rth/pos-rt/Linux_Linux64_nvme0n1p4.txt
...
# Device assignment with vendor ID / device ID and MSI-X emulationed Mode
[/PCI/3]
    "OS" = uint32: 1
    "vendor_ID" = uint32: 0x8086
    "device_ID" = uint32: 0x4BA0  # EHL mGbE Ethernet TSN endpoint #1
    "interrupt_mode" = uint32: 3

[/PCI/4]
    "OS" = uint32: 1
    "vendor_ID" = uint32: 0x8086
    "device_ID" = uint32: 0x4B32 # EHL mGbE Ethernet TSN endpoint #2
    "interrupt_mode" = uint32: 3

[/PCI/5]
    "OS" = uint32: 1
    "vendor_ID" = uint32: 0x8086
    "device_ID" = uint32: 0xA0AC # TGL-U mGbE Ethernet TSN endpoint
    "interrupt_mode" = uint32: 3


$ readtrace -o 1
...
PCI devices:
bus dev func vendor device pin IRQ MSI mode | OS | description
  0   0  0   0x8086 0x452E  -    -  n   --  |  0 | Host/PCI bridge
  0   2  0   0x8086 0x4571  A   16  y  INTx |  0 | Display controller
  0   8  0   0x8086 0x4511  A   16  y   --  |    | Base system peripheral
  0  16  0   0x8086 0x4B44  A   16  n  INTx |  0 | Serial bus controller
  0  16  1   0x8086 0x4B45  B   17  n  INTx |  0 | Serial bus controller
  0  17  0   0x8086 0x4B96  A   16  y  INTx |  0 | Simple comm. controller
  0  17  1   0x8086 0x4B97  B   17  y  INTx |  0 | Simple comm. controller
  0  19  0   0x8086 0x4B84  A   16  y  INTx |  0 | Serial bus controller
  0  20  0   0x8086 0x4B7D  A   16  y  INTx |  0 | USB xHCI controller
  0  20  2   0x8086 0x4B7F  -    -  n   --  |  0 | Memory controller
  0  21  0   0x8086 0x4B78  A   27  n  INTx |  0 | Serial bus controller
  0  21  2   0x8086 0x4B7A  C   29  n  INTx |  0 | Serial bus controller
  0  21  3   0x8086 0x4B7B  D   30  n  INTx |  0 | Serial bus controller
  0  22  0   0x8086 0x4B70  A   16  y  INTx |  0 | Simple comm. controller
  0  23  0   0x8086 0x4B63  A   16  y  MSI  |    | AHCI SATA controller
  0  25  0   0x8086 0x4B4B  A   31  n  INTx |  0 | Serial bus controller
  0  25  2   0x8086 0x4B4D  C   33  n  INTx |  0 | Simple comm. controller
  0  26  0   0x8086 0x4B47  A   16  n  INTx |  0 | SD Host Controller
  0  26  1   0x8086 0x4B48  B   17  n  INTx |  0 | SD Host Controller
  0  27  0   0x8086 0x4BB9  A   16  y  INTx |  0 | Serial bus controller
  0  27  1   0x8086 0x4BBA  B   17  y  INTx |  0 | Serial bus controller
  0  27  6   0x8086 0x4BBF  C   18  y  INTx |  0 | Serial bus controller
  0  28  0   0x8086 0x4B38  A    -  y   --  |    | PCIe Root Port to bus 1
  0  29  0   0x8086 0x4BB3  A   16  y   --  |    | Base system peripheral
  0  29  1   0x8086 0x4BA0  A   16  y  emul |  1 | Network controller
  0  30  0   0x8086 0x4B28  A   16  n  INTx |  0 | Simple comm. controller
  0  30  1   0x8086 0x4B29  B   17  n  INTx |  0 | Simple comm. controller
  0  30  4   0x8086 0x4B32  A   16  y  emul |  1 | Network controller
  0  31  0   0x8086 0x4B00  -    -  n   --  |  0 | PCI/ISA bridge
  0  31  3   0x8086 0x4B58  A   16  y  INTx |  0 | Audio multimedia device
  0  31  4   0x8086 0x4B23  A   16  n  INTx |  0 | SMBus controller
  0  31  5   0x8086 0x4B24  -    -  n   --  |  0 | Serial bus controller
  1   0  0   0x8086 0x15F2  A   16  y  MSI  |  1 | Network controller

      
    

The setting is only valid if:

the device has an MSI capability and no MSI-X capability (check with lspci -vvv)
the IOMMU is enabled from UEFI menu (ie VT-d=enabled)
the PCI device is assigned to an OS running in the Privileged Mode with virtual_MMU and restricted_IO active.

rts-buster image make generic assumptions about the desired POS Windriver® VxWorks® 7 (pos-vx) or Blackberry QNX® Neutrino® (pos-qn) RTOS. Modification to the ECI-R templates /boot/rth/pos-vx/Linux_VxWorks64.txt or /boot/rth/pos-qn/Linux_QNX64.txt may be necessary for customizing the POS.

Section	Description
Set Windriver® VxWorks® 7 RTH privileged-mode (pos-vx)	Windriver® VxWorks® 7 privileged-mode `pos-vx` RTH configuration
Set Blackberry QNX® Neutrino® RTH privileged-mode (pos-qn)	Blackberry QNX® Neutrino® privileged-mode `pos-qn` RTH configuration

Note

If you are using, or plan to use, a different version of the operating system, please direct an E-mail inquiry to: info@real-time-systems.com for further directions.

RTH Sanity-Check Testing¶

Sanity-Check #1: Communication between Virtualized and Privileged OS runtimes¶

The following section is applicable to:

RTS Real-Time Hypervisor (aka RTH) comes with several ways to communicate across operating systems:

kernel boot dmesg and journald logs redirect
Shared memory
Virtual network

This tutorial will show you how to use the virtual network driver, which has been built-in into the Edge Controls for Industrial (ECI) privileged-RT OS runtime and also installed in your virtualized-OS either Ubuntu or Windows.

The virtual network driver is provided by the hypervisor. It operates using standard TCP/IP drivers. More details can be found in the RTH User guide.

Step 1: Verify privileged-mode OS (POS) is loaded successfully¶

Verify the privileged-mode OS redirect all boot log message to RTH shm/1

$ cat /proc/cmdline
root=/dev/ram ip=192.168.2.2/255.255.255.0 <...> earlyprintk=ttyS0,115200 console=ttyS0,115200n8 debug

Verify readtrace reading of shm/1 containing OS/1 (RT-OS) dmesg and journald messages output reports successful boot

$ readtrace -o 1
Real-Time Hypervisor
Copyright 2006 - 2020 Real-Time Systems GmbH
Version R5.3.01.26202

Init CPU
Init heap
Init registry
Read configuration
WARNING:  DEVELOPMENT MODE ACTIVE.   NOT TO BE USED IN PRODUCTION.
Init memory
   start          - end            (size          ) type
 0 0x000000000000 - 0x000000057FFF (0x000000058000) 0x01
 1 0x000000058000 - 0x000000058FFF (0x000000001000) 0x02
 2 0x000000059000 - 0x00000009EFFF (0x000000046000) 0x01
 3 0x00000009F000 - 0x0000000FFFFF (0x000000061000) 0x02
 4 0x000000100000 - 0x000077DD0FFF (0x000077CD1000) 0x01
 5 0x000077DD1000 - 0x000077DD1FFF (0x000000001000) 0x04
 6 0x000077DD2000 - 0x000077DD2FFF (0x000000001000) 0x02
 7 0x000077DD3000 - 0x00007E44BFFF (0x000006679000) 0x01
 8 0x00007E44C000 - 0x00007E78AFFF (0x00000033F000) 0x02
 9 0x00007E78B000 - 0x00007E8FAFFF (0x000000170000) 0x01
10 0x00007E8FB000 - 0x00007EFE5FFF (0x0000006EB000) 0x04
11 0x00007EFE6000 - 0x00007F3A6FFF (0x0000003C1000) 0x02
12 0x00007F3A7000 - 0x00007F3FEFFF (0x000000058000) 0x14
13 0x00007F3FF000 - 0x00007F3FFFFF (0x000000001000) 0x01
14 0x00007F400000 - 0x00008FFFFFFF (0x000010C00000) 0x02
15 0x0000E0000000 - 0x0000EFFFFFFF (0x000010000000) 0x02
16 0x0000FE000000 - 0x0000FE010FFF (0x000000011000) 0x02
17 0x0000FEC00000 - 0x0000FEC00FFF (0x000000001000) 0x02
18 0x0000FED00000 - 0x0000FED00FFF (0x000000001000) 0x02
19 0x0000FEE00000 - 0x0000FEE00FFF (0x000000001000) 0x02
20 0x0000FF000000 - 0x0000FFFFFFFF (0x000001000000) 0x02
21 0x000100000000 - 0x00026DFFFFFF (0x00016E000000) 0x01
Available RAM total:      7877 MB
Available RAM below 4 GB: 2021 MB
Check license
    company: Intel Deutschland GmbH
    project: Intel ECI 2.0 evaluation until 2021-03-01
Init interrupts
IOMMU detected
Intel(R) Core(TM) i5-6500TE CPU @ 2.30GHz, ID: 06_5EH, stepping: 3H
 * 64-Bit
 * VT-x
 * TPR Shadowing
 * Virtualized APIC access
 * Extended Page Tables (EPT)
 * Unrestricted Guest
 * Speculation Control
CPU microcode revision 0xC2
Detected 4 logical CPUs
Detected 1 physical CPU
Cache topology:
    |---------------|
CPU |  1|  2|  3|  4|
    |---------------|
L1d |   |   |   |   |
    |---------------|
L1i |   |   |   |   |
    |---------------|
L2  |   |   |   |   |
    |---------------|
L3  |               |  inclusive
    |---------------|
4589745D2A52272BB4F5BCFC4AE45309B872F469
A04BB9AED48127C8BC794B2737A2E241B0053C85
9AB3786B9DDF28AC5287125D9D53850527BB4C9F
944F860A5FDF5EDD3B1D2B363D578F22BF8E42D1 vmlinuz
45BB561269A6232C9C4776F4FA994904EE51B37F initramfs.img
DA2019732E096B1692C59A0B54F57C79CA38FC48 mbLinuz
1B8C8B8059DC02C8E1EEDB8766602A1FA5A4510F initrd-mbLinux.gz
OS ID  0 name: "Ubuntu 16.04"
         CPU: 1 2
OS ID  1 name: "ECI-R 1.0 Yocto Poky Linux-rt Intel LTS"
         CPU: 3 4
Init shared memory
Key "security_level" is set to zero.
All guest operating systems have unrestricted API permissions.
Init devices
Date: 2020-12-06
No device with Vendor ID 0x1059 and Device ID 0xA100 present.
PCI devices:
bus dev func vendor device pin IRQ MSI mode | OS | description
  0   0  0   0x8086 0x191F  -    -  n   --  |  0 | Host/PCI bridge
  0   1  0   0x8086 0x1901  A    -  y   --  |    | PCIe Root Port to bus 1
  0   2  0   0x8086 0x1912  A   16  y  INTx |  0 | Display controller
  0   8  0   0x8086 0x1911  A   16  y   --  |    | Base system peripheral
  0  20  0   0x8086 0xA12F  A   16  y  INTx |  0 | USB xHCI controller
  0  20  2   0x8086 0xA131  C   18  y  INTx |  0 | Data acquisition controller
  0  22  0   0x8086 0xA13A  A   16  y  INTx |  0 | Simple comm. controller
  0  22  3   0x8086 0xA13D  D   19  y  INTx |  0 | Serial Controller
  0  23  0   0x8086 0xA102  A   16  y  INTx |  0 | AHCI SATA controller
  0  28  0   0x8086 0xA114  A    -  y   --  |    | PCIe Root Port to bus 2
  0  31  0   0x8086 0xA146  -    -  n   --  |  0 | PCI/ISA bridge
  0  31  2   0x8086 0xA121  -    -  n   --  |  0 | Memory controller
  0  31  3   0x8086 0xA170  A   16  y  INTx |  0 | Audio device
  0  31  4   0x8086 0xA123  A   16  n  INTx |  0 | SMBus controller
  0  31  6   0x8086 0x15B7  A   16  y  INTx |  0 | Network controller
  1   0  0   0x8086 0x15F2  A   16  y  MSI  |  1 | Network controller
  2   0  0   0x8086 0x1533  A   16  y  MSI  |  1 | Network controller
PCI memory:
  0   2  0   BAR 0 0x0000DE000000 - 0x0000DEFFFFFF
  0   2  0   BAR 2 0x0000C0000000 - 0x0000CFFFFFFF
  0   8  0   BAR 0 0x0000DF450000 - 0x0000DF450FFF
  0  20  0   BAR 0 0x0000DF430000 - 0x0000DF43FFFF
  0  20  2   BAR 0 0x0000DF44F000 - 0x0000DF44FFFF
  0  22  0   BAR 0 0x0000DF44E000 - 0x0000DF44EFFF
  0  22  3   BAR 1 0x0000DF44D000 - 0x0000DF44DFFF
  0  23  0   BAR 0 0x0000DF448000 - 0x0000DF449FFF
  0  23  0   BAR 1 0x0000DF44C000 - 0x0000DF44C0FF
  0  23  0   BAR 5 0x0000DF44B000 - 0x0000DF44B7FF
  0  31  2   BAR 0 0x0000DF444000 - 0x0000DF447FFF
  0  31  3   BAR 0 0x0000DF440000 - 0x0000DF443FFF
  0  31  3   BAR 4 0x0000DF420000 - 0x0000DF42FFFF
  0  31  4   BAR 0 0x0000DF44A000 - 0x0000DF44A0FF
  0  31  6   BAR 0 0x0000DF400000 - 0x0000DF41FFFF
  1   0  0   BAR 0 0x0000DF200000 - 0x0000DF2FFFFF
  1   0  0   BAR 3 0x0000DF300000 - 0x0000DF303FFF
  2   0  0   BAR 0 0x0000DF000000 - 0x0000DF0FFFFF
  2   0  0   BAR 3 0x0000DF100000 - 0x0000DF103FFF
PCI I/O space:
  0   2  0   BAR 4 0xF000 - 0xF03F
  0  22  3   BAR 0 0xF0A0 - 0xF0A7
  0  23  0   BAR 2 0xF090 - 0xF097
  0  23  0   BAR 3 0xF080 - 0xF083
  0  23  0   BAR 4 0xF060 - 0xF07F
  0  31  4   BAR 4 0xF040 - 0xF05F
  2   0  0   BAR 2 0xE000 - 0xE01F
Init IOMMU
HPET found. Possible IRQs: 20 21 22 23
COM1: I/O port 0x03F8 IRQ  4 unassigned
COM2: I/O port 0x02F8 IRQ  3 OS 1
COM3: I/O port 0x03E8 IRQ 10 OS 1
COM4: I/O port 0x02E8 IRQ 10 OS 1
COM5: I/O port 0x03E0 IRQ 10 OS 1
COM6: I/O port 0x02E0 IRQ 10 OS 1
AHCI V1.31 controller (0/23/0) (Vendor:0x8086 Device:0xA102):
  Port 0: ATA Disk Drive - Link Speed: GEN3
    No Drive assignment for port 0.
  Port 1: No device
  Port 2: No device
  Port 3: ATA Disk Drive - Link Speed: GEN3
    No Drive assignment for port 3.
/OS/1 "virtual_COM-to-log_port" 0x3F8 is configured but the IRQ of the COM port is also assigned.
Assigned IRQs:
  IRQ | OS ID
    1 |  0
    2 |  0
    3 |  1
    4 |  1
    5 |  0
    6 |  0
    7 |  0
    8 |  0
    9 |  0
   10 |  1
   11 |  0
   12 |  0
   13 |  0
   14 |  0
   15 |  0
   16 |  0
   18 |  0
   19 |  0
I/O port configuration:
start  -  end   ( size ) OS
0x0060 - 0x0060 (0x0001) 0
0x0064 - 0x0064 (0x0001) 0
0x02E0 - 0x02EF (0x0010) 1
0x02F8 - 0x02FF (0x0008) 1
0x03E0 - 0x03EF (0x0010) 1
0x03F8 - 0x03FF (0x0008) 0
0x0CF8 - 0x0CF8 (0x0001) -
0x0CFC - 0x0CFF (0x0004) -
0xE000 - 0xE01F (0x0020) 1
0xF000 - 0xF083 (0x0084) 0
0xF090 - 0xF097 (0x0008) 0
0xF0A0 - 0xF0A7 (0x0008) 0
System Information:
    Manufacturer: "iEi"
    Product Name: "SER0"
Initial CPU frequency:  2300 MHz
Configurable range from 1200 MHz to 2300 MHz in increments of 100 MHz
CPU C1E state disabled
GPU frequency range from 350 MHz to 1000 MHz in increments of 50 MHz
CAT supported on L3 Cache 1 (CPUs 1 - 4):
    Max. Class of Service (COS): 4
    Cache segments             : 12
Supported GPU cache restriction settings:
  0: Cache segments: 1 2 3 4 5 6 7 8 9 10 11 12
  1: Cache segments: 7 8 9 10 11 12
  2: Cache segments: 11 12
  3: Cache segments: 12
Cache allocation:
  Package 1
    L3 #1  (12 segments)  6 MB
      OS  0 |      ******|  6 segments = 3 MB
      OS  1 |******      |  6 segments = 3 MB
      GPU   |      ******|  6 segments
Init time synchronization
Found OS "Ubuntu 20.04"
Found OS "ECI-R 2.0 Yocto Poky Linux-rt Intel LTS"
Init Registry
Setup Operating Systems
Shared Memory Partitions
  0 0x00007E7D8000 - 0x00007E7E6FFF (0x00000000F000) "trace-ubuntu"
  1 0x00007E7C9000 - 0x00007E7D7FFF (0x00000000F000) "trace-yocto"
/OS/1 Booting runtime 0
/OS/1 Activating IOMMU
/OS/1 Activating Virtual MMU
/OS/1 SMI logging enabled
/OS/1 Attempt to read  data from physical address 0x0000000F0000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000F1000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000F2000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000F3000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000F4000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000F5000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000F6000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000F7000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000F8000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000F9000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000FA000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000FB000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000FC000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000FD000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000FE000. Redirecting.
/OS/1 Attempt to read  data from physical address 0x0000000FF000. Redirecting.
/OS/1 COM: 000: Linux version 5.4.59-rt36-rts2.2.00.15195-intel-pk-preempt-rt (oe-user@oe-h
/OS/1 COM: ost) (gcc version 9.3.0 (GCC)) #1 SMP PREEMPT_RT Tue Sep 1 04:58:38 UTC 2020
/OS/1 COM: 000: Command line: root=/dev/ram ip=192.168.2.2/255.255.255.0 console=tty0  cons
/OS/1 COM: ole=ttyS0,115200n8 console=ttyS2,115200n8 debug=all verbose=all tsnep=10.11.12.1
/OS/1 COM: 00/255.255.255.0 qdisc=TAPRIO tsn.vlan=3 rthCtrl=0x1A000000 rthCtrlSize=0xF000 r
/OS/1 COM: thActive pci=conf1 acpi=off
/OS/1 COM: 000: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
...

      
    

Step 2: Verify virtual network driver status¶

Run ifconfig or ip link and verify the enp0s3f1 network has an IP address of 192.168.2.1 .

Note

If the enp0s3f1 device does not have an IP address, assign it one using sudo ip addr add 192.168.2.1/24 dev enp0s3f1 . Depending on your system, the device may also be named vnet0.

Step 3: SSH from Virtualized-OS to the privileged-OS runtime¶

SSH from GPOS to the POS runtime using IP address set in RTH configuration file boot command line ip=192.168.2.2/255.255.255.0.
$ ssh root@192.168.2.2
Note

This IP address was set in the hypervisor configuration file, /boot/rth/Linux_Linux64.txt , and can be set to something else later if you wish. Details on setting the IP address are found in the RTH User Guide..

Note

If it asks you “Are you sure you want to continue connecting (yes/no), type yes
You can verify you are in the RTOS and not in Ubuntu by running
$ uname -r
Note

In this setup, the ECI privileged OS-runtime kernel and root file system (rootfs) is running in RAM to improve performance, consequently modifications made will be erased after a restart. Optional persistent rootfs can enabled.

Step 4: Optional - Port forwarding SSH into the ECI RT-OS runtime¶

While this step is not strictly necessary to start using ECI, it will be very helpful later on when you need to debug what’s going on in the RT-OS runtime.

In this step you will SSH from another system on the same network to the RTOS via the GPOS virtualized-OS using port-forwarding.

The port forwarding has already been setup in the installation script. Port 80 for the POS runtime shall be mapped to 8080 in GPOS. So to SSH into the RTOS, SSH to your GPOS using root user and port 8080.

$ ssh root@<GPOS local IPv4 Address>:8080

The Ubuntu OS has several other ports that have been forwarded. This configuration was done by the installation script.

Check the port configuration by running in Ubuntu

$ sudo iptables --list

You should see 5 rules.

These rules allow you to connect via SSH from another machine to the ECI POS runtime via the GPOS OS (port 80, or ssh in the screen output). The rest of the rules: 1217, 4840, 11740, and 9022 (http in the screen output) are to allow the CODESYS Development System to connect from a Windows machine to the ECI RT-OS runtime and for another machine to access the visualization from the CODESYS application.

Note

By default, any i210 device with the Vendor ID of 0x8086 and Device ID of 0x1533 (all i210 devices should have these IDs) has been mapped to the ECI RT-OS runtime in the /boot/rth/Linux_Linux64.txt hypervisor configuration file located on Ubuntu /boot/rth or Windows C:\rth\ files system.

Sanity-Check #2: Manage privileged-ECI OS runtime lifecycle¶

The following section is applicable to:

RTS Real-Time Hypervisor (aka RTH) comes with several ways to manage each operating systems lifecycle independently at runtime (os-reboot, edit kernel cmdline ):

RTH Event API
HW cpu warm reset reset, cpu halt
OS images reload
OS boot cmdline edit

This tutorial will show you how to use the RTH tools, which has been built-in into Edge Controls for Industrial (ECI) POS runtime and also installed in your virtualized GPOS.

Step 1: Verify rthbase driver status¶

In a console on ECI privileged-OS:

Run lsmod command and verify the rthbasedrv module is loaded in kernel-space.
Verify sudo rth -h commands returns the help menu insuring /bin/rth and /lib/libRth.so x86_64 binaries are dynamically-linked into user-space process.

In a console on Windows (or Ubuntu) Virtualized-OS:

Run lsmod command and verify the virtrthbasedrv module is loaded in kernel-space.
Verify sudo rth -h commands returns the help menu insuring /bin/rth and /lib/libRth.so x86_64 binaries are dynamically-linked into user-space process.

Step 2: Verify CPU core warm-reset¶

In a console on Windows (or Ubuntu) Virtualized-OS:

Verify sudo rth -o 1 -reset CPU warm-reset commands returns successful OS/1 reboot of privileged RT-OS.
$ sudo rth -o 1 -reset

Step 3: Reboot ECI privileged-OS individually¶

Verify sudo rth -o 1 -shutdown OS/1 shutdown commands returns graceful shutdown of privileged RT-OS
$ sudo rth -o 1 -halt
Verify sudo rth -o 1 -boot OS/1 boot commands returns success boot of privileged RT-OS
$ sudo rth -o 1 -boot

Step 4: system reset of all CPUs : Hypervisor, virtualized OS and privileged-OS¶

Verify sudo rth -sysreboot && reboot system reboot commands returns graceful shutdown of the x86_64 target machine (ie. PMIC cold-reset event)
$ sudo rth -sysshutdown
Verify sudo rth -sysreboot && reboot system reboot commands returns successful reboot of Hypervisor, privileged RT-OS and virtualized OS
$ sudo rth -sysreboot

Sanity-Check #3: Troubleshooting privileged-ECI OS runtime¶

RTS Real-Time Hypervisor (aka RTH) comes with several ways to debug each operating systems runtime state ():

redirect Hypervisor and Kernel boot (printk) log messages to a dedicated Ethernet interface
redirect Kernel boot (printk) log messages and a console to a dedicated COMx/ttySx port (TBD)
redirect Kernel debugger (KGDB) and a console to a dedicated COMx/ttySx port (TBD)

This tutorial will show you how to use the RTH hypervisor debug tools, which has been built-in into Edge Controls for Industrial (ECI) privileged-RT OS runtime and also installed in your virtualized-OS either Ubuntu or Windows.

Step 0: Find the Ethernet controllers PCI ID¶

$ lspci | grep Ethernet
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-LM (rev 05)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

Step 1: Dedicate an Ethernet Port for RTH logging¶

Edit RTH hypervisor configuration file append the following section:

[/LOG]
"bus" = uint32: 0
"device" = uint32: 19
"function" = uint32: 0
"host_IP" = "192.168.100.48"
"target_IP" = "192.168.100.250"

      
    

Step 2: Read log-message remotely¶

From another Linux host machine to read Hypervisor Live boot message

$ sudo apt-get install netcat
$ ip addr add 192.168.100.48/24  eth0
$ netcat -l -u 514

Note

also possible with PuTTY on a Windows environment

Please refer to Real-Time Hypervisor Installation Manual Pre-installed Microsoft® Windows™ Systems (rev. 103.017.180730) for further details.

RTH Best Known Methods¶

Document / Section	Description
RTH BKMs	This section contains a collection of BKMs (Best Known Methods) for the RTS Hypervisor.