Benchmarks & Performance Characterization¶
This section discusses the various benchmark tests and configuration to evaluate system performance.
The following section is applicable to:
48-Hour Benchmark¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
48-Hour Benchmark |
(see respective benchmark) |
1.1 |
Intel created |
The 48-Hour Benchmark exercises the Caterpillar and Cyclictest Workload benchmarks with and without Cache Allocation Technology to demonstrate the benefits of Cache Allocation Technology on real-time workloads.
The ECI 48-Hour Benchmark exercises the following tests and configurations:
- Test 1:
Benchmark: Caterpillar
Benchmark Affinity: Core 3
Benchmark Priority:
chrt 99Noisy Neighbor: stress-ng
memcpyNoisy Neighbor Affinity: 0
Duration: ~3 hours
Cache Allocation Technology Enabled: No
- Test 2:
Benchmark: Caterpillar
Benchmark Affinity: Core 3
Benchmark Priority:
chrt 99Noisy Neighbor: stress-ng
memcpyNoisy Neighbor Affinity: Core 0
Duration: ~3 hours
Cache Allocation Technology Enabled: Yes (
COS0=0x0f→ Cores 0-2,COS3=0xf0→ Core 3)
- Test 3:
Benchmark: Cyclictest Workload with interval of 250μs
Benchmark Affinity: Core 3
Benchmark Priority:
chrt 99Noisy Neighbor: stress-ng
memcpyNoisy Neighbor Affinity: Core 0
Duration: 24 hours
Cache Allocation Technology Enabled: No
- Test 4:
Benchmark: Cyclictest Workload with interval of 250μs
Benchmark Affinity: Core 3
Benchmark Priority:
chrt 99Noisy Neighbor: stress-ng
memcpyNoisy Neighbor Affinity: Core 0
Duration: 24 hours
Cache Allocation Technology Enabled: Yes (
COS0=0x0f→ Cores 0-2,COS1=0xf0→ Core 3)
Install 48-Hour Benchmark¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
$ sudo apt install mega-benchmark
Execute 48-Hour Benchmark¶
To start the benchmark, run the following command:
$ /opt/benchmarking/mega-benchmark/48_hour_benchmark.sh
Results are logged to the files named: results_48_hour_benchmark_<date>, caterpillar_without_cat.log, and caterpillar_with_cat.log.
Interpret 48-Hour Benchmark Results¶
Since the 48-Hour Benchmark comprises a subset of benchmarks, refer to the respective benchmark sections:
Example of 48-Hour Benchmark Data¶
Refer to 48-Hour Benchmark Results for the sample data collected from the 48-Hour benchmark.
Mega Benchmark¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
Mega Benchmark |
(see respective benchmark) |
1.1 |
Intel created |
The ECI mega-benchmark exercises the following tests:
Rhealstone xLatency
Rhealstone stress test
Install Mega Benchmark¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
$ sudo apt install mega-benchmark
Execute Mega Benchmark¶
To start the benchmark, run the following command:
$ /opt/benchmarking/mega-benchmark/mega_benchmark.sh
Results are logged to a file named: results_mega_benchmark_<date>
Interpret Mega Benchmark Results¶
Since the Mega Benchmark comprises a subset of benchmarks, refer to the respective benchmark sections:
Cyclictest Workload¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
Cyclictest |
microseconds |
1.50 |
https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/snapshot/rt-tests-1.5.tar.gz |
Cyclictest is most commonly used for benchmarking real-time (RT) systems. It is one of the most frequently used tools for evaluating the relative performance of an RT. Cyclictest accurately and repeatedly measures the difference between a thread’s intended wake-up time and the time at which it actually wakes up to provide statistics about the system’s latency. It can measure latency in real-time systems caused by the hardware, the firmware, and the operating system.
Install Cyclictest Workload¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
$ sudo apt install rt-tests-scripts
Execute Cyclictest Workload¶
An example script that runs the cyclictest benchmark and the README is available at /opt/benchmarking/rt-tests. The script performs the following runtime optimizations before executing the benchmark:
Assigns benchmark thread affinity to core 3
Assigns non-benchmark thread affinity to core 0
Changes the priority of benchmark thread to 95 (using:
chrt -f 95)Disables kernel machine check interrupt
Increases the thread runtime utilization to infinity
To start the benchmark, run the following command:
$ sudo /opt/benchmarking/rt-tests/start-cyclic.py
Default parameters are used unless otherwise specified. Run the script with --help to list the modifiable arguments.
See also
Sanity Check 0 - Cyclictest Workload provides an example on how to run this benchmark and display the results.
Interpret Cyclictest Results¶
Short |
Explanation |
|---|---|
T |
Thread: Thread index and thread ID |
P |
Priority: RT thread priority |
I |
Interval: Intended wake up period for the latency measuring threads |
C |
Count: Number of times the latency was measured that is, iteration count |
Min |
Minimum: Minimum latency that was measured |
Act |
Actual: Latency measured during the latest completed iteration |
Avg |
Average: Average latency that was measured |
Max |
Maximum: Maximum latency that was measured |
On a non-realtime system, the result might be similar to the following:
T: 0 ( 3431) P:99 I:1000 C: 100000 Min: 5 Act: 10 Avg: 14 Max: 39242
T: 1 ( 3432) P:98 I:1500 C: 66934 Min: 4 Act: 10 Avg: 17 Max: 39661
The right-most column contains the most important result, that is, the worst-case latency of 39.242 ms (Max value).
On a realtime-enabled system, the result might be similar to the following:
T: 0 ( 3407) P:99 I:1000 C: 100000 Min: 7 Act: 10 Avg: 10 Max: 18
T: 1 ( 3408) P:98 I:1500 C: 67043 Min: 7 Act: 8 Avg: 10 Max: 22
This result indicates an apparent short-term worst-case latency of 18 ms. According to this, it is important to pay attention to the Max values as these are indicators of outliers. Even if the system has decent Avg (average) values, a single outlier as indicated by Max is enough to break or disturb a real-time system.
According to the README from the https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git repository:
Running cyclictest only over a short period of time and without creating appropriate real-time stress conditions is rather meaningless, since the execution of an asynchronous event from idle state is normally always quite fast, and every - even non-RT system - can do that. The challenge is to minimize the latency when reacting to an asynchronuous event, irrespective of what code path is executed at the time when the external event arrives. Therefore, specific stress conditions must be present while cyclictest is running to reliably determine the worst-case latency of a given system.
Additional Cyclictest Workload¶
Another script, rt_bmark.py, performs stress workloads when executing the benchmark cyclictest.
To start the benchmark, run the following command:
$ sudo /opt/benchmarking/rt-tests/rt_bmark.py
Jitter¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
Jitter |
CPU Cycles |
1.9 |
Intel created |
The Jitter benchmark measures the execution time variation of a CPU test workload. The performance of the workload is impacted by kernel interrupts. Minimizing these interrupts also minimizes the jitter that applications could potentially experience.
Install Jitter¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
$ sudo apt install jitter
Execute Jitter¶
An example script that runs the Jitter benchmark is available at /opt/benchmarking/jitter/jitter. The script performs the following runtime optimizations before executing the benchmark:
Assigns benchmark thread affinity to core 3
Assigns non-benchmark thread affinity to core 0
Changes the priority of benchmark thread to 95 (using:
chrt -f 95)Disables kernel machine check interrupt
Increases the thread runtime utilization to infinity
Starts a “noisy neighbor” on an adjacent core using
stress-ng --cpu 1 --memcpy
To assess the impact of Cache Allocation Technology, it is useful to perform the benchmark with and without Cache Allocation Technology enabled. The two examples below provide a method of achieving this.
The first step is to establish a baseline performance to compare against. Start the benchmark without Cache Allocation Technology. Take note of the
Inst_Jittervalue, since indicates the execution jitter in CPU cycles experienced by the benchmark.$ sudo /opt/benchmarking/jitter/start-benchmark.py
Now, start the benchmark with Cache Allocation Technology. The pqos-helper tool is being used here to partition the CPU cache and isolate the cache used by the benchmark. Note how only core 3 is assigned to Class of Service (COS) 1, and the only task executing on core 3 will be the benchmark. This cache configuration should reduce cache evictions experienced by the benchmark caused by the “noisy neighbor”, resulting in reduced execution jitter.
$ sudo /opt/pqos/pqos-helper.py --cos0 0x0f --cos1 0xf0 --assign_cos "0=0 0=1 0=2 1=3" --pqos_rst --pqos_msr --command "/opt/benchmarking/jitter/start-benchmark.py"
Compare the results of the two runs. Ideally, the run with Cache Allocation Technology enabled should exhibit lower execution jitter.
Default parameters are used unless otherwise specified. Run the script with --help to list the modifiable arguments.
See also
Sanity Check 2 - Jitter Workload provides an example on how to run this benchmark and display the results.
Interpret Jitter Results¶
Inst_Min Inst_Max Inst_jitter last_Exec Abs_min Abs_max tmp Interval Sample No
177205 235598 58393 177219 164702 243978 1227096064 3604948177 200 66777
The most important measurement is Inst_jitter. This measurement describes the execution jitter (in CPU cycles) during the display update interval. It is desired that Inst_jitter be as low as possible. The delta between Abs_max and Abs_min is the overall jitter spread. Ideally, the spread must be as low as possible.
LMbench¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
LMbench |
Nanosecond |
3.0a9 |
LMbench is a suite of simple, portable, ANSI/C microbenchmarks for UNIX/POSIX. In general, it measures two key features: latency and bandwidth. LMbench is intended to provide system developers an insight into basic costs of key operations.
lat_mem_rd measures memory read latency for varying memory sizes and strides. The results are reported in nanoseconds per load and have been verified to be accurate to within a few nanoseconds on an SGI Indy. The entire memory hierarchy is measured, including onboard cache latency and size, external cache latency and size, main memory latency, and TLB miss latency. Only data accesses are measured; the instruction cache is not measured.
The specific cache and memory latency benchmark is available at /usr/bin/lat_mem_rd.
Install LMBench¶
You can install this component from the open-source Debian* or Ubuntu* APT repository. Run the following command to install this component.
- Install from individual Deb package
$ sudo apt install lmbench
Execute LMBench¶
To start this benchmark, run the following command:
$ cd ~/;taskset -c $CORE_AFFINITY /usr/bin/lat_mem_rd -P 1 $MEMRD_SIZE $MEMRD_CHUNCKS &> /tmp/result_lat_mem_rd.txt
Default parameters are used unless otherwise specified (MEMRD_SIZE and MEMRD_CHUNCKS variables). Run the script with --help to list the modifiable arguments.
See also
Sanity Check 1 - LMbench Workload provides an example on how to run this benchmark and display the results.
Interpret LMbench Results¶
The output is best examined graphically. You get a graph with four plateaus. The graph should plotted in log base 2 of the array size on the X axis and the latency on the Y axis. Each stride is then plotted as a curve. The plateaus that appear correspond to the onboard cache (if present), external cache (if present, main memory latency, and TLB miss latency.
Caterpillar¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
Caterpillar |
CPU Cycles |
1.2 |
Intel created |
The Caterpillar benchmark measures the execution time variation of a memory test workload. The performance of the workload is impacted by cache misses. Using Cache Allocation Technology improves application performance by assigning CPU affinity to cache ways, which can be dedicated to real-time applications.
Install Caterpillar¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
$ sudo apt install caterpillar
Execute Caterpillar¶
An example script running the Caterpillar benchmark is available at /opt/benchmarking/caterpillar/start-benchmark.py. The script performs the following runtime optimizations before executing the benchmark:
Assigns benchmark thread affinity to core 3
Assigns non-benchmark thread affinity to core 0
Changes priority of benchmark thread to 95 (using:
chrt -f 95)Disables kernel machine check interrupt
Increases thread runtime utilization to infinity
Starts a “noisy neighbor” on an adjacent core using
stress-ng --cpu 1 --memcpy
To assess the impact of Cache Allocation Technology, it is useful to perform the benchmark with and without Cache Allocation Technology enabled. The two examples below provide a method of achieving this.
The first step is to establish a baseline performance to compare against. Start the benchmark without Cache Allocation Technology. Take note of the
SmplJittervalue, since indicates the execution jitter in CPU cycles experienced by the benchmark.$ sudo /opt/benchmarking/caterpillar/start-benchmark.py
Now, start the benchmark with Cache Allocation Technology. The pqos-helper tool is being used here to partition the CPU cache and isolate the cache used by the benchmark. Note how only core 3 is assigned to Class of Service (COS) 1, and the only task executing on core 3 will be the benchmark. This cache configuration should reduce cache evictions experienced by the benchmark caused by the “noisy neighbor”, resulting in reduced execution jitter.
$ sudo /opt/pqos/pqos-helper.py --cos0 0x0f --cos1 0xf0 --assign_cos "0=0 0=1 0=2 1=3" --pqos_rst --pqos_msr --command "/opt/benchmarking/caterpillar/start-benchmark.py"
Compare the results of the two runs. Ideally, the run with Cache Allocation Technology enabled should exhibit lower execution jitter.
Default parameters are used unless otherwise specified. You may run the script with --help to list the modifiable arguments.
See also
Sanity Check 3 - Caterpillar Workload provides an example on how to run this benchmark and display the results.
Interpret Caterpillar Results¶
SampleMin SampleMax SmplJitter SessionMin SessionMax SessionJitter Sample
254023 300255 9649 233516 303071 9743 200
The most important measurement is SessionJitter. This measurement describes the maximum execution jitter (in CPU cycles) during the entire execution of the benchmark. It is desired that SessionJitter be as low as possible. The delta between SessionMax and SessionMin is the overall execution spread. Ideally, the spread must be as low as possible.
RT-app¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
rt-app |
N/A |
|
RT-app is a tool that can be used to emulate a use case. Not only the sleep and run pattern can be emulated, but also the dependency between tasks like accessing same critical resources, creating sequential wake up, or syncing the wake up of threads. The use case is described in a JSON like file, which is first parsed by workgen and then rt-app.
Install RT-app¶
This component can be installed from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
$ sudo apt install rt-app-scripts
Execute RT-app¶
An example script that runs the RT-app tool is available at /opt/benchmarking/rt-app.
Rhealstone¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
Rhealstone |
nanoseconds |
commitID c9bdf48a |
https://gitlab.denx.de/Xenomai/xenomai/tree/master/testsuite/latency |
Rhealstone is a measurement targeted specifically toward true multitasking solutions. In this benchmark, five categories of activities crucial to the performance of real-time systems are represented:
Task switching time
Preemption time
Semaphore shuffling time
Interrupt latency time
Deadlock breaking time
Install Rhealstone¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
# For non-Xenomai kernels $ sudo apt install rhealstone # For Xenomai kernels $ sudo apt install rhealstone-xenomai
Execute Rhealstone¶
Two example scripts that run the Rhealstone benchmark are available at /opt/benchmarking/(rhealstone|rhealstone-xenomai). The script run_xlatency.py performs the following runtime optimizations before executing the benchmark latency:
Assigns benchmark thread affinity to core 0
Assigns non-benchmark thread affinity to core 1, as stress workloads.
Dumps histogram to <file> in a format easily readable with
gnuplotPrints statistics of minimum, average, and maximum latencies
Sets sampling period to 250 us
To start the benchmark, run the following script:
//without stress
$ cd /opt/benchmarking/rhealstone
$ sudo ./run_xlatency.py -T <runtime>
//with stress
$ cd /opt/benchmarking/rhealstone
$ sudo ./run_xlatency.py -T <runtime> --stress
//with stress exclude gfx for image without glxgears
$ cd /opt/benchmarking/rhealstone
$ sudo ./run_xlatency.py -T <runtime> --stress --no-gfx
Command line parameters:
runtime: Set test period in seconds.
Default parameters are used unless otherwise specified. Run the script with --help to list the modifiable arguments. Otherwise, run the latency benchmark directly with --help to list the modifiable arguments.
See also
Sanity Check 4 - Latency Workload provides an example on how to run this benchmark and display the results.
The script run_rhealstone_bmark.py executes benchmarks ctx_lat, deadlock_bt, preempt_lat, and semaphore_lat separately for <num> times and gets the average result for each.
To start the benchmark, run the following command:
$ cd /opt/benchmarking/rhealstone
$ sudo ./run_rhealstone_bmark.py <num>
See also
Sanity Check 7 - Rhealstone Workload provides an example on how to run this benchmark and display the results.
Interpret Rhealstone Results¶
The results obtained from run_xlatency.py are measured interrupt latency. Lower values are better.
...
#Avg: 0.683 us
#Max: 4.656 us
#Max list: [4.656]
...
The results obtained from run_rhealstone_bmark.py are measured time to perform: task switching, preemption, semaphore shuffling, and deadlock breaking. Lower values are better.
// Task Switching
#ctx_sum is 84878.48
#ctx_avg is 848.78
// Deadlock Breaking
#dead_sum is 225127.860003
#dead_avg is 2251.278600
// Preemption
#pree_sum is 114065.000000
#pree_avg is 1140.650000
// Semaphore Shuffling
#sem_sum is 208111.365000
#sem_avg is 2081.113650
MSI Latency¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
MSI Latency |
nanoseconds |
0.5.0-k |
Intel created |
Message Signaled Interrupt (MSI) Latency measures interrupt latency for MSIs generated by a discrete or integrated peripheral device, such as the Intel® I210 Ethernet Controller Series and serviced by the IA core. This benchmark measures the MSI interrupt triggered back latency of Intel® I210 Ethernet Controller Series kernel module.
Attention
This benchmark only functions in tandem with an Intel® I210 Ethernet Controller Series.
Install MSI Latency¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
$ sudo apt install msi-latency
Execute MSI Latency¶
An example script that runs the MSI latency benchmark is available at /opt/benchmarking/msi-latency. The script performs the following runtime optimizations before executing the benchmark:
Moves all IRQs (except timer and cascade) to core 0
Moves all rcu tasks to either core 0, 4, 5, 6, 7, except 1-3
Changes the real-time attributes of all rcu tasks to
SCHED_OTHERand priority 0Changes the real-time attributes of all tasks on core 1, 2 and 3 to
SCHED_OTHERand priority 0Changes the real-time attributes of all tasks to
SCHED_OTHERand priority 0
Required kernel boot parameters:
isolcpus=1,3 rcu_nocbs=1,3 nohz_full=1,3 igb.blacklist=yes
To start the benchmark, run the following commands:
$ cd /opt/benchmarking/msi-latency
$ sudo ./msiLatencyTest.sh <RUN_CORE> <IRQ_PERIOD(ms)> <IRQ_COUNT>
$ cat /sys/kernel/debug/msi_latency_test/current_value # Get current result
Default parameters are used when insert msi_lat.ko, unless otherwise specified. Command line parameters:
Parameter
Description
coreSpecIRQCore on which the
msi_latencytest IRQ handler is run. [0 to n]
coreSpecWQCore on which the
msi_latencytest work queue is run. [0 to n]
irqSpecRun IRQ as a pthread [0] or in legacy mode [1].
irqPeriodPeriod between interrupts in milliseconds.
irqCountNumber of IRQs to send. n > 0 will be finite. n < 0 will be infinite.
verbosityVerbosity of periodic prints to
dmesg. [0 to 3]
offsetStartOffset that is subtracted from each starting timestamp when calculating latency and generating histogram buckets (does not effect raw values). Mechanism used to remove delta time (nanosecond) from when SW requests interrupt to HW services interrupt.
blockIRQDuration (clock ticks) for which the interrupts are blocked after requesting MSI.
See also
Sanity Check 5 - MSI Latency provides an example on how to run this benchmark and display the results.
Interpret MSI Latency Results¶
************ RESULTS (ns) ************
[15996.755567] * Max: 9056
[15996.755570] * Avg: 6626
[15996.755573] * Min: 5360
The reported value is the measured time to service MSI requests to the Intel® I210 Ethernet Controller Series. Lower values are better.
MSI Jitter¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
MSI Jitter |
nanoseconds |
0.1.0-m |
Intel created |
MSI jitter benchmark tool is a benchmark tool to check the jitter of cyclical MSI from an Intel® I210 Ethernet Controller Series. This tool records each cyclical MSI timestamps in the software interrupt handler and calculates the delta between each timestamps and compares the delta with cycle time as the jitter.
Attention
This benchmark only functions in tandem with an Intel® I210 Ethernet Controller Series.
Install MSI Jitter¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
$ sudo apt install msi-jitter
Execute MSI Jitter¶
An example script that runs the MSI latency benchmark is available at /opt/benchmarking/msi-jitter. The script performs the following runtime optimizations before executing the benchmark:
Moves all IRQs (except timer and cascade) to core 0
Moves all rcu tasks to either core 0, 4, 5, 6, 7, except 1-3
$ sudo /opt/benchmarking/msi-jitter/irq_rcu.sh
To start the benchmark, run the following commands:
$ cd /opt/benchmarking/msi-jitter
$ sudo ./run_msijitter.sh <unbind_igb_id> <run_core> <interval(ms)> <runtime(s)>
$ cat /sys/kernel/debug/msi_jitter_test/current_value # Get current result
Command line parameters:
unbind_igb_id: Could get igb id need to unbind by `$lspci -v | grep 'Ethernet controller: Intel Corporation I210 Gigabit Network Connection'`.
run_core: Set which core will handle MSI interrupt.
interval: Set cycle time.
runtime: Set test period.
See also
Sanity Check 6 - MSI Jitter provides an example on how to run this benchmark and display the results.
Interpret MSI Jitter Results¶
************ RESULTS (ns) ************
[ 4177.021407] * Max: 3184
[ 4177.023363] * Avg: 222
[ 4177.025218] * Min: 0
The reported value is the measured jitter of servicing MSI requests to the Intel® I210 Ethernet Controller Series. Lower values are better.
MMIO Latency¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
MMIO Latency |
nanoseconds |
1.0 |
Intel created |
MMIO-latency is a simple driver that creates an affinitized thread to read a virtual map physical address (Memory-mapped I/O). Memory read latency is measured. The thread is created and started on initialization and loops LOOPS number of times. It also provides a char device that reads the current statistic counters, by using inline assembly and kernel function to get a very close benchmark.
Install MMIO Latency¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
$ sudo apt install mmio-latency
Execute MMIO Latency¶
An example script that runs the MSI latency benchmark is available at /opt/benchmarking/mmio-latency. The script performs the following runtime optimizations before executing the benchmark:
Moves all IRQs (except timer and cascade) to core 0
Moves all rcu task to either core 0, 4, 5, 6, 7, except 1-3
Changes realtime attributes of all rcu tasks to
SCHED_OTHERand priority 0Changes realtime attributes of all tasks on core 1, 2 and 3 to
SCHED_OTHERand priority 0Changes realtime attributes of all tasks to
SCHED_OTHERand priority 0
To start the benchmark, run the following command:
$ sudo /opt/benchmarking/mmio-latency/mmioLatency.sh <MMIO_address> <block_irq(1 or 0)>
Default parameters are used unless otherwise specified.
Find the physical mmio address to test using the command: lspci -vvv -s $BDF.
For example:
$ lspci -nn ==> 00:02.0 SATA controller
$ lspci -vvv -s 00:02.0 ==> Region 0: Memory at 80002000
See also
Sanity Check 8 - MMIO Latency Workload provides an example on how to run this benchmark and displays the results.
Interpret MMIO Latency Results¶
RT Core Module
------------------------
Stats:
max= 6472
avg= 2814
min= 588
total= 973791140
loops= 346010
mmio-outliers: 247
sched-outliers: 9
The reported value is the latency in reading memory-mapped I/O. Lower values are better.
Real-Time Performance Measurement (RTPM)¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
RTPM |
(See respective benchmark) |
1.9 |
Intel created |
About RTPM¶
Real-time computing is critical to industrial usage scenarios. Intel real-time solutions focus on hard real-time use cases where there could be capability failures if the solutions are not executed within the allotted time span. Examples of the usage scenarios include robotics, automotive, and so on.
In a real-time computing system, several factors could impact the latency of reaction to the trigger event. These factors include hardware design, BIOS configuration, OS kernel configuration, system settings, and so on. RTPM is designed to check the key settings of the system and help you to identify the hot-spot of the system for real-time performance and provide a recommendation based on the Best-known Configuration (BKC). In addition, RTPM provides a way to measure the system scheduling latency with some open source tools.
Install Real-Time Performance Measurement¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-realtime-benchmarking
- Install from individual Deb package
# For non-Xenomai kernels $ sudo apt install rtpm # For Xenomai kernels $ sudo apt install rtpm-xenomai
RTPM Test Modules¶
Real-Time Readiness Check¶
This module leverages one of the Intel® Time Coordinated Computing (Intel® TCC) Tools to check the many attributes that might affect real-time performance.
This module:
Verifies whether the system has a supported processor, BIOS, and OS
Checks for features, such as Intel® Turbo Boost Technology, Enhanced Intel SpeedStep® Technology, and processor power-saving states, that might affect real-time performance
Reports CPU and GPU frequencies
Operates at the OS level
Boot Command Line Check¶
This module checks the real-time OS boot command line parameters and recommends settings as per BKCs.
Interpreting Boot Command Line Check Results
=============================BOOT CMDLINE CHECK Mon Dec 1 00:00:00 UTC 2021=============================
----------------------------------------------------------------------------------------------------------------------------------------------------------
|CMDLINE ENTRY |CURRENT VALUE |BKC |
----------------------------------------------------------------------------------------------------------------------------------------------------------
|processor.max_cstate |0 |0 |
|intel_idle.max_cstate |0 |0 |
|clocksource |tsc |tsc |
|tsc |reliable |reliable |
|nmi_watchdog |0 |0 |
|nosoftlockup |nosoftlockup |nosoftlockup |
|intel_pstate |disable |disable |
|efi |runtime |runtime |
|nohalt |Missing |nohalt |
|nohz |Missing |nohz |
|irqaffinity |0 |0 |
|hugepages |Missing |1024 |
|cpufreq.off |Missing |1 |
|i915.enable_rc6 |Missing |0 |
|i915.enable_dc |Missing |0 |
|i915.disable_power_well |Missing |0 |
|mce |Missing |off |
|hpet |Missing |disable |
|numa_balancing |Missing |disable |
|nohz_full |1,3 |[xxxx] |
|isolcpus |1,3 |[xxxx] |
|rcu_nocbs |1,3 |[xxxx] |
----------------------------------------------------------------------------------------------------------------------------------------------------------
Boot cmdline check finished.
Kernel Configuration Check¶
This module checks the real-time OS kernel configuration and recommends settings as per the best-known configurations.
Interpret Kernel Configuration Check Results
=============================KERNEL CONFIGURATION CHECK Mon Dec 1 00:00:01 UTC 2021=============================
Kernel config file: /boot/config-5.4.115-rt57-intel-pk-standard+
----------------------------------------------------------------------------------------------------------------------------------------------------------
|KERNEL CONFIG ENTRY |CURRENT VALUE |BKC |
----------------------------------------------------------------------------------------------------------------------------------------------------------
|CONFIG_SMP |CONFIG_SMP=y |CONFIG_SMP=y |
|CONFIG_PREEMPT_RCU |CONFIG_PREEMPT_RCU=y |CONFIG_PREEMPT_RCU=y |
|CONFIG_GENERIC_IRQ_MIGRATION |CONFIG_GENERIC_IRQ_MIGRATION=y |CONFIG_GENERIC_IRQ_MIGRATION=y |
|CONFIG_EXPERT |CONFIG_EXPERT=y |CONFIG_EXPERT=y |
|CONFIG_PCIE_PTM |CONFIG_PCIE_PTM=y |CONFIG_PCIE_PTM=y |
|CONFIG_EFI |CONFIG_EFI=y |CONFIG_EFI=y |
|CONFIG_HIGH_RES_TIMERS |CONFIG_HIGH_RES_TIMERS=y |CONFIG_HIGH_RES_TIMERS=y |
|CONFIG_RCU_NOCB_CPU |CONFIG_RCU_NOCB_CPU=y |CONFIG_RCU_NOCB_CPU=y |
|CONFIG_HUGETLBFS |CONFIG_HUGETLBFS=y |CONFIG_HUGETLBFS=y |
|CONFIG_SCHED_MC_PRIO |Missing |CONFIG_SCHED_MC_PRIO=n |
|CONFIG_PREEMPT_RT |# CONFIG_PREEMPT_RT is not set |CONFIG_PREEMPT_RT=y |
|CONFIG_CPU_FREQ |# CONFIG_CPU_FREQ is not set |CONFIG_CPU_FREQ=n |
|CONFIG_CPU_ISOLATION |CONFIG_CPU_ISOLATION=y |CONFIG_CPU_ISOLATION=y |
|CONFIG_MIGRATION |CONFIG_MIGRATION=y |CONFIG_MIGRATION=y |
|CONFIG_PCIEPORTBUS |CONFIG_PCIEPORTBUS=y |CONFIG_PCIEPORTBUS=y |
----------------------------------------------------------------------------------------------------------------------------------------------------------
Kernel configuration check finished.
Real-Time Performance Test¶
This module contains the following benchmarks to evaluate the performance of the target system:
Xenomai Latency Test
Interpret Real-Time Performance Test Results
Click to toggle gnuplot script
=============================Real-Time PERFORMANCE TEST Mon Dec 1 00:00:02 UTC 2021=============================
Real-time task CPU affinity is set to core 1 by default. Stress is added to other CPU cores if enabled. It's recommended to isolate the core 1 for real-time task from others.
To change the default test duration. Please modify test_cfg file.
Calculating Stress workload running duration...
Setup stress on core 0
Setup stress-ng on core 2
Setup hackbench on core 3
>>> Processing latency...Test duration: 21600s
>>> Latency Test Command: taskset -c 1 /opt/benchmarking/rhealstone/latency -c 1 -p 250 -T 21600 -s -g /tmp/rtpm/latency_test_1643673592.log
# 06:00:00 (periodic user-mode task, 250 us period, priority 99)
# ----lat min|----lat avg|----lat max|-overrun|---msw|
# 7.924| 9.728| 60.457| 0| 0|
7 1
7.5 72
8.5 10063785
9.5 49405401
10.5 22821895
11.5 1447656
...
>>> Processing cyclictest...Test duration: 21600s
>>> Current cyclictest version: cyclictest V 1.00
>>> Cyclictest Test Command: taskset -c 1 cyclictest -p 99 -i 250 -m -a 1 -N -o 3970 -v -r -q -n -D 21600
Max CPUs = 4
# /dev/cpu_dma_latency set to 0us
Thread 0 Interval: 750
Thread 0 using cpu 1.
0: 0: 16319
0: 4449: 16103
0: 10551: 9183
0: 13394: 15544
...
T: 0 (1181838) P:99 I:250 C:85141567 Min: 2910 Act: 3675 Avg: 3627 Max: 20109
>>> Processing msi_latency test...Test duration: 60s
Msi-latency test finish, max:3936 ns, avg:2682 ns, min:2464 ns
>>> Processing msi_jitter test...Test duration: 60s
Msi-jitter test finish, max:1552 ns, avg:304 ns, min:0 ns
>>> Processing rhealstone...Test cycles: 2
>>> Rhealstone Test Command: Cycle 'taskset -c 1 /opt/benchmarking/rhealstone/ctx_lat' 2 times and calcuate the average value
>>> Rhealstone Test Command: Cycle 'taskset -c 1 /opt/benchmarking/rhealstone/deadlock_bt' 2 times and calcuate the average value
>>> Rhealstone Test Command: Cycle 'taskset -c 1 /opt/benchmarking/rhealstone/preempt_lat' 2 times and calcuate the average value
>>> Rhealstone Test Command: Cycle 'taskset -c 1 /opt/benchmarking/rhealstone/semaphore_lat' 2 times and calcuate the average value
[1777.43, 1753.55]
[1777.43, 1753.55]
ctx_sum is 3530.98
ctx_avg is 1765.49
[6895.52, 6299.76]
[6895.52, 6299.76]
dead_sum is 13195.280000
dead_avg is 6597.640000
[3626.61, 5100.095]
[3626.61, 5100.095]
sem_sum is 8726.705000
sem_avg is 4363.352500
[6762.415, 6823.44]
[6762.415, 6823.44]
pree_sum is 13585.855000
pree_avg is 6792.927500
>>> Processing mmio_latency test...Test memory region: 7fe00000 on Device: 01:00.0, Test duration: 300s
MMIO-Latency test finish, max:14698 ns, avg:4871 ns, min:4186 ns
>>> Processing Caterpillar test...
>>> Caterpillar Command: /opt/benchmarking/caterpillar/caterpillar -c 1 -n 10000 -i 50000 -s 200
Cache Allocation Technology (CAT)erpillar benchmark v1.1
The benchmark will execute an operation function 50000 times.
Samples are taken every 10000 test cycles.
A total of 200 samples will be measured.
Thread affinity will be set to core_id: 1
Timings are in CPU Core cycles
SampleMin: Minimum execution time during the display update interval
SampleMax: Maximum execution time during the display update interval
SmplJitter: Jitter in the execution time during the display update interval
SessionMin: Minimum execution time since the program started or statistics were reset
SessionMax: Maximum execution time since the program started or statistics were reset
SessionJitter: Jitter in the execution since the program started or statistics were reset
Sample: Sample number
Priming cache...
SampleMin SampleMax SmplJitter SessionMin SessionMax SessionJitter Sample
152506 156402 557 152506 156402 557 0
152429 156426 551 152429 156426 554 1
152180 156335 561 152180 156426 556 2
152486 156420 553 152180 156426 556 3
152278 156600 554 152180 156600 555 4
152750 156294 547 152180 156600 554 5
152580 156199 588 152180 156600 559 6
...
>>> Caterpillar test finish, Max session jitter: 584 CPU Cycles
>>> Processing Jitter test...
>>> Jitter Command: /opt/benchmarking/jitter/jitter -c 1 -r 20000 -l 80000 -i 200
Linux Jitter testing program version 1.9
Iterations=200
The pragram will execute a dummy function 80000 times
Display is updated every 20000 displayUpdate intervals
Thread affinity will be set to core_id:1
Timings are in CPU Core cycles
Inst_Min: Minimum Excution time during the display update interval(default is ~1 second)
Inst_Max: Maximum Excution time during the display update interval(default is ~1 second)
Inst_jitter: Jitter in the Excution time during rhe display update interval. This is the value of interest
last_Exec: The Excution time of last iteration just before the display update
Abs_Min: Absolute Minimum Excution time since the program started or statistics were reset
Abs_Max: Absolute Maximum Excution time since the program started or statistics were reset
tmp: Cumulative value calcualted by the dummy function
Interval: Time interval between the display updates in Core Cycles
Sample No: Sample number
Inst_Min Inst_Max Inst_jitter last_Exec Abs_min Abs_max tmp Interval Sample No
158807 158939 132 158809 158807 158939 3158835200 3179339706 1 132
158807 158944 137 158852 158807 158944 2370306048 3179421895 2 137
158807 158938 131 158860 158807 158944 1581776896 3179361092 3 137
158807 158940 133 158840 158807 158944 793247744 3179379386 4 137
158807 158944 137 158843 158807 158944 4718592 3179371249 5 137
...
>>> Jitter test finish, Max inst jitter: 12772 CPU Cycles
>>> Processing LMbench test...
>>> LMbench Command: taskset -c 1 /usr/bin/lat_mem_rd -P 1 128 512
"stride=512
0.00049 1.590
0.00098 1.590
0.00195 1.590
0.00293 1.590
0.00391 1.590
0.00586 1.590
0.00781 1.590
...
>>> LMbench test finish, 1M memory read latency: 13.655 ns
Real-time performance test finished.
To start the benchmark, run the following commands:
$ sudo /opt/benchmarking/rtpm/rtpm_exec -p "./output" -n "report.csv" -a
The results are logged in: ./output/rtpm_test.log
RTPM Command Line Parameters¶
Parameter |
Explanation |
|---|---|
|
Specify the path of the test file |
|
Specify the test report name in CSV format (must end with .csv) |
|
Execute Latency or Cyclictest test when running performance test |
|
Execute Rhealstone test when running performance test |
|
Execute MSI-latency test when running performance test |
|
Execute MSI-jitter test when running performance test |
|
Execute MMIO-Latency test when running performance test |
|
Execute Caterpillar test when running performance test. |
|
Execute Jitter test when running performance test. |
|
Execute LMbench test when running performance test. |
|
Enable stress when running performance test |
|
Execute all performance tests except Latency & Cyclictest, the program will help to identify current Real-Time framework, if it is Xenomai, will run Latency otherwise if it is Preempt-RT, it will run Cyclictest. If not specified, settings in configuration file will be used. |
|
Show the help list |
For more information on using Intel ECI with RTPM, refer to the guide.
CODESYS Benchmarking Applications¶
Benchmark |
Units |
Version |
Source |
Description |
|---|---|---|---|---|
microseconds |
1.4 |
Intel created |
The CODESYS Benchmark is an example CODESYS Web-UI, OPC-UA and SoftPLC logic application that allows remote-user access to cyclic control of compute and IO tasks. |
|
microseconds |
1.4 |
Intel created |
The CODESYS ST-Fragment is an example application that allows configurable and observable CODESYS SoftPLC logic regression testing. |
6-axis EtherCAT Benchmark¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
6-axis EtherCAT benchmark |
microseconds |
0.1.0 |
Intel created |
About 6-axis EtherCAT Benchmark¶
The benchmark uses the IgH EtherCAT Master Stack and PLCopen Motion Control library (RTmotion) to drive a real-world 6-axis EtherCAT test bench (6 EtherCAT Servos, 8 EtherCAT IO) and measures the jitter and motion execution time of the control workload in each real-time cycle (1ms). The goal of this benchmark is to evaluate ECI real-time performance when driving EtherCAT devices with different non-real-time workloads.
Using different non-real-time noises, the benchmark is implemented and tested in three configurations to simulate real-world usage scenarios:
Install 6-axis EtherCAT Benchmark¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-softplc-plcopen
- Install from individual Deb package
# For non-Xenomai kernels $ sudo apt install plcopen-servo # For Xenomai kernels $ sudo apt install plcopen-servo-xenomai
Execute 6-axis EtherCAT Benchmark¶
To start the benchmark, run the following command:
# For running in 1ms cycle time
$ sudo /opt/plcopen/six_rtmotion_demo -n /opt/plcopen/inovance_six_1ms.xml -i 1000
# For running in 250us cycle time
$ sudo /opt/plcopen/six_rtmotion_demo -n /opt/plcopen/inovance_six_250us.xml -i 250
Interpret 6-axis EtherCAT Benchmark Results¶
Using /opt/plcopen/EtherCAT_6xInov_ENI.xml
MOTION_ENTRY: Requesting master index 0:...
MOTION_ENTRY: Creating domain...
MOTION_ENTRY: Register servo driver...
Activating master...
Axis #0 powered on, axis_pos_ = -36.065203
Axis #1 powered on, axis_pos_ = 50.606118
Axis #2 powered on, axis_pos_ = 129.662308
Axis #3 powered on, axis_pos_ = 141.972963
Axis #4 powered on, axis_pos_ = -61.609828
Axis #5 powered on, axis_pos_ = -61.655695
^C******************************************************
jitter time 2.220... 2.689... 9.867
motion time 0.000... 7.347... 51.513
******************************************************
End of Program
Press crtl+c to stop the real-time PLC program. The jitter time shows the [minimum, average, maximum] values of real-time cycle jitter. The motion time shows the [minimum, average, maximum] execution time of 6-axis PLCopen motion calculation.
Robotic 6-axis Motion Control Benchmark¶
Benchmark |
Units |
Version |
Source |
|---|---|---|---|
Robotic 6-axis Motion Control Benchmark |
microseconds |
0.1.0 |
Intel created |
About Robotic 6-axis Motion Control Benchmark¶
The benchmark uses the PLCopen Motion Control library (RTmotion) and ROS2 MoveIt2 to generate online trajectories for an industrial robotic arm in simulation and measures the jitter and motion execution time of the control workload in each real-time cycle. The goal of this benchmark is to evaluate ECI real-time performance when running ROS2 non-real-time workloads (Path Planning, 3D Visualization, DDS communication). The trajectory of the robotic arm is generated by ROS2 MoveIt2 and contains waypoints, each of which contains the joint [position, velocity, acceleration] targets and a timestamp. The real-time program received the trajectory from MoveIt 2 through the shared memory, and uses PLCopen function blocks to interpolates path between waypoints with 1ms cycle time and interprets each waypoint into servo motor commands.
Install Robotic 6-axis Motion Control Benchmark¶
You can install this component from the ECI APT repository. Setup the ECI APT repository, then perform either of the following commands to install this component:
- Install from meta-package
$ sudo apt install eci-softplc-plcopen eci-robotics eci-realtime
- Install from individual Deb package
# For non-Xenomai kernels $ sudo apt install plcopen-databus libshmringbuf-dev libshmringbuf mc-gateway-hiwin # For Xenomai kernels $ sudo apt install plcopen-databus-xenomai libshmringbuf-dev libshmringbuf mc-gateway-hiwin
Execute Robotic 6-axis Motion Control Benchmark¶
To start the benchmark, follow the instructions in Sanity Check 6: HIWIN 6-axis Robotic Arm Demo Launch Check of Robot Operating System Software.
Interpret Robotic 6-axis Motion Control Benchmark¶
Axis 0 initialized.
Axis 1 initialized.
Axis 2 initialized.
Axis 3 initialized.
Axis 4 initialized.
Axis 5 initialized.
Function blocks initialized.
Axis #0 powered on, axis_pos_ = 0.000000
Axis #1 powered on, axis_pos_ = 0.000000
Axis #2 powered on, axis_pos_ = 0.000000
Axis #3 powered on, axis_pos_ = 0.000000
Axis #4 powered on, axis_pos_ = 0.000000
Axis #5 powered on, axis_pos_ = 0.000000
^Ccount: 68s
******************************************************
jitter time 4.924... 7.446... 24.842
motion time 0.895... 2.822... 55.255
******************************************************
End of Program
Press crtl+c to stop the real-time PLC program. The jitter time shows the [minimum, average, maximum] values of real-time cycle jitter. The motion time shows the [minimum, average, maximum] execution time of 6-axis PLCopen motion calculation.
Ethernet TSN Intel® Reference Benchmark¶
Benchmark |
Units |
Version |
Source |
Description |
|---|---|---|---|---|
microseconds |
0.8.27 |
Tool to check Ethernet endpoints ingress and egress Traffic-class QoS under IEEE 802.1Q-2018 Enhancements for Scheduled Traffic (EST) polices with various Intel® Ethernet Controller TSN hw-offloading capabilities |
Performance Sanity Check Testing¶
For step-by-step instructions on using the various benchmarks, refer to Performance Sanity Check Testing.