RTC Testbench - OPCUA over TSN

Benchmark

Units

Version

Source

Linutronix RTC-Testbench

microseconds

5.2

Linutronix

RTC-Testbench Architecture

Program Setup

Figure 1: RTC-Testbench Architecture

RTC-Testbench Prerequisites

Install Linutronix RTC Testbench

You can install this component from the ECI repository. Setup the ECI repository, then perform either of the following commands to install this component:

logo_red-hat

Install from individual RPM package

$ sudo dnf install rtc-testbench

Kernel Boot Parameters Configuration

The RTC-Testbench requires a real-time kernel with specific kernel boot parameters.

After Installing a Real-time Linux Kernel, verify the kernel boot parameters are correct:

logo_red-hat

$ cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-570.41.1.el9_6.x86_64+rt root=/dev/mapper/rhel-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet hpet=disable clocksource=tsc tsc=reliable intel_pstate=disable intel_idle.max_cstate=0 intel.max_cstate=0 processor.max_cstate=0 processor_idle.max_cstate=0 rcupdate.rcu_cpu_stall_suppress=1 mce=off nmi_watchdog=0 nosoftlockup noht numa_balancing=disable hugepages=1024 rcu_nocb_poll audit=0 irqaffinity=0 isolcpus=1-3 rcu_nocbs=1-3 nohz_full=1-3 i915.enable_dc=0 i915.disable_power_well=0

RTC-Testbench Overview

Physical Setup

Figure 2: Physical Setup

The following example assumes a peer-to-peer setup between two TSN capable devices as shown in Figure 1. We will use systemd services from the tsn-endstation-netlink rpm provided by the ECI DNF repository for establishing PTP time synchronization. We will then install the rtc-testbench rpm and modify the scripts and configuration files to generate OPCUA traffic over the established TSN connection.

RTC-Testbench Configuration

  1. PTP Configuration

    Please see gPTP Time Synchronization Service for Intel® Ethernet Controllers for PTP configuration on Linux Machines 1 & 2:

    1. See systemd Network Manager TSN Endstation Configuration for Intel® Ethernet Controllers to install the TSN systemd services provided by ECI.

    2. See eci-ethirqs@.service- Networking Interrupts CPU Affinity for Intel® Ethernet Controllers to use the systemd service to configure network interrupt affinity to an isolated CPU.

      For example, assuming the TSN interface is enp1s0:

      $ systemctl enable eci-ethirqs@enp1s0.service && systemctl start eci-ethirqs@enp1s0.service && systemctl status eci-ethirqs@enp1s0.service

    3. See eci-ptp4l-p2p@.service - gPTP Time Synchronization Service for Intel® Ethernet Controllers to use the systemd service for PTP on the TSN interface and ensure time synchronization for both Linux Machine 1 & 2.

      For example, assuming the TSN interface is enp1s0:

      $ systemctl enable eci-ptp4l-p2p@enp1s0.service && systemctl start eci-ptp4l-p2p@enp1s0.service && systemctl status eci-ptp4l-p2p@enp1s0.service

    4. See eci-phc2sys@.service - PHC Time Synchronization Service for Intel® Ethernet Controllers to use the systemd service for PHC2SYS on the TSN interface and ensure time synchronization for both Linux Machine 1 & 2.

      For example, assuming the TSN interface is enp1s0:

      $ systemctl enable eci-phc2sys@enp1s0.service && systemctl start eci-phc2sys@enp1s0.service && systemctl status eci-phc2sys@enp1s0.service

    5. See eci-tsn-wait-sync-8021as@.service - TSN 802.1AS Services for Intel® Ethernet Controllers to use the systemd service for check_clocks to verify clock synchronization.

      For example, assuming the TSN interface is enp1s0:

      $ systemctl enable eci-tsn-wait-sync-8021as@enp1s0.service && systemctl start eci-tsn-wait-sync-8021as@enp1s0.service && systemctl status eci-tsn-wait-sync-8021as@enp1s0.service

  2. Update rtc-testbench scripts

    1. After rtc-testbench is installed and PTP is configured on both machines, navigate to /usr/share/rtc-testbench/tests/opcua/ on both systems.

      $ cd /usr/share/rtc-testbench/tests/opcua/

    2. On Linux Machine 2, update mirror.sh to point to your system TSN interface and comment out the PTP script commands, update the copy and run commands:

      set -e

cd "$(dirname "$0")"

# Start PTP
#../../scripts/ptp.sh enp5s0 # Handled by ECI eci-ptp4l-p2p systemd service
#sleep 30

# Configure flow
./flow_cml.sh enp5s0
sleep 30

# Start one instance of mirror application
cp /usr/lib64/rtc-testbench/ebpf/xdp_kern_*.o .
taskset -c 3 /usr/bin/mirror -c mirror_opcua_vid200_cml.yaml >mirror1.log &

exit 0

    3. On Linux Machine 1, update ref.sh to point to your system TSN interface and comment out the PTP script commands, update the copy and run commands:

      set -e

cd "$(dirname "$0")"

# Start PTP
#../../scripts/ptp.sh enp5s0 # Handled by ECI eci-ptp4l-p2p systemd service
#sleep 30

# Configure flow
./flow_cml.sh enp5s0
sleep 30

# Start one instance of reference application
cp /usr/lib64/rtc-testbench/ebpf/xdp_kern_*.o .
taskset -c 3 /usr/bin/reference -c reference_opcua_vid200_cml.yaml >ref1.log &

exit 0

  3. Update yaml files: reference_opcua_vid200_cml.yaml on Linux Machine 1 and mirror_opcua_vid200_cml.yaml on Linux Machine 2:

    1. Make sure GenericL2Interface is pointing to the local TSN capable interface on both Linux Machine 1 & 2

    2. Change GenericL2Destination to point to the MAC Address of the opposite system’s TSN interface (for example see ip a output on Linux Machine 1 and update mirror_opcua_vid200_cml.yaml on Linux Machine 2)

    YAML Configuration

    YAML Configuration Example

Execute RTC-Testbench

  1. Navigate to /usr/share/rtc-testbench/tests/opcua/ on both systems:

    $ cd /usr/share/rtc-testbench/tests/opcua/

  2. Run mirror.sh on Linux Machine 2

    $ ./mirror.sh

  3. Run ref.sh on Linux Machine 1

    $ ./ref.sh

  4. Monitor output - View logs on Linux Machine 1 to verify OPCUA RTT statistics

    $ tail -f /var/log/reference_opcua_vid200.log

    1. Ensure OpcUaRttOutliers=0 and OpcUaOnewayOutliers=0 consistently when running with the TSN configuration. OpcUaRttAvg should also be <2000 us when running with the default cycle time of 1000 us even with increased network traffic.

    2. See https://linutronix.github.io/RTC-Testbench/test.html for additional details

RTC-Testbench Sample Logs

[1760567735. 98328147]: [INFO]: OpcUaSent=672416 | OpcUaReceived=672352 | OpcUaRttMin=1494 [us] | OpcUaRttMax=1506 [us] | OpcUaRttAvg=1499.639338 [us] | OpcUaOnewayMin=478 [us] | OpcUaOnewayMax=500 [us] | OpcUaOnewayAvg=491.236150 [us] | OpcUaRttOutliers=0 | OpcUaOnewayOutliers=0 |
[1760567736. 98327985]: [INFO]: OpcUaSent=704416 | OpcUaReceived=704352 | OpcUaRttMin=1494 [us] | OpcUaRttMax=1506 [us] | OpcUaRttAvg=1499.639149 [us] | OpcUaOnewayMin=478 [us] | OpcUaOnewayMax=500 [us] | OpcUaOnewayAvg=491.234555 [us] | OpcUaRttOutliers=0 | OpcUaOnewayOutliers=0 |
[1760567737. 98328189]: [INFO]: OpcUaSent=736448 | OpcUaReceived=736384 | OpcUaRttMin=1494 [us] | OpcUaRttMax=1506 [us] | OpcUaRttAvg=1499.638210 [us] | OpcUaOnewayMin=478 [us] | OpcUaOnewayMax=500 [us] | OpcUaOnewayAvg=491.225099 [us] | OpcUaRttOutliers=0 | OpcUaOnewayOutliers=0 |
[1760567738. 98328117]: [INFO]: OpcUaSent=768448 | OpcUaReceived=768384 | OpcUaRttMin=1494 [us] | OpcUaRttMax=1506 [us] | OpcUaRttAvg=1499.638148 [us] | OpcUaOnewayMin=478 [us] | OpcUaOnewayMax=500 [us] | OpcUaOnewayAvg=491.218115 [us] | OpcUaRttOutliers=0 | OpcUaOnewayOutliers=0 |
[1760567739. 98327813]: [INFO]: OpcUaSent=800480 | OpcUaReceived=800416 | OpcUaRttMin=1494 [us] | OpcUaRttMax=1506 [us] | OpcUaRttAvg=1499.637560 [us] | OpcUaOnewayMin=478 [us] | OpcUaOnewayMax=500 [us] | OpcUaOnewayAvg=491.216827 [us] | OpcUaRttOutliers=0 | OpcUaOnewayOutliers=0 |
[1760567740. 98328153]: [INFO]: OpcUaSent=832512 | OpcUaReceived=832448 | OpcUaRttMin=1494 [us] | OpcUaRttMax=1506 [us] | OpcUaRttAvg=1499.637955 [us] | OpcUaOnewayMin=478 [us] | OpcUaOnewayMax=500 [us] | OpcUaOnewayAvg=491.217131 [us] | OpcUaRttOutliers=0 | OpcUaOnewayOutliers=0 |
[1760567741. 98328322]: [INFO]: OpcUaSent=864512 | OpcUaReceived=864448 | OpcUaRttMin=1494 [us] | OpcUaRttMax=1506 [us] | OpcUaRttAvg=1499.636525 [us] | OpcUaOnewayMin=478 [us] | OpcUaOnewayMax=500 [us] | OpcUaOnewayAvg=491.214683 [us] | OpcUaRttOutliers=0 | OpcUaOnewayOutliers=0 |
[1760567742. 98328189]: [INFO]: OpcUaSent=896544 | OpcUaReceived=896480 | OpcUaRttMin=1494 [us] | OpcUaRttMax=1506 [us] | OpcUaRttAvg=1499.636416 [us] | OpcUaOnewayMin=478 [us] | OpcUaOnewayMax=500 [us] | OpcUaOnewayAvg=491.213534 [us] | OpcUaRttOutliers=0 | OpcUaOnewayOutliers=0 |