Attention

You are viewing an older version of the documentation. The latest version is v3.3.

open62541 OPC-UA Pubsub performance testing framework¶

open62541 (http://open62541.org) is an open-source and free implementation of OPC UA (OPC Unified Architecture) written in the common subset of the C99 and C++98 languages. The library is usable with all major compilers and provides the necessary tools to implement dedicated OPC UA clients and servers, or to integrate OPC UA-based communication into existing applications. open62541 library is platform independent. All platform-specific functionality is implemented via exchangeable plugins. Plugin implementations are provided for the major operating systems.

open62541 is licensed under the Mozilla Public License v2.0 (MPLv2). This allows the open62541 library to be combined and distributed with any proprietary software. Only changes to the open62541 library itself need to be licensed under the MPLv2 when copied and distributed. The plugins, as well as the server and client examples are in the public domain (CC0 license). They can be reused under any license and changes do not have to be published.

Intel® has joined the Open Source Automation Development Lab (https://www.osadl.org/), aka. OSADL, with the objective to obtain interoperable connectivity. It was proposed some time ago to equip the already existing OPC UA protocol with additional features needed by the industry and establish it as an unified communication protocol of the future.

For more information, please review: Project phase #2

Additional References:

All ECI images builds a specific-version of open62541.org project official examples and QA-testsuite [fa5ed7a] for ECI sanity-checking against major release.

Certain /usr/share/open62541/examples aim more specifically to publish and subscribe IEC-62541 OPC-UA (part4) information in IEEE 802.1Q-2018 EST and IEEE 802.1AS governed OT networks.

./pubsub_TSN_publisher and ./pubsub_TSN_loopback example are TSN-Endpoint performance testing with Talker, Listener and UserApplication rt-thread (e.g. SCHED_FIFO and priority-based pthread), affinitized to given cpu core:

The UserApplication rt-thread is used to shim the functionality of the IEC-61131-3 control loop program. For the sake of simplicity, the UserApplication rt-threads simply increments the counterData, which is published by the Talker rt-thread and also reads the subscribed data from the Information Model and writes the updated counterData into distinct CSV files during each cycle. The user application thread will wakeup at 30% of cycle time and handles the user data, read the data from Information Model for the Listener thread and writes the updated counterData in distinct csv files for both rt-threads.

The Listener rt-thread ./pubsub_TSN_loopback runtime application subscribes to the counterData published by the Talker rt-thread of ./pubsub_TSN_publisher runtime application.

The Talker rt-thread of ./pubsub_TSN_loopback runtime application publishes counterData (an incremental data), that is subscribed by the Listener rt-thread of ./pubsub_TSN_publisher runtime application.

In a Linux preemptive scheduling OS runtime, the round-trip time of the counterData is four-time (4x) IEEE 802.1Q-2018 EST network cycletime -cycleTimeInMsec, for example a round-trip time is 1ms.

Once the message is received, the Subscriber thread updates the value in the Information Model and again waits for the next message.

This process is repeated until the application is terminated using Ctrl+C

The flow of this communication and the trace points are given in the diagram below.

/**
 * **Trace point setup**
 *
 *            +--------------+                        +----------------+
 *         T1 | OPCUA PubSub |  T8                 T5 | OPCUA loopback |  T4
 *         |  |  Application |  ^                  |  |  Application   |  ^
 *         |  +--------------+  |                  |  +----------------+  |
 *  User   |  |              |  |                  |  |                |  |
 *  Space  |  |              |  |                  |  |                |  |
 *         |  |              |  |                  |  |                |  |
 *------------|--------------|------------------------|----------------|--------
 *         |  |    Node 1    |  |                  |  |     Node 2     |  |
 *  Kernel |  |              |  |                  |  |                |  |
 *  Space  |  |              |  |                  |  |                |  |
 *         |  |              |  |                  |  |                |  |
 *         v  +--------------+  |                  v  +----------------+  |
 *         T2 |  TX tcpdump  |  T7<----------------T6 |   RX tcpdump   |  T3
 *         |  +--------------+                        +----------------+  ^
 *         |                                                              |
 *         ----------------------------------------------------------------
 */

Additionally,csv files can be written for Talker and Listener rt-thread.

These csv files can be used with -enableCsvLog to compute latency for following combinations of Tracepoints, T1-T4 and T1-T8.

CSV files format include the counterData that is being either Talker and Listener along with the epoch timestamp. T1-T8 - Gives the Round-trip time of a counterData, as the value published by the Talker rt-thread of ./pubsub_TSN_publisher runtime application is subscribed by the Listener rt-thread ./pubsub_TSN_loopback runtime application and is published back to the ./pubsub_TSN_publisher runtime application.

Another additional feature called the Blocking Socket is employed in the Listener rt-thread. This feature is optional and can be enabled or disabled when running application by using command line argument -enableBlockingSocket. When using Blocking Socket,the Talker and Listener rt-threads are scheduled to run on different cpu cores, and the Listener rt-threads remains in “blocking mode” until a message is received, in other words the timeout is overwritten and the rt-thread continuously waits for the message.

To ensure IEEE 802.1Q-2018 Enhanced-Scheduled Traffic (EST), Talker rt-thread uses Earliest TxTime First (NET_SCHED_ETF) QDisc to publish information at the calculated transmission absolute time windows (set with -qbvOffset in [us] on CLOCK_TAI managed Gate Control List) and Listener rt-thread can be used Linux eXpress Data Path (XDP) to achieve guaranteed-deadline of 802.1Q VLAN PCP (-socketPriority) tagged L2-Ethernet frames.

transmission_time = ((UA_UInt64)nextnanosleeptime.tv_sec * SECONDS + (UA_UInt64)nextnanosleeptime.tv_nsec) + roundOffCycleTime + (UA_UInt64)(qbvOffset * 1000);
ethernettransportSettings.transmission_time = transmission_time;

Below of options provided by example applications :

/usr/share/open62541/examples# ./pubsub_TSN_loopback --help

usage: pubsub_TSN_loopback [options]

 -interface       [name] Use network interface 'name'
 -cycleTimeInMsec [num]  Cycle time in milli seconds (default 0.250000)
 -socketPriority  [num]  Set publisher SO_PRIORITY to (default 3)
 -pubPriority     [num]  Publisher thread priority value (default 78)
 -subPriority     [num]  Subscriber thread priority value (default 81)
 -userAppPriority [num]  User application thread priority value (default 75)
 -pubCore         [num]  Run on CPU for publisher (default 2)
 -subCore         [num]  Run on CPU for subscriber (default 2)
 -userAppCore     [num]  Run on CPU for userApplication (default 3)
 -pubMacAddress   [name] Publisher Mac address (default opc.eth://01-00-5E-00-00-01:8.3 - where 8 is the VLAN ID and 3 is the PCP)
 -subMacAddress   [name] Subscriber Mac address (default opc.eth://01-00-5E-7F-00-01:8.3 - where 8 is the VLAN ID and 3 is the PCP)
 -qbvOffset       [num]  QBV offset value (default 125)
 -disableSoTxtime        Do not use SO_TXTIME
 -enableCsvLog           Experimental: To log the data in csv files. Support up to 1 million samples
 -enableconsolePrint     Experimental: To print the data in console output. Support for higher cycle time
 -enableBlockingSocket   Run application with blocking socket option. While using blocking socket option need to
                         run both the Publisher and Loopback application. Otherwise application will not terminate.
 -enableXdpSubscribe     Enable XDP feature for subscriber. XDP_COPY and XDP_FLAGS_SKB_MODE is used by default. Not recommended to be enabled along with blocking socket.
 -xdpQueue        [num]  XDP queue value (default 2)
 -xdpFlagDrvMode         Use XDP in DRV mode

/usr/share/open62541/examples# ./pubsub_TSN_publisher --help

usage: pubsub_TSN_publisher [options]

 -interface       [name] Use network interface 'name'
 -cycleTimeInMsec [num]  Cycle time in milli seconds (default 0.250000)
 -socketPriority  [num]  Set publisher SO_PRIORITY to (default 3)
 -pubPriority     [num]  Publisher thread priority value (default 78)
 -subPriority     [num]  Subscriber thread priority value (default 81)
 -userAppPriority [num]  User application thread priority value (default 75)
 -pubCore         [num]  Run on CPU for publisher (default 2)
 -subCore         [num]  Run on CPU for subscriber (default 2)
 -userAppCore     [num]  Run on CPU for userApplication (default 3)
 -pubMacAddress   [name] Publisher Mac address (default opc.eth://01-00-5E-7F-00-01:8.3 - where 8 is the VLAN ID and 3 is the PCP)
 -subMacAddress   [name] Subscriber Mac address (default opc.eth://01-00-5E-00-00-01:8.3 - where 8 is the VLAN ID and 3 is the PCP)
 -qbvOffset       [num]  QBV offset value (default 125)
 -disableSoTxtime        Do not use SO_TXTIME
 -enableCsvLog           Experimental: To log the data in csv files. Support up to 1 million samples
 -enableLatencyCsvLog    Experimental: To compute and create RTT latency csv. Support up to 1 million samples
 -enableconsolePrint     Experimental: To print the data in console output. Support for higher cycle time
 -enableBlockingSocket   Run application with blocking socket option. While using blocking socket option need to
                         run both the Publisher and Loopback application. Otherwise application will not terminate.
 -enableXdpSubscribe     Enable XDP feature for subscriber. XDP_COPY and XDP_FLAGS_SKB_MODE is used by default. Not recommended to be enabled along with blocking socket.
 -xdpQueue        [num]  XDP queue value (default 2)
 -xdpFlagDrvMode         Use XDP in DRV mode
 -xdpBindFlagZeroCopy    Use Zero-Copy mode in XDP

open62541 Sanity-Check Testing¶

open62541 Reference open62541/examples/pubsub_realtime/README_pubsub_TSN.txt [fa5ed7a] has been used to setup ECI nodes Sanity-Check.

open62541 pre-conditions applicable to:

Check if 8021q is rightfully loaded
# lsmod | grep '8021q'
Check in all ECI nodes that ONLY ETF qdisc egress TBS - Time Base Scheduling shaper (aka launchtime) hw-offload are defined with ONLY mqprio as root (NO TAPRIO qdisc)

It is strongly recommended to apply countermeasures to avoid Common PTP sync issues.

Tip

It is highly recommended to clean previous tc qdisc and tc filter (or ethtool flow-type) setting before any new egress traffic shaping or ingress traffic steering rules.

tc qdisc del dev $IFACE parent root
tc qdisc del dev $IFACE parent ffff:
tc filter del dev $IFACE parent ffff:
ethtool -U $IFACE delete $ID
ip addr flush dev $IFACE
ip neigh flush all dev $IFACE
ip link delete dev $IFACE.vlan

Important

Intel Ethernet Controller I210 [Springville] Traffic Shaping hw-offload is ONLY available on hw-queues 0 and 1.

It is highly recommended to turn-off VLAN Stripping and set hw-queues combined queue, Minimum is 4 combined rx/tx.
ethtool -K enp1s0 rxvlan off hw-tc-offload on
ethtool --set-eee enp1s0 eee off
ethtool -L enp1s0 combined 4
ethtool -X enp1s0 equal 2
ethtool -G enp1s0  tx 1024 rx 1024

Application /usr/share/open62541/example

Linux Traffic Control (TC)

Comments

IEC-62541 250us cyclic Publisher with SCM_TXTIME

./pubsub_TSN_loopback -interface enp1s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback with SCM_TXTIME

./pubsub_TSN_publisher -interface enp1s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableCsvLog

802.1Q Egress packets Timed-Based Scheduler (TBS) on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

tc qdisc add dev enp1s0 parent root handle 100 mqprio \
num_tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
queues 1@0 1@1 2@2 hw 0

tc qdisc replace dev enp1s0 parent 100:1 etf  \
clockid CLOCK_TAI delta 150000 offload
tc qdisc replace dev enp1s0 parent 100:2 etf  \
clockid CLOCK_TAI delta 150000 offload

tc qdisc replace dev enp1s0 ingress
ethtool -U enp1s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF action 1
ethtool -U enp1s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF action 2

ip link add link enp1s0 name enp1s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done

igb tc...mqprio map is reverted: see Earliest TxTime First (NET_SCHED_ETF) QDisc
igb tc...flower is limited: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher with SCM_TXTIME

./pubsub_TSN_loopback -interface enp1s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback without SCM_TXTIME

./pubsub_TSN_publisher -interface enp1s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime \
-enableCsvLog

802.1Q Egress packets w/ TAPRIO tx-assisted GCL on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

BASE=$(expr $(date +%s) + 5)000000000
tc -d qdisc replace dev enp1s0 parent root handle 100 taprio \
num_tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
queues 1@0 1@1 2@2 \
base-time $BASE \
sched-entry S 01 125000 \
sched-entry S 06 125000 \
flags 0x1 txtime-delay 70000 clockid CLOCK_TAI

tc qdisc replace dev enp1s0 parent 100:1 etf  \
clockid CLOCK_TAI delta 70000 offload  skip_sock_check
tc qdisc replace dev enp1s0 parent 100:2 etf  \
clockid CLOCK_TAI delta 70000 offload skip_sock_check

tc qdisc replace dev enp1s0 ingress
ethtool -U enp1s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF action 1
ethtool -U enp1s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF action 2

ip link add link enp1s0 name enp1s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done

igb tc...taprio map is reverted: see Time Aware Priority (NET_SCHED_TAPRIO) QDisc
igb tc...flower is limited: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher with SCM_TXTIME and XDP_FLAGS_DRV_MODE umem-copy Subscriber

./pubsub_TSN_loopback -interface enp1s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableXdpSubscribe -xdpQueue 2 -xdpFlagDrvMode \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback with SCM_TXTIME

./pubsub_TSN_publisher -interface enp1s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableXdpSubscribe -xdpQueue 2 -xdpFlagDrvMode \
-enableCsvLog

802.1Q Egress packets Timed-Based Scheduler (TBS) on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

tc qdisc add dev enp1s0 parent root handle 100 mqprio \
num_tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
queues 1@0 1@1 2@2 hw 0


tc qdisc replace dev enp1s0 parent 100:1 etf  \
clockid CLOCK_TAI delta 150000 offload
tc qdisc replace dev enp1s0 parent 100:2 etf  \
clockid CLOCK_TAI delta 150000 offload

tc qdisc replace dev enp1s0 ingress
ethtool -U enp1s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF action 1
ethtool -U enp1s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF action 2

ip link add link enp1s0 name enp1s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done

igb tc...mqprio map is reverted: see Earliest TxTime First (NET_SCHED_ETF) QDisc
igb tc...flower is limited: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

Important

Intel® Ethernet Controller I225 [Foxville] Software User Manual .

It is highly recommended to turn-off VLAN Stripping and set hw-queues combined queue, Minimum is 4 combined rx/tx.
ethtool -K enp2s0 rxvlan off hw-tc-offload on
ethtool --set-eee enp2s0 eee off
ethtool -L enp2s0 combined 4
ethtool -X enp2s0 equal 2
ethtool -G enp2s0  tx 1024 rx 1024

Application /usr/share/open62541/example

Linux Traffic Control (TC)

Comments

IEC-62541 250us cyclic Publisher with SCM_TXTIME

./pubsub_TSN_loopback -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback with SCM_TXTIME

./pubsub_TSN_publisher -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableCsvLog

802.1Q Egress packets Timed-Based Scheduler (TBS) on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

tc qdisc add dev enp2s0 parent root handle 100 mqprio \
num_tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
queues 1@0 1@1 2@2 hw 0

tc qdisc replace dev enp2s0 parent 100:1 etf  \
clockid CLOCK_TAI delta 150000 offload
tc qdisc replace dev enp2s0 parent 100:2 etf  \
clockid CLOCK_TAI delta 150000 offload

tc qdisc replace dev enp2s0 ingress
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF action 1
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF action 2

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done

igc tc...mqprio map is reverted: see Earliest TxTime First (NET_SCHED_ETF) QDisc
igc tc...flower not supported: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher with SCM_TXTIME

./pubsub_TSN_loopback -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback without SCM_TXTIME

./pubsub_TSN_publisher -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime \
-enableCsvLog

802.1Q Egress packets w/ TAPRIO tx-assisted GCL on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

BASE=$(expr $(date +%s) + 5)000000000
tc -d qdisc replace dev enp2s0 parent root handle 100 taprio \
num_tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
queues 1@0 1@1 2@2 \
base-time $BASE \
sched-entry S 01 125000 \
sched-entry S 06 125000 \
flags 0x1 txtime-delay 70000 clockid CLOCK_TAI

tc qdisc replace dev enp2s0 parent 100:1 etf  \
clockid CLOCK_TAI delta 70000 offload  skip_sock_check
tc qdisc replace dev enp2s0 parent 100:2 etf  \
clockid CLOCK_TAI delta 70000 offload skip_sock_check

tc qdisc replace dev enp2s0 ingress
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF action 1
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF action 2

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done

igc tc...taprio map is reverted: see Time Aware Priority (NET_SCHED_TAPRIO) QDisc
igc tc...flower not supported: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher with SCM_TXTIME

./pubsub_TSN_loopback -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback without SCM_TXTIME

./pubsub_TSN_publisher -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime \
-enableCsvLog

802.1Q Egress packets w/ TAPRIO tx-offload GCL on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

BASE=$(date +%s%N)
tc -d qdisc replace dev enp2s0 parent root handle 100 taprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 \
base-time $BASE \
sched-entry S 01 125000 \
sched-entry S 06 125000 \
flags 0x2 txtime-delay 0

tc qdisc replace dev enp2s0 ingress
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF action 1
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF action 2

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done

igc base-time must set CLOCK_TAI initial epoch time prior GCL offload

igc tc...flower not supported: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher with SCM_TXTIME and XDP_FLAGS_DRV_MODE umem-copy Subscriber

./pubsub_TSN_loopback -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableXdpSubscribe -xdpQueue 2 -xdpFlagDrvMode \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback with SCM_TXTIME

./pubsub_TSN_publisher -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableXdpSubscribe -xdpQueue 2 -xdpFlagDrvMode \
-enableCsvLog

802.1Q Egress packets Timed-Based Scheduler (TBS) on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

tc qdisc add dev enp2s0 parent root handle 100 mqprio \
num_tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
queues 1@0 1@1 2@2 hw 0


tc qdisc replace dev enp2s0 parent 100:1 etf  \
clockid CLOCK_TAI delta 150000 offload
tc qdisc replace dev enp2s0 parent 100:2 etf  \
clockid CLOCK_TAI delta 150000 offload

tc qdisc replace dev enp2s0 ingress
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF action 1
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF action 2

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done

igc tc...taprio map is reverted: see Time Aware Priority (NET_SCHED_TAPRIO) QDisc
igc tc...flower not supported: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher without SCM_TXTIME and XDP_FLAGS_DRV_MODE umem-copy Subscriber

./pubsub_TSN_loopback -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime -enableXdpSubscribe -xdpQueue 3 \
-xdpFlagDrvMode -enableCsvLog

or

IEC-62541 250us cyclic Loopback without SCM_TXTIME

./pubsub_TSN_publisher -interface enp2s0 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime -enableXdpSubscribe -xdpQueue 3 \
-xdpFlagDrvMode -enableCsvLog

802.1Q Egress packets w/ TAPRIO tx-offload GCL 802.1Q Ingress packets steering to Queue #2 and #3

BASE=$(date +%s%N)
tc -d qdisc replace dev enp2s0 parent root handle 100 taprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 \
base-time $BASE \
sched-entry S 07 125000 \
sched-entry S 08 125000 \
flags 0x2 txtime-delay 0

tc qdisc replace dev enp2s0 ingress
ethtool -U enp2s0 flow-type ether  \
proto 0x88f7 queue 2
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x6008 \
m 0x1FFF action 3

ip link add link enp2s0 name enp2s0.vlan type vlan id 8
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done

igc base-time must set CLOCK_TAI initial epoch time prior GCL offload

igc tc...flower not supported: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 8, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

Important

Ethernet Time-Sensitive Networking for 11th Gen Intel® Core™ Processors [Tiger Lake] Traffic Shaping hw-offload are ONLY available on hw-queues 2 and 3.

It is highly recommended to turn-off VLAN Stripping and set hw-queues an even queue pair, Minimum is 4 rx 4 tx.
ethtool -K enp0s30f4 rxvlan off hw-tc-offload on
ethtool --set-eee enp0s30f4 eee off
ethtool -L enp0s30f4 rx 4 tx 4
ethtool -X enp0s30f4 equal 2
ethtool -G enp0s30f4 tx 1024 rx 1024

Application /usr/share/open62541/example

Linux Traffic Control (TC)

Comments

IEC-62541 250us cyclic Publisher with SCM_TXTIME

./pubsub_TSN_loopback -interface enp0s30f4 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback with SCM_TXTIME

./pubsub_TSN_publisher -interface enp0s30f4 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableCsvLog

802.1Q Egress packets Timed-Based Scheduler (TBS) on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

tc qdisc add dev enp0s30f4 parent root handle 100 mqprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 hw 0

tc qdisc replace dev enp0s30f4 parent 100:1 etf  \
clockid CLOCK_TAI delta 150000 offload
tc qdisc replace dev enp0s30f4 parent 100:2 etf  \
clockid CLOCK_TAI delta 150000 offload

tc qdisc replace dev enp0s30f4 ingress
tc filter add dev enp0s30f4 parent ffff: \
protocol 802.1Q flower vlan_prio 1 hw_tc 1
tc filter add dev enp0s30f4 parent ffff: \
protocol 802.1Q flower vlan_prio 2 hw_tc 2
tc filter add dev enp0s30f4 parent ffff: \
protocol 802.1Q flower vlan_prio 3 hw_tc 1

ip link add link enp0s30f4 name enp0s30f4.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp0s30f4.vlan type vlan egress-qos-map $j:$j ; done

stmmac tc...flower is supported: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher without SCM_TXTIME

./pubsub_TSN_loopback -interface enp0s30f4 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback without SCM_TXTIME

./pubsub_TSN_publisher -interface enp0s30f4 \
-cycleTimeInMsec 0.25 -socketPriority 3 \
-disableSoTxtime \
-enableCsvLog

802.1Q Egress packets w/ TAPRIO tx-offload GCL on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

BASE=$(expr $(date +%s) + 5)000000000
tc -d qdisc replace dev enp0s30f4 parent root handle 100 taprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 \
base-time $BASE \
sched-entry S 01 125000 \
sched-entry S 06 125000 \
flags 0x2 txtime-delay 0

tc qdisc replace dev enp0s30f4 ingress
tc filter add dev enp0s30f4 parent ffff: \
protocol 802.1Q flower vlan_prio 1 hw_tc 1
tc filter add dev enp0s30f4 parent ffff: \
protocol 802.1Q flower vlan_prio 2 hw_tc 2
tc filter add dev enp0s30f4 parent ffff: \
protocol 802.1Q flower vlan_prio 3 hw_tc 1

ip link add link enp0s30f4 name enp0s30f4.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp0s30f4.vlan type vlan egress-qos-map $j:$j ; done

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher with SCM_TXTIME and XDP_FLAGS_DRV_MODE zero-copy Subscriber

./pubsub_TSN_loopback -interface enp0s30f4 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableXdpSubscribe -xdpQueue 1 -xdpFlagDrvMode \
-xdpBindFlagZeroCopy -enableCsvLog

or

IEC-62541 250us cyclic Loopback with SCM_TXTIME

./pubsub_TSN_publisher -interface enp0s30f4 \
-cycleTimeInMsec 0.25 -socketPriority 3 -qbvOffset 125 \
-enableXdpSubscribe -xdpQueue 1 -xdpFlagDrvMode \
-xdpBindFlagZeroCopy -enableCsvLog

802.1Q Egress packets Timed-Based Scheduler (TBS) on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

tc qdisc add dev enp0s30f4 parent root handle 100 mqprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 hw 0


tc qdisc replace dev enp0s30f4 parent 100:1 etf  \
clockid CLOCK_TAI delta 150000 offload
tc qdisc replace dev enp0s30f4 parent 100:2 etf  \
clockid CLOCK_TAI delta 150000 offload

tc qdisc replace dev enp0s30f4 ingress
tc filter add dev enp0s30f4 parent ffff: \
protocol 802.1Q flower vlan_prio 1 hw_tc 1
tc filter add dev enp0s30f4 parent ffff: \
protocol 802.1Q flower vlan_prio 2 hw_tc 2
tc filter add dev enp0s30f4 parent ffff: \
protocol 802.1Q flower vlan_prio 3 hw_tc 1

ip link add link enp0s30f4 name enp0s30f4.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp0s30f4.vlan type vlan egress-qos-map $j:$j ; done

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

Important

Ethernet GbE Time-Sensitive Networking PRM Guide for Intel® Atom™ x6000 Processors [Elkhart Lake] Traffic Shaping hw-offload are ONLY available on hw-queues 5 and 6.

It is highly recommended to turn-off VLAN Stripping and set hw-queues an even queue pair, recommended is 8 rx 8 tx, Disable EEE option
ethtool -K enp0s29f1 rxvlan off hw-tc-offload on
ethtool --set-eee enp0s29f1 eee off
ethtool -L enp0s29f1 rx 8 tx 8
ethtool -X enp0s29f1 equal 2
ethtool -G enp0s29f1 tx 1024 rx 1024

Application /usr/share/open62541/example

Linux Traffic Control (TC)

Comments

IEC-62541 250us cyclic Publisher with SCM_TXTIME

./pubsub_TSN_loopback -interface enp0s29f1 \
-cycleTimeInMsec 0.25 -socketPriority 6 -qbvOffset 125 \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback with SCM_TXTIME

./pubsub_TSN_publisher -interface enp0s29f1 \
-cycleTimeInMsec 0.25 -socketPriority 6 -qbvOffset 125 \
-enableCsvLog

802.1Q Egress packets Timed-Based Scheduler (TBS) on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

tc qdisc add dev enp0s29f1 parent root handle 100 mqprio \
num_tc 8 map 0 1 2 3 4 5 6 7 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7 hw 0

tc qdisc replace dev enp0s29f1 parent 100:6 etf  \
clockid CLOCK_TAI delta 150000 offload
tc qdisc replace dev enp0s29f1 parent 100:5 etf  \
clockid CLOCK_TAI delta 150000 offload

tc qdisc replace dev enp0s29f1 ingress
tc filter add dev enp0s29f1 parent ffff: \
protocol 802.1Q flower vlan_prio 1 hw_tc 1
tc filter add dev enp0s29f1 parent ffff: \
protocol 802.1Q flower vlan_prio 2 hw_tc 2
tc filter add dev enp0s29f1 parent ffff: \
protocol 802.1Q flower vlan_prio 6 hw_tc 2

ip link add link enp0s29f1 name enp0s29f1.vlan type vlan id 6
for j in `seq 0 7`; \
do ip link set enp0s29f1.vlan type vlan egress-qos-map $j:$j ; done

stmmac tc...flower is supported: see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 6, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher without SCM_TXTIME

./pubsub_TSN_loopback -interface enp0s29f1 \
-cycleTimeInMsec 0.25 -socketPriority 6 \
-disableSoTxtime \
-enableCsvLog

or

IEC-62541 250us cyclic Loopback without SCM_TXTIME

./pubsub_TSN_publisher -interface enp0s29f1 \
-cycleTimeInMsec 0.25 -socketPriority 6 \
-disableSoTxtime \
-enableCsvLog

802.1Q Egress packets w/ TAPRIO tx-offload GCL on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

BASE=$(expr $(date +%s) + 5)000000000
tc -d qdisc replace dev enp0s29f1 parent root handle 100 taprio \
num_tc 8 map 0 1 2 3 4 5 6 7 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7 \
base-time $BASE \
sched-entry S 42 125000 \
sched-entry S 43 125000 \
flags 0x2 txtime-delay 0

tc qdisc replace dev enp0s29f1 ingress
tc filter add dev enp0s29f1 parent ffff: \
protocol 802.1Q flower vlan_prio 1 hw_tc 1
tc filter add dev enp0s29f1 parent ffff: \
protocol 802.1Q flower vlan_prio 2 hw_tc 2
tc filter add dev enp0s29f1 parent ffff: \
protocol 802.1Q flower vlan_prio 6 hw_tc 2

ip link add link enp0s29f1 name enp0s29f1.vlan type vlan id 6
for j in `seq 0 7`; \
do ip link set enp0s29f1.vlan type vlan egress-qos-map $j:$j ; done

Set VLAN ID to 6, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

IEC-62541 250us cyclic Publisher with SCM_TXTIME and XDP_FLAGS_DRV_MODE zero-copy Subscriber

./pubsub_TSN_loopback -interface enp0s29f1 \
-cycleTimeInMsec 0.25 -socketPriority 6 -qbvOffset 125 \
-enableXdpSubscribe -xdpQueue 2 -xdpFlagDrvMode \
-xdpBindFlagZeroCopy -enableCsvLog

or

IEC-62541 250us cyclic Loopback with SCM_TXTIME

./pubsub_TSN_publisher -interface enp0s29f1 \
-cycleTimeInMsec 0.25 -socketPriority 6 -qbvOffset 125 \
-enableXdpSubscribe -xdpQueue 2 -xdpFlagDrvMode \
-xdpBindFlagZeroCopy -enableCsvLog

802.1Q Egress packets Timed-Based Scheduler (TBS) on ETF Queue-disciplines 802.1Q Ingress packets steering to Queue #1 and #2

tc qdisc add dev enp0s29f1 parent root handle 100 mqprio \
num_tc 8 map 0 1 2 3 4 5 6 7 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7 hw 0


tc qdisc replace dev enp0s29f1 parent 100:6 etf  \
clockid CLOCK_TAI delta 400000 offload
tc qdisc replace dev enp0s29f1 parent 100:5 etf  \
clockid CLOCK_TAI delta 150000 offload

tc qdisc replace dev enp0s29f1 ingress
tc filter add dev enp0s29f1 parent ffff: \
protocol 802.1Q flower vlan_prio 1 hw_tc 1
tc filter add dev enp0s29f1 parent ffff: \
protocol 802.1Q flower vlan_prio 2 hw_tc 2
tc filter add dev enp0s29f1 parent ffff: \
protocol 802.1Q flower vlan_prio 6 hw_tc 2

ip link add link enp0s29f1 name enp0s29f1.vlan type vlan id 6
for j in `seq 0 7`; \
do ip link set enp0s29f1.vlan type vlan egress-qos-map $j:$j ; done

Set VLAN ID to 6, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

Note

The OPC UA publisher MAC address is given along with VLAN ID and PCP e.g.: opc.eth://01-00-5E-00-00-01:8.3 where 8 is the VLAN ID and 3 is the PCP. The OPC UA subscriber MAC address is given along with VLAN ID and PCP e.g.: opc.eth://01-00-5E-7F-00-01:8.3 where 8 is the VLAN ID and 3 is the PCP.

To increase the payload size, change the REPEATED_NODECOUNTS macro in pubsub_TSN_publisher.c and pubsub_TSN_loopback.c applications (for 1 REPEATED_NODECOUNTS, 8 bytes of data will increase in payload) for example. To increase the payload to 64 bytes, change REPEATED_NODECOUNTS to 4

Sanity-Check #1: egress OPC UA pubsub UADP ETH traffic-class on SO_TXTIME socket cmsg¶

In this example, just pubsub_TSN_publisher applications runtime Talk AF_PACKETv3 socket mounted Intel® i210/igb with L2-Ethernet Frames 01:00:5e:7f:00:01, ethertype 802.1Q (0x8100), length 77: vlan 8, p 3, ethertype 0xb62c TSN-endpoint egress traffic queued to hw-TXQ #3 at 1ms -cycleTimeInMsec publishing intervals.
/usr/share/open62541/examples/pubsub_TSN_publisher -cycleTimeInMsec 1.0 -interface enp1s0 -socketPriority 3 -enableCsvLog -enableconsolePrint
Published counterData and timestamp are written in csv file (-enableCsvLog) on both pubsub_TSN_publisher and pubsub_TSN_loopback applications for benchmarking purpose after terminating both the applications using Ctrl + C

Sanity-Check #2: ingress OPC UA pubsub UADP ETH traffic-class on AF_XDP socket BPF offload XDP_FLAGS_SKB_MODE xdpgeneric program¶

Linux Express Data Path (XDP) is ECI an optional user feature (also kas Project configuration files opt-xdp-bpf.dunfell.yml).

In this example, both pubsub_TSN_publisher and pubsub_TSN_loopback applications runtime talk through AF_PACKETv3 and listen AF_XDP xdpgeneric socket mounted on Intel® EHL/stmmac with L2-Ethernet frames 01:00:5e:7f:00:01, ethertype 802.1Q (0x8100), length 77: vlan 8, p 3, ethertype 0xb62c TSN-endpoint ingress traffic queued to hw-RXQ #2 and egress traffic queued to hw-TXQ #3 at 1ms -cycleTimeInMsec publishing intervals.

Tip

It is recommended to run pubsub_TSN_loopback application first to avoid missing the initial data published by pubsub_TSN_publisher

/usr/share/open62541/examples/pubsub_TSN_loopback -interface enp0s29f1 -cycleTimeInMsec 1.0 -enableCsvLog -enableconsolePrint -enableXdpSubscribe -xdpQueue 2
[2019-10-27 23:08:16.066 (UTC+0000)] warn/server        Username/Password configured, but no encrypting SecurityPolicy. This can leak credentials on the network.
[2019-10-27 23:08:16.067 (UTC+0000)] warn/userland      AcceptAll Certificate Verification. Any remote certificate will be accepted.
[2019-10-27 23:08:16.067 (UTC+0000)] info/session       Connection 0 | SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNodes: No TypeDefinition for i=50510; Use the default TypeDefinition for the Variable/Object
[2019-10-27 23:08:16.068 (UTC+0000)] info/session       Connection 0 | SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNodes: No TypeDefinition for ns=1;s=SubscriberCounter; Use the default TypeDefinition for the Variable/Object
[2019-10-27 23:08:16.068 (UTC+0000)] info/userland      PubSub channel requested
[2019-10-27 23:08:16.068 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2019-10-27 23:08:16.069 (UTC+0000)] info/userland
pthread_setschedparam:Publisher Thread priority is 78

[2019-10-27 23:08:16.069 (UTC+0000)] info/userland      Publisher CPU CORE: 2

[2019-10-27 23:08:16.069 (UTC+0000)] info/userland      PubSub channel requested
[2019-10-27 23:08:16.069 (UTC+0000)] info/server        Open PubSub ethernet connection with XDP.
[2019-10-27 23:08:16.324 (UTC+0000)] info/server        The PubSub Connection was created successfully!
[2019-10-27 23:08:16.324 (UTC+0000)] info/userland
pthread_setschedparam:Subscriber Thread priority is 81

[2019-10-27 23:08:16.324 (UTC+0000)] info/userland      Subscriber CPU CORE: 2

[2019-10-27 23:08:16.324 (UTC+0000)] info/userland
pthread_setschedparam:UserApplicationPubSub Thread priority is 75

[2019-10-27 23:08:16.324 (UTC+0000)] info/userland      UserApplicationPubSub CPU CORE: 3

[2019-10-27 23:08:16.325 (UTC+0000)] info/network       TCP network layer listening on opc.tcp://localhost:62541/

# ip link show dev enp7s0
7: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 xdpgeneric qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:07:32:6b:a7:7d brd ff:ff:ff:ff:ff:ff
    prog/xdp id 26 tag c85daa2f1b3c395f

In this example, both pubsub_TSN_publisher applications runtime Talk through AF_PACKETv3 and Listen AF_XDP xdp socket mounted on Intel® i210/igb with L2-Ethernet Frames 01:00:5e:7f:00:01, ethertype 802.1Q (0x8100), length 77: vlan 8, p 3, ethertype 0xb62c TSN-endpoint ingress traffic queued to hw-RXQ #3 and egress traffic queued to hw-TXQ #2 at 1ms -cycleTimeInMsec publishing intervals.

/usr/share/open62541/examples/pubsub_TSN_publisher -cycleTimeInMsec 1.0 -interface enp1s0 -socketPriority 3 -enableCsvLog -enableconsolePrint -enableLatencyCsvLog -enableXdpSubscribe -xdpQueue 3 -xdpFlagDrvMode
[2021-03-12 13:30:51.717 (UTC+0000)] warn/server        AccessControl: Unconfigured AccessControl. Users have all permissions.
[2021-03-12 13:30:51.717 (UTC+0000)] info/server        AccessControl: Anonymous login is enabled
[2021-03-12 13:30:51.718 (UTC+0000)] warn/server        Username/Password configured, but no encrypting SecurityPolicy. This can leak credentials on the network.
[2021-03-12 13:30:51.718 (UTC+0000)] warn/userland      AcceptAll Certificate Verification. Any remote certificate will be accepted.
[2021-03-12 13:30:51.719 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (i=50510): No TypeDefinition. Use the default TypeDefinition for the Variable/       Object
[2021-03-12 13:30:51.720 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;s=PublisherCounter): No TypeDefinition. Use the default TypeDefinition         for the Variable/Object
[2021-03-12 13:30:51.720 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;s=SubscriberCounter): No TypeDefinition. Use the default TypeDefinition        for the Variable/Object
[2021-03-12 13:30:51.721 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=10000): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.721 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=10001): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.722 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=20000): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.722 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=50000): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.723 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=50001): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.723 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=30000): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.724 (UTC+0000)] info/userland      PubSub channel requested
[2021-03-12 13:30:51.724 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2021-03-12 13:30:51.725 (UTC+0000)] info/server        MetaData creation. Found DataType Boolean.
[2021-03-12 13:30:51.725 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-03-12 13:30:51.725 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-03-12 13:30:51.726 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-03-12 13:30:51.727 (UTC+0000)] info/userland
pthread_setschedparam:Publisher Thread priority is 78

[2021-03-12 13:30:51.727 (UTC+0000)] info/userland      Publisher CPU CORE: 2

[2021-03-12 13:30:51.727 (UTC+0000)] info/userland      Publisher thread callback Id: 139991995401984

[2021-03-12 13:30:51.727 (UTC+0000)] info/userland      PubSub channel requested
[2021-03-12 13:30:51.727 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2021-03-12 13:30:51.785 (UTC+0000)] info/server        The PubSub Connection was created successfully!
[2021-03-12 13:30:51.786 (UTC+0000)] info/userland
pthread_setschedparam:Subscriber Thread priority is 81

[2021-03-12 13:30:51.786 (UTC+0000)] info/userland      Subscriber CPU CORE: 2

[2021-03-12 13:30:51.786 (UTC+0000)] info/userland      Subscriber thread callback Id: 139991982700288

[2021-03-12 13:30:51.786 (UTC+0000)] info/userland
pthread_setschedparam:UserApplicationPubSub Thread priority is 75

[2021-03-12 13:30:51.786 (UTC+0000)] info/userland      UserApplicationPubSub CPU CORE: 3

[2021-03-12 13:30:51.787 (UTC+0000)] info/network       TCP network layer listening on opc.tcp://intel-rt-corei7-64:62541/
[2021-03-12 13:30:56.000 (UTC+0000)] info/userland      Pub:1,1615555856.000312701

[2021-03-12 13:30:56.001 (UTC+0000)] info/userland      Pub:2,1615555856.001312032

[2021-03-12 13:30:56.002 (UTC+0000)] info/userland      Pub:3,1615555856.002311098

[2021-03-12 13:30:56.003 (UTC+0000)] info/userland      Pub:4,1615555856.003311384

[2021-03-12 13:30:56.004 (UTC+0000)] info/userland      Pub:5,1615555856.004310701

[2021-03-12 13:30:56.005 (UTC+0000)] info/userland      Pub:6,1615555856.005311110

[2021-03-12 13:30:56.006 (UTC+0000)] info/userland      Pub:7,1615555856.006311227

[2021-03-12 13:30:56.007 (UTC+0000)] info/userland      Pub:8,1615555856.007312370

[2021-03-12 13:30:56.008 (UTC+0000)] info/userland      Pub:9,1615555856.008311499

[2021-03-12 13:30:56.009 (UTC+0000)] info/userland      Pub:10,1615555856.009311512

# ip link show dev enp1s0
7: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 xdpgeneric qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:07:32:6b:a7:7d brd ff:ff:ff:ff:ff:ff
    prog/xdp id 29 tag c85daa2f1b3c395f

# bpftool map show
21: array  name qidconf_map  flags 0x0
        key 4B  value 4B  max_entries 1  memlock 4096B
22: xskmap  name xsks_map  flags 0x0
        key 4B  value 4B  max_entries 4  memlock 4096B
23: percpu_array  name rr_map  flags 0x0
        key 4B  value 4B  max_entries 1  memlock 4096B

Hit CTRL-C stop both pubsub_TSN_publisher pubsub_TSN_loopback application and unload xdpgeneric programs gracefully.

[2021-03-12 13:30:56.531 (UTC+0000)] info/userland      Pub:532,1615555856.531314527

[2021-03-12 13:30:56.532 (UTC+0000)] info/userland      Pub:533,1615555856.532311041

^C[2021-03-12 13:30:56.890 (UTC+0000)] info/server      received ctrl-c
[2021-03-12 13:30:57.540 (UTC+0000)] info/network       Shutting down the TCP network layer
[2021-03-12 13:30:57.541 (UTC+0000)] info/server        PubSub cleanup was called.
^C[2021-03-12 13:30:57.649 (UTC+0000)] info/server      received ctrl-c
[2021-03-12 13:30:57.705 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.

# ip link show dev enp1s0
6: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mqprio state UP mode DEFAULT group default qlen 1000
    link/ether 00:07:32:6b:a7:7d brd ff:ff:ff:ff:ff:ff

Sanity-Check #3: ingress OPC UA pubsub UADP ETH traffic-class on AF_XDP socket BPF offload XDP_FLAGS_DRV_MODE xdp program¶

Linux Express Data Path (XDP) is ECI an optional user feature (also kas Project configuration files opt-xdp-bpf.dunfell.yml).

It is recommended to run pubsub_TSN_loopback application first to avoid missing the initial data published by pubsub_TSN_publisher

In this example, both pubsub_TSN_loopback applications runtime Talk through AF_PACKETv3 and Listen AF_XDP xdp socket mounted on Intel® EHL/stmmac with L2-Ethernet Frames 01:00:5e:7f:00:01, ethertype 802.1Q (0x8100), length 77: vlan 8, p 3, ethertype 0xb62c TSN-endpoint ingress traffic queued to hw-RXQ #3 and egress traffic queued to hw-TXQ #6 at -cycleTimeInMsec publishing intervals.

/usr/share/open62541/examples/pubsub_TSN_loopback -interface enp0s29f1 -cycleTimeInMsec 1.0 -enableCsvLog -enableconsolePrint -enableXdpSubscribe -xdpQueue 2 -xdpFlagDrvMode -xdpBindFlagZeroCopy
[2019-10-27 23:08:16.066 (UTC+0000)] warn/server        Username/Password configured, but no encrypting SecurityPolicy. This can leak credentials on the network.
[2019-10-27 23:08:16.067 (UTC+0000)] warn/userland      AcceptAll Certificate Verification. Any remote certificate will be accepted.
[2019-10-27 23:08:16.067 (UTC+0000)] info/session       Connection 0 | SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNodes: No TypeDefinition for i=50510; Use the default TypeDefinition for the Variable/Object
[2019-10-27 23:08:16.068 (UTC+0000)] info/session       Connection 0 | SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNodes: No TypeDefinition for ns=1;s=SubscriberCounter; Use the default TypeDefinition for the Variable/Object
[2019-10-27 23:08:16.068 (UTC+0000)] info/userland      PubSub channel requested
[2019-10-27 23:08:16.068 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2019-10-27 23:08:16.069 (UTC+0000)] info/userland
pthread_setschedparam:Publisher Thread priority is 78

[2019-10-27 23:08:16.069 (UTC+0000)] info/userland      Publisher CPU CORE: 2

[2019-10-27 23:08:16.069 (UTC+0000)] info/userland      PubSub channel requested
[2019-10-27 23:08:16.069 (UTC+0000)] info/server        Open PubSub ethernet connection with XDP.
[2019-10-27 23:08:16.324 (UTC+0000)] info/server        The PubSub Connection was created successfully!
[2019-10-27 23:08:16.324 (UTC+0000)] info/userland
pthread_setschedparam:Subscriber Thread priority is 81

[2019-10-27 23:08:16.324 (UTC+0000)] info/userland      Subscriber CPU CORE: 2

[2019-10-27 23:08:16.324 (UTC+0000)] info/userland
pthread_setschedparam:UserApplicationPubSub Thread priority is 75

[2019-10-27 23:08:16.324 (UTC+0000)] info/userland      UserApplicationPubSub CPU CORE: 3

[2019-10-27 23:08:16.325 (UTC+0000)] info/network       TCP network layer listening on opc.tcp://localhost:62541/

# ip link show dev enp7s0
7: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 xdpgeneric qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:07:32:6b:a7:7d brd ff:ff:ff:ff:ff:ff
    prog/xdp id 26 tag c85daa2f1b3c395f

In this example, both pubsub_TSN_publisher applications runtime Talk through AF_PACKETv3 and Listen AF_XDP xdp socket mounted on Intel® i210/igb with L2-Ethernet Frames 01:00:5e:7f:00:01, ethertype 802.1Q (0x8100), length 77: vlan 8, p 3, ethertype 0xb62c TSN-endpoint ingress traffic queued to hw-RXQ #3 and egress traffic queued to hw-TXQ #2 at -cycleTimeInMsec publishing intervals.

/usr/share/open62541/examples/pubsub_TSN_publisher -cycleTimeInMsec 1 -interface enp1s0 -socketPriority 3 -enableCsvLog -enableconsolePrint -enableLatencyCsvLog -enableXdpSubscribe -xdpQueue 3 -xdpFlagDrvMode

[2021-03-12 13:30:51.717 (UTC+0000)] warn/server        AccessControl: Unconfigured AccessControl. Users have all permissions.
[2021-03-12 13:30:51.717 (UTC+0000)] info/server        AccessControl: Anonymous login is enabled
[2021-03-12 13:30:51.718 (UTC+0000)] warn/server        Username/Password configured, but no encrypting SecurityPolicy. This can leak credentials on the network.
[2021-03-12 13:30:51.718 (UTC+0000)] warn/userland      AcceptAll Certificate Verification. Any remote certificate will be accepted.
[2021-03-12 13:30:51.719 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (i=50510): No TypeDefinition. Use the default TypeDefinition for the Variable/       Object
[2021-03-12 13:30:51.720 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;s=PublisherCounter): No TypeDefinition. Use the default TypeDefinition         for the Variable/Object
[2021-03-12 13:30:51.720 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;s=SubscriberCounter): No TypeDefinition. Use the default TypeDefinition        for the Variable/Object
[2021-03-12 13:30:51.721 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=10000): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.721 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=10001): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.722 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=20000): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.722 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=50000): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.723 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=50001): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.723 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=30000): No TypeDefinition. Use the default TypeDefinition for the        Variable/Object
[2021-03-12 13:30:51.724 (UTC+0000)] info/userland      PubSub channel requested
[2021-03-12 13:30:51.724 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2021-03-12 13:30:51.725 (UTC+0000)] info/server        MetaData creation. Found DataType Boolean.
[2021-03-12 13:30:51.725 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-03-12 13:30:51.725 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-03-12 13:30:51.726 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-03-12 13:30:51.727 (UTC+0000)] info/userland
pthread_setschedparam:Publisher Thread priority is 78

[2021-03-12 13:30:51.727 (UTC+0000)] info/userland      Publisher CPU CORE: 2

[2021-03-12 13:30:51.727 (UTC+0000)] info/userland      Publisher thread callback Id: 139991995401984

[2021-03-12 13:30:51.727 (UTC+0000)] info/userland      PubSub channel requested
[2021-03-12 13:30:51.727 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2021-03-12 13:30:51.785 (UTC+0000)] info/server        The PubSub Connection was created successfully!
[2021-03-12 13:30:51.786 (UTC+0000)] info/userland
pthread_setschedparam:Subscriber Thread priority is 81

[2021-03-12 13:30:51.786 (UTC+0000)] info/userland      Subscriber CPU CORE: 2

[2021-03-12 13:30:51.786 (UTC+0000)] info/userland      Subscriber thread callback Id: 139991982700288

[2021-03-12 13:30:51.786 (UTC+0000)] info/userland
pthread_setschedparam:UserApplicationPubSub Thread priority is 75

[2021-03-12 13:30:51.786 (UTC+0000)] info/userland      UserApplicationPubSub CPU CORE: 3

[2021-03-12 13:30:51.787 (UTC+0000)] info/network       TCP network layer listening on opc.tcp://intel-rt-corei7-64:62541/
[2021-03-12 13:30:56.000 (UTC+0000)] info/userland      Pub:1,1615555856.000312701

[2021-03-12 13:30:56.001 (UTC+0000)] info/userland      Pub:2,1615555856.001312032

[2021-03-12 13:30:56.002 (UTC+0000)] info/userland      Pub:3,1615555856.002311098

[2021-03-12 13:30:56.003 (UTC+0000)] info/userland      Pub:4,1615555856.003311384

[2021-03-12 13:30:56.004 (UTC+0000)] info/userland      Pub:5,1615555856.004310701

[2021-03-12 13:30:56.005 (UTC+0000)] info/userland      Pub:6,1615555856.005311110

[2021-03-12 13:30:56.006 (UTC+0000)] info/userland      Pub:7,1615555856.006311227

[2021-03-12 13:30:56.007 (UTC+0000)] info/userland      Pub:8,1615555856.007312370

[2021-03-12 13:30:56.008 (UTC+0000)] info/userland      Pub:9,1615555856.008311499

[2021-03-12 13:30:56.009 (UTC+0000)] info/userland      Pub:10,1615555856.009311512

# ip link show dev enp1s0
7: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 xdpgeneric qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:07:32:6b:a7:7d brd ff:ff:ff:ff:ff:ff
    prog/xdp id 29 tag c85daa2f1b3c395f

# bpftool map show
21: array  name qidconf_map  flags 0x0
        key 4B  value 4B  max_entries 1  memlock 4096B
22: xskmap  name xsks_map  flags 0x0
        key 4B  value 4B  max_entries 4  memlock 4096B
23: percpu_array  name rr_map  flags 0x0
        key 4B  value 4B  max_entries 1  memlock 4096B

Hit CTRL-C stop both pubsub_TSN_publisher pubsub_TSN_loopback application and unload xdpgeneric programs gracefully.

[2021-03-12 13:30:56.531 (UTC+0000)] info/userland      Pub:532,1615555856.531314527

[2021-03-12 13:30:56.532 (UTC+0000)] info/userland      Pub:533,1615555856.532311041

^C[2021-03-12 13:30:56.890 (UTC+0000)] info/server      received ctrl-c
[2021-03-12 13:30:57.540 (UTC+0000)] info/network       Shutting down the TCP network layer
[2021-03-12 13:30:57.541 (UTC+0000)] info/server        PubSub cleanup was called.
^C[2021-03-12 13:30:57.649 (UTC+0000)] info/server      received ctrl-c
[2021-03-12 13:30:57.705 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-03-12 13:30:57.705 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.

# ip link show dev enp1s0
6: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mqprio state UP mode DEFAULT group default qlen 1000
    link/ether 00:07:32:6b:a7:7d brd ff:ff:ff:ff:ff:ff

Sanity-Check #4: egress OPC UA pubsub UADP ETH traffic-class on TAPRIO tx-offload and ingress OPC UA pubsub UADP ETH traffic-class on BPF offload XDP_FLAGS_DRV_MODE xdp program¶

Linux Express Data Path (XDP) is ECI an optional user feature (also kas Project configuration files opt-xdp-bpf.dunfell.yml).

It is recommended to run pubsub_TSN_loopback application first to avoid missing the initial data published by pubsub_TSN_publisher

In this example, both pubsub_TSN_loopback applications runtime Talk through AF_PACKET and Listen AF_XDP xdp socket mounted on Intel® i225/igc with L2-Ethernet Frames 01:00:5e:7f:00:01, ethertype 802.1Q (0x8100), length 77: vlan 8, p 3, ethertype 0xb62c TSN-endpoint ingress traffic queued to hw-RXQ #3 and egress traffic queued to hw-TXQ #3 at -cycleTimeInMsec publishing intervals.

/usr/share/open62541/examples# ./pubsub_TSN_loopback -interface $IFACE -cycleTimeInMsec 0.25 -socketPriority 3 -disableSoTxtime -enableXdpSubscribe -xdpQueue 3 -xdpFlagDrvMode -enableCsvLog
[2021-08-17 15:10:06.923 (UTC+0000)] warn/server        AccessControl: Unconfigured AccessControl. Users have all permissions.
[2021-08-17 15:10:06.923 (UTC+0000)] info/server        AccessControl: Anonymous login is enabled
[2021-08-17 15:10:06.923 (UTC+0000)] warn/server        Username/Password configured, but no encrypting SecurityPolicy. This can leak credentials on the network.
[2021-08-17 15:10:06.923 (UTC+0000)] warn/userland      AcceptAll Certificate Verification. Any remote certificate will be accepted.
[2021-08-17 15:10:06.924 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (i=50510): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:06.924 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;s=PublisherCounter): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:06.924 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;s=SubscriberCounter): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:06.924 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=10000): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:06.924 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=10001): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:06.924 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=20000): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:06.924 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=50000): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:06.924 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=50001): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:06.924 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=30000): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:06.924 (UTC+0000)] info/userland      PubSub channel requested
[2021-08-17 15:10:06.924 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2021-08-17 15:10:06.924 (UTC+0000)] info/server        MetaData creation. Found DataType Boolean.
[2021-08-17 15:10:06.924 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-08-17 15:10:06.924 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-08-17 15:10:06.924 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-08-17 15:10:06.924 (UTC+0000)] info/userland
pthread_setschedparam:Publisher Thread priority is 78

[2021-08-17 15:10:06.924 (UTC+0000)] info/userland      Publisher CPU CORE: 2

[2021-08-17 15:10:06.924 (UTC+0000)] info/userland      Publisher thread callback Id: 140250372617984

[2021-08-17 15:10:06.924 (UTC+0000)] info/userland      PubSub channel requested
[2021-08-17 15:10:06.924 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2021-08-17 15:10:07.434 (UTC+0000)] info/server        The PubSub Connection was created successfully!
[2021-08-17 15:10:07.434 (UTC+0000)] info/userland
pthread_setschedparam:Subscriber Thread priority is 81

[2021-08-17 15:10:07.434 (UTC+0000)] info/userland      Subscriber CPU CORE: 2

[2021-08-17 15:10:07.434 (UTC+0000)] info/userland      Subscriber thread callback Id: 140250359969536

[2021-08-17 15:10:07.434 (UTC+0000)] info/userland
pthread_setschedparam:UserApplicationPubSub Thread priority is 75

[2021-08-17 15:10:07.434 (UTC+0000)] info/userland      UserApplicationPubSub CPU CORE: 3

[2021-08-17 15:10:07.551 (UTC+0000)] info/network       TCP network layer listening on opc.tcp://ecs-intel-d995:62541/

# ip link show dev $IFACE
4: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc taprio state UP mode DEFAULT group default qlen 1000
    link/ether 00:a0:c9:00:00:02 brd ff:ff:ff:ff:ff:ff
    prog/xdp id 19 tag 992d9ddc835e5629

In this example, both pubsub_TSN_publisher applications runtime Talk through AF_PACKET and Listen AF_XDP xdp socket mounted on Intel® i225/igc with L2-Ethernet Frames 01:00:5e:7f:00:01, ethertype 802.1Q (0x8100), length 77: vlan 8, p 3, ethertype 0xb62c TSN-endpoint ingress traffic queued to hw-RXQ #3 and egress traffic queued to hw-TXQ #3 at -cycleTimeInMsec publishing intervals.

/usr/share/open62541/examples# ./pubsub_TSN_publisher -interface $IFACE -cycleTimeInMsec 0.25 -socketPriority 3 -disableSoTxtime -enableXdpSubscribe -xdpQueue 3 -xdpFlagDrvMode -enableCsvLog
[2021-08-17 15:10:10.761 (UTC+0000)] warn/server        AccessControl: Unconfigured AccessControl. Users have all permissions.
[2021-08-17 15:10:10.761 (UTC+0000)] info/server        AccessControl: Anonymous login is enabled
[2021-08-17 15:10:10.761 (UTC+0000)] warn/server        Username/Password configured, but no encrypting SecurityPolicy. This can leak credentials on the network.
[2021-08-17 15:10:10.761 (UTC+0000)] warn/userland      AcceptAll Certificate Verification. Any remote certificate will be accepted.
[2021-08-17 15:10:10.784 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (i=50510): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:10.784 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;s=PublisherCounter): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:10.784 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;s=SubscriberCounter): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:10.784 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=10000): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:10.784 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=10001): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:10.784 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=20000): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:10.784 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=50000): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:10.784 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=50001): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:10.785 (UTC+0000)] info/session       SecureChannel 0 | Session g=00000001-0000-0000-0000-000000000000 | AddNode (ns=1;i=30000): No TypeDefinition. Use the default TypeDefinition for the Variable/Object
[2021-08-17 15:10:10.785 (UTC+0000)] info/userland      PubSub channel requested
[2021-08-17 15:10:10.785 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2021-08-17 15:10:10.785 (UTC+0000)] info/server        MetaData creation. Found DataType Boolean.
[2021-08-17 15:10:10.785 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-08-17 15:10:10.785 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-08-17 15:10:10.785 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-08-17 15:10:10.785 (UTC+0000)] info/userland
pthread_setschedparam:Publisher Thread priority is 78

[2021-08-17 15:10:10.785 (UTC+0000)] info/userland      Publisher CPU CORE: 2

[2021-08-17 15:10:10.785 (UTC+0000)] info/userland      Publisher thread callback Id: 140456772486912

[2021-08-17 15:10:10.785 (UTC+0000)] info/userland      PubSub channel requested
[2021-08-17 15:10:10.785 (UTC+0000)] info/server        Open PubSub ethernet connection.
[2021-08-17 15:10:10.964 (UTC+0000)] info/server        The PubSub Connection was created successfully!
[2021-08-17 15:10:10.964 (UTC+0000)] info/userland
pthread_setschedparam:Subscriber Thread priority is 81

[2021-08-17 15:10:10.964 (UTC+0000)] info/userland      Subscriber CPU CORE: 2

[2021-08-17 15:10:10.964 (UTC+0000)] info/userland      Subscriber thread callback Id: 140456759838464

[2021-08-17 15:10:10.964 (UTC+0000)] info/userland
pthread_setschedparam:UserApplicationPubSub Thread priority is 75

[2021-08-17 15:10:10.964 (UTC+0000)] info/userland      UserApplicationPubSub CPU CORE: 3

[2021-08-17 15:10:11.015 (UTC+0000)] info/network       TCP network layer listening on opc.tcp://ecs-intel-c3cd:62541/

# ip link show dev $IFACE
5: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc taprio state UP mode DEFAULT group default qlen 1000
    link/ether 00:a0:c9:00:00:03 brd ff:ff:ff:ff:ff:ff
    prog/xdp id 17 tag 992d9ddc835e5629

# bpftool map show
5: xskmap  name xsks_map  flags 0x0
        key 4B  value 4B  max_entries 4  memlock 4096B

Hit CTRL-C stop both pubsub_TSN_publisher pubsub_TSN_loopback application and unload xdpgeneric programs gracefully.

^C[2021-08-17 15:14:26.016 (UTC+0000)] info/server      received ctrl-c
[2021-08-17 15:14:26.016 (UTC+0000)] info/network       Shutting down the TCP network layer
[2021-08-17 15:14:26.319 (UTC+0000)] info/server        PubSub cleanup was called.
^C[2021-08-17 15:14:26.319 (UTC+0000)] info/server      received ctrl-c
[2021-08-17 15:14:26.583 (UTC+0000)] info/server        MetaData creation. Found DataType UInt64.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. Reading ArrayDimension failed.
[2021-08-17 15:14:26.583 (UTC+0000)] warn/server        PubSub meta data generation. DataType Node is UA_NODEID_NULL.

# ip link show dev $IFACE
4: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc taprio state UP mode DEFAULT group default qlen 1000
    link/ether 00:a0:c9:00:00:02 brd ff:ff:ff:ff:ff:ff

# tc -s qdisc show dev $IFACE
qdisc taprio 100: root refcnt 5 tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0
queues offset 0 count 1 offset 1 count 1 offset 2 count 1 offset 3 count 1
clockid invalid flags 0x2       base-time 1629090112000000000 cycle-time 250000 cycle-time-extension 0
        index 0 cmd S gatemask 0x7 interval 125000
        index 1 cmd S gatemask 0x8 interval 125000

 Sent 463531981 bytes 6021408 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc pfifo 0: parent 100:4 limit 1000p
 Sent 461996073 bytes 5999949 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc pfifo 0: parent 100:3 limit 1000p
 Sent 1432160 bytes 21043 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc pfifo 0: parent 100:2 limit 1000p
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc pfifo 0: parent 100:1 limit 1000p
 Sent 103748 bytes 416 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0