Attention

You are viewing an older version of the documentation. The latest version is v3.3.

Intel® Ethernet TSN IoTG Reference Sample Applications

Intel® IoT Group (IoTG) released with the 11th Gen Intel® Core™ and Intel Atom® x6000E Series Ethernet Controllers Time-Sensitive Networking (TSN) Reference Sample Applications a powerful performance testing framework utilizing synthetic and IEC-62541 UADP packets to compare Linux Intel® Industrial Ethernet driver hw-offload capabilities: Linux Traffic Control TAPRIO/ETF Queue-disciplines (qdisc) -vs- Linux Express Data Path (XDP) Native-mode.

This section targets audience that should already be familiar with working in a Linux command-line environment to understand the building blocks used to develop solutions using Ethernet Time-Sensitive Networking (TSN) technologies in a Linux environment.

ECI image comes prebuilt with https://github.com/intel/iotg_tsn_ref_sw [v0.8.9] (commit-id 9ca756a2dc505e4d42be808de1a772823277d21d) providing several Sample Applications to perform the Intel® Industrial Ethernet TSN controller parametric test :

• /opt/intel/iotg_tsn_ref_sw/tsq for Time Synchronization Quality Measurement. This helps an OT admin verifying the nanosecond-precision time synchronization between platforms using pulse-per-second (PPS) output & auxiliary timestamping (AUX_TS) input pins.

• /opt/intel/iotg_tsn_ref_sw/txrx-tsn for AF_PACKET & AF_XDP socket latency characterization. This helps an OT admin verifying that bounded latency transmission/reception is achieved over standard Linux AF_PACKET & AF_XDP sockets APIs using synthetic packets, on a single-Ethernet port, using available Intel® Industrial Ethernet hw-offload capabilities.

• /opt/intel/iotg_tsn_ref_sw/opcua-server for AF_PACKET & AF_XDP OPC UA pubsub (part4) application latency characterization. This helps an OT admin verifying that bounded latency transmission/reception is achieved over standard Linux AF_PACKET & AF_XDP sockets APIs, on a single-Ethernet port or dual-Ethernet ports system, using 3rd party IEC-62541 framework using available Intel® Industrial Ethernet hw-offload capabilities.

Important

Stay tuned with https://github.com/intel/iotg_tsn_ref_sw (master) open-source project for latest & greatest.

Note

Direct 802.1as time-sync need to be established see Linuxptp stack 802.1AS gPTP Profile such as one Linux TSN endpoint port act as Grand-master (GM) Clock (below enp1s0)

taskset -c 1 ptp4l -mP2Hi enp1s0.vlan -f scripts/gPTP.cfg --step_threshold=2 --socket_priority 2 2&> /var/log/ptp4l.log &
pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248 clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37 leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1 frequencyTraceable 0 timeSource 0xa0"
taskset -c 1 phc2sys -c CLOCK_REALTIME --step_threshold=1 -s enp1s0 --transportSpecific=1 -O 0 -w -ml 7 2&> /var/log/phc2sys.log &

Optionally enp1s0.vlan can be set with VLANPCP=2 onto every egress L2/Ethernet PTP event message (Ethertype 0x88f7) Ethernet frames using affinitization to Ethernet controller hw-queues socket_priority=2 and CPU core taskset -c 1 in this example.

Time Synchronization Quality measurement

The /opt/intel/iotg_tsn_ref_sw/tsq workload retrieves externally-triggered hardware timestamps (by PPS) and calculates the difference between the timestamps collected from two platforms which PTP clocks synchronized IEEE 802.1AS Time-domain using ptp4l.

Note

This test is hardware dependent on PPS (pulse-per-second) and AUXTS (auxiliary timestamping) signals and and pin header connections being present on Motherboard or PCIe for Time Synchronization Quality Measurement :

Board	PPS	AUXTS	Comments
TGL-mGBE	J2H4 pin 1	J2H4 pin 6	Tiger Lake UP3 Intel® RVP
i210-T1	SDP0	SDP1	Server i210-T1 x1 PCIe boar d
i225-LM	SDP0	SDP1	Experimental-only support

If no pin can be physically accessed, please discard the Time Synchronization Quality Measurement section and corresponding sanity-Check

echo 0 0 0 1 0 > /sys/class/ptp/ptpX/period

Usage: echo <idx> <ts> <tns> <ps> <pns> > /sys/class/ptp/ptpX/period

Where :

• <idx> PPS number

• <ts> Start time (second), based on PTP time

• <tns> Start time (nanosecond), based on PTP time

• <ps> Period (s)

• <pns> Period (ns)

• ptpX PTP device on Ethernet slave or master port

To see other option please run ./tsq --help

/opt/intel/iotg_tsn_ref_sw# ./tsq --help
Usage: tsq [OPTION...] -i <IP_ADDR> -p <PORT> -d /dev/ptp<X> -u <UID> {-T|-L}
Time Sync Quality Measurement application

  -d, --device=FILE          PTP device to read (eg. /dev/ptp1)
  -i, --ip=ADDR              Server IP address (eg. 192.1.2.3)
  -L, --listener             Listener mode (listen for 2 talker and compare)
  -o, --output=FILE          Save output to FILE (eg. temp.txt)
  -p, --port=PORT            Port number
                                Def: 5678 | Min: 999 | Max: 9999
  -t, --timeout=MSEC         Polling timeout in ms      Def: 1100ms | Min: 1ms |
                             Max: 2000ms
  -T, --talker               Talker mode (read AUXTS)
  -u, --uid=COUNT            Unique Talker ID   Def: 1234 | Min: 999 | Max: 9999
  -v, --verbose              Produce verbose output
  -?, --help                 Give this help list
      --usage                Give a short usage message

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.

Important

[EXPERIMENTAL] intel-wired-lan 5.4.y backport add PPS and SDP support for igc/i225 (UNDER UPSTREAM 5.11 ) only available when opt-experimental.yml optional user-feature is selected Project configuration files

• 0017-igc-Enable-internal-i225-PPS.patch

• 0018-igc-enable-auxiliary-PHC-functions-for-the-i225.patch

Optionally, a test engineer may desire to pull from target filesystem ./tsq ... > tsq-listener-data.txt all recorded PTP timestamps into a visual plot as in tsqplot.gnu XY gnuplot as described below :

Click to toggle visibility

set xrange[0:100]
set yrange[-50:50]
plot filename using ($3) with lines title "nano-seconds"
set output 'tsqplot.png'
pause 1
reread
set autoscale

$ gnuplot -e "filename='tsq-listener-data.txt'" $DIR/tsqplot.gnu &

Linux socket RX/TX latency measurement

/opt/intel/iotg_tsn_ref_sw/txrx-tsn workload helps an OT administrator to compares endpoint ingress and egress Traffic-class QoS in IEEE 802.1Q-2018 Enhancements for Scheduled Traffic (EST) polices under various hw-offloading conditions.

/opt/intel/iotg_tsn_ref_sw# ./txrx-tsn --help
Usage: txrx-tsn [OPTION...] -i <interface> -P [r|t]
  or:  txrx-tsn [OPTION...] -i <interface> -X [r|t|f] [z|c|s] -q <queue>
  AF_XDP & AF_PACKET Transmit-Receive Application

  -i, --interface=NAME       interface name

 Socket Mode:
  -P, --afpkt                run using AF_PACKET socket
  -X, --afxdp                run using AF_XDP socket

 Mode:
  -b, --boomerang            L2-send-receive packets (AF_XDP only)
  -f, --forward              L2-forward (receive-send) packets (AF_XDP only)
  -r, --receive              receive only
  -t, --transmit             transmit only

 XDP Mode:
  -c, --native-copy          direct/native copy mode
  -s, --skb                  skb mode
  -z, --zero-copy            zero-copy mode

 AF_XDP control:
  -p, --polling              enable polling mode
  -q, --vlan-prio=NUM        packet vlan priority, also socket priority
                                Def: 0 | Min: 0 | Max: 7
  -T, --launchtime           enable time-based per-packet tx scheduling (TBS)
  -w, --wakeup               enable need_wakeup

 TX control:
  -d, --dst-mac-addr=MAC_ADDR   destination mac address
                                Def: 22:bb:22:bb:22:bb
  -l, --packet-size=NUM      packet size/length incl. headers in bytes
                                Def: 64 | Min: 64 | Max: 1500
  -n, --frames-to-send=NUM   number of packets to transmit
                                Def: 1000 | Min: 1 | Max: 10000000
  -y, --cycle-time=NSEC      tx period/interval/cycle-time
                                Def: 100000ns | Min: 25000ns | Max: 50000000ns

 LaunchTime/TBS-specific:
 (where base is the 0th ns of current second)
  -e, --early-offset=NSEC    early execution negative offset
                                Def: 100000ns | Min: 0ns | Max: 10000000ns
  -o, --transmit-offset=NSEC packet txtime positive offset
                                Def: 0ns | Min: 0ns | Max: 100000000ns

 Misc:
  -h, --hw-timestamps        retrieve per-packet hardware timestamps
                             (AF_PACKET)
  -v, --verbose              verbose & print warnings

  -?, --help                 Give this help list
      --usage                Give a short usage message

Optionally, a test engineer may desire to pull from target filesystem ./txrx-tsn -P ... > /tmp/afpkt-traffic.txt and/or ./txrx-tsn -X ... > /tmp/afxdp-traffic.txt all recorded packets timestamps into a visual plot as in txrx-tsn-plot.gnu multi XY gnuplot examples:

Click to toggle visibility

reset

  set terminal pngcairo size 1920,1080
  set output 'txrx-tsn-plot.png'

  set size 1,1
  set datafile missing'?' # Skip invalid/missing data
  set locale "en_US.utf8" # Set to en_US locale for thousands seperator

  set tmargin 5
  set bmargin 5
  set lmargin 15
  set rmargin 5

  set yrange [0:10000000] #Legacy socket plot YMAX

  set grid
  set key center tmargin
  set border

  set style fill solid 0.25
  set boxwidth 0.5
  # set logscale y 10

  set title "Transmission latency from TX User-space to RX User-space"
  set xlabel "Packet count"
  set ylabel "Latency in nanoseconds"

  set multiplot layout 1,2

  stats FILENAME u 1 name "a" nooutput
  stats FILENAME2 u 1 name "b" nooutput

  min_us = a_min/1000
  max_us = a_max/1000
  avg_us = a_mean/1000

  set label front sprintf("Minimum: %d us", min_us) at graph 0.01, graph 0.95
  set label front sprintf("Maximum: %d us", max_us) at graph 0.01, graph 0.90
  set label front sprintf("Average: %d us", avg_us) at graph 0.01, graph 0.85

  plot FILENAME \
    using ($1) title "Legacy socket"  lc rgb "red" w points

  unset label

  set yrange [0:2000000]  #XDP socket plot YMAX

  min_us = b_min/1000
  max_us = b_max/1000
  avg_us = b_mean/1000

  set label front sprintf("Minimum: %d us", min_us) at graph 0.01, graph 0.95
  set label front sprintf("Maximum: %d us", max_us) at graph 0.01, graph 0.90
  set label front sprintf("Average: %d us", avg_us) at graph 0.01, graph 0.85

  plot FILENAME2 \
    using ($1) title "XDP socket"  lc rgb "red" w points

  unset multiplot
  unset output

  #Replot to GUI
  set terminal x11 size 1720,980
  set output
  set multiplot layout 1,2
  plot FILENAME \
    using ($1) title "Legacy socket"  lc rgb "red" w points

  plot FILENAME2 \
    using ($1) title "XDP socket"  lc rgb "red" w points

pause 10 #in case some one forgets to add -p

$ gnuplot -e "FILENAME='afpkt-traffic.txt';FILENAME2='afxdp-traffic.txt';" $DIR/txrx-tsn-plot.gnu -p 2> /dev/null &

Optionally, a test engineer may desire to introduce noisy-neighbor client-server to stress DUTs traffic-class QoS robustness :

• To setup a dummy TCP/IPv4 server

  iperf3 -s -B <IPv4> -i 10 -A 0 -1 2&> /dev/null'
  

• To instantiate a client that concurrently to stress the data link on Best-effort VLANPCP=0 traffic-class.

  iperf3 -c <IPv4> -u -b 200M -l 1440 -f m -i 10 -t 30000 -A 0 2&> /dev/null'
  

AF_PACKET socket subtest

The txrx-afpkt.c (commit-id ea1f241b37200c4c9fa18062c7053c67ffc9cb88) User space programs spawn a communication thread which creates a socket on a specified priority and continuously loops to either send or receive packets.

The process main thread behavior depends of two operational modes:

• in MODE_TX a SOCK_DGRAM socket is opened to transmit -n <total number> of egress packets each -l <in byte incl. headers> length in size to a given interface, by calling afpkt_send_thread() with cyclic-timer -y <nanoseconds tx-interval>. Optionally with -T  -e <nanoseconds cyclic sampling-offset> -o <nanoseconds cyclic tx-offset> enabling the time-based per-packet tx scheduling (TBS), by calling afpkt_send_thread_etf() see Earliest TxTime First (NET_SCHED_ETF) QDisc

• in MODE_RX a SOCK_RAW socket is opened to receive all ingress packets from a given interface. Optionally -h hw-timestamping for all ingress packets using time stamping policy at the network driver level with the SIOCSHWTSTAMP

To characterize afpkt latency overhead and jitter using synthetic L2/Ethernet packets 64B payload traversing Linux Socket API, netlink QDisc, netdev Intel® Ethernet kernel driver :

Tip

It is highly recommended to clean previous tc qdisc and tc filter (or ethtool flow-type) setting before any new egress traffic shaping or ingress traffic steering rules.

tc qdisc del dev $IFACE parent root
tc qdisc del dev $IFACE parent ffff:
tc filter del dev $IFACE parent ffff:
ethtool -U $IFACE delete $ID
ip addr flush dev $IFACE
ip neigh flush all dev $IFACE
ip link delete dev $IFACE.vlan

Intel® I210/igbIntel®I225-LM/igcIntel®TGL/stmmacIntel® EHL/stmmac

Important

Intel Ethernet Controller I210 [Springville] Traffic Shaping hw-offload is ONLY available on hw-queues 0 and 1.

It is highly recommended to turn-off VLAN Stripping and set hw-queues combined queue, Minimum is 4 combined rx/tx.

ethtool -K enp1s0 rxvlan off hw-tc-offload on
ethtool --set-eee enp1s0 eee off
ethtool -L enp1s0 combined 4
ethtool -G enp1s0  tx 1024 rx 1024

Application w/ Linux POSIX SOCKET API	Linux Traffic Control (TC)	Comments
MODE_TX sock cyclic write SCM_TXTIME & VLAN tagged packet ./txrx-tsn -v -i enp1s0 -P -l 64 -q 1 \ --transmit -n 10000 -y 1000000 -T -e 700000 -o 20000 \ -d 22:bb:22:bb:22:bb > /tmp/igb-afpk1a_etf.txt & # Assign SCHED_FIFO policy to CPU2 TX_PID=$! taskset -p 2 $TX_PID chrt --fifo -p 90 $TX_PID	802.1Q Egress packets Timed-Based Sched.(TBS) on ETF Queue-disciplines tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 tc qdisc replace dev enp1s0 parent 100:1 etf \ clockid CLOCK_TAI delta 400000 offload ip link add link enp1s0 name enp1s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done	igb tc...mqprio map is reverted see Earliest TxTime First (NET_SCHED_ETF) QDisc Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map
MODE_TX sock cyclic write VLAN-tagged packet ./txrx-tsn -v -i enp1s0 -P -l 64 -q 1 \ --transmit -n 10000 -y 1000000 \ -d 22:bb:22:bb:22:bb > /tmp/igb-afpk1a_taprio-0x1.txt & # Assign SCHED_FIFO policy to CPU2 TX_PID=$! taskset -p 2 $TX_PID chrt --fifo -p 90 $TX_PID	802.1Q Egress packets w/ TAPRIO tx-assisted GCL on ETF Queue-disciplines BASE=$(expr $(date +%s) + 5)000000000 tc -d qdisc replace dev enp1s0 parent root handle 100 taprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 \ base-time $BASE \ sched-entry S 01 500000 \ sched-entry S 0e 500000 \ flags 0x1 txtime-delay 300000 clockid CLOCK_TAI tc qdisc replace dev enp1s0 parent 100:1 etf \ clockid CLOCK_TAI delta 400000 offload skip_sock_check ip link add link enp1s0 name enp1s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done	igb tc...taprio map is reverted see Time Aware Priority (NET_SCHED_TAPRIO) QDisc Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map
MODE_RX cyclic Listener rt-thread with SIOCSHWTSTAMP ./txrx-tsn -v -i enp1s0 -P -q 3 \ --receive --hw-timestamps > /tmp/igb-afpk1b.txt & # Assign SCHED_FIFO policy to CPU2 RX_PID=$! taskset -p 2 $RX_PID chrt --fifo -p 90 $RX_PID	802.1Q Ingress PTP and VLAN prior 1 and 3 packets steering hw-filters tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 tc qdisc replace dev enp1s0 ingress ethtool -U enp1s0 flow-type ether \ proto 0x8100 vlan 0x2000 \ m 0x1FFF action 1 ethtool -U enp1s0 flow-type ether \ proto 0x88f7 queue 2 ethtool -U enp1s0 flow-type ether \ proto 0x8100 vlan 0x6000 \ m 0x1FFF action 3 ip link add link enp1s0 name enp1s0.vlan type vlan id 3	igb tc...taprio map is reverted see Time Aware Priority (NET_SCHED_TAPRIO) QDisc igb tc...flower is limited see Packet Classifier (CONFIG_NET_CLS_FLOWER) Set VLAN ID to 3, all traffic fixed

AF_XDP socket subtest

The txrx-afxdp.c (commit-id ea1f241b37200c4c9fa18062c7053c67ffc9cb88) User space programs spawn a communication thread which creates a AF_XDP socket set to specified priority queues and continuously loops to either send or receive packets. (see AF_XDP socket (CONFIG_XDP_SOCKETS) section)

The process main thread behavior depends of two operational modes:

• in MODE_TX a xsk_socket__create AF_XDP socket is opened to transmit -n <total number> of egress packets each -l <in byte incl. headers> length in size to a given interface, by calling afxdp_send_pkt() with cyclic-timer -y <nanoseconds tx-interval> under different configurations:

  • with -s fallback onto generic XDP sk_buff Ethernet driver support.

  • with -c activate native XDP XDP_COPY bind flags

  • with -z activate native XDP XDP_ZERO_COPY bind flags

  • with -zw activate native XDP XDP_ZERO_COPY and XDP_USE_NEED_WAKEUP bind flag usually leads to better performance especially if you run the application and the driver on the same core, but also if you use different cores for the application and the kernel driver, as it reduces the number of syscalls needed for the TX path.

  • with -zT  -e <nanoseconds cyclic sampling-offset> -o <nanoseconds cyclic tx-offset> enabling the time-based per-packet tx scheduling (TBS), by specifying each xdp_frame LaunchTime as XDP ring tx_desc entry xsk_ring_prod__tx_desc(&xsk->tx_ring, idx)->txtime = tx_timestamp

• in MODE_RX a xsk_socket__create AF_XDP socket is opened to receive all ingress packets from a given interface under different XDP_COPY and XDP_ZERO_COPY bind flags configurations:

  • with -s generic fallback onto sk_buff Ethernet driver.

  • with -c activate XDP_COPY bind flags

  • with -z activate XDP_ZERO_COPY bind flags

  • with -zwp activate XDP_ZERO_COPY XDP_USE_NEED_WAKEUP bind flag usually leads to better performance especially if user space can put buffers on the FILL ring and then call poll() so that the kernel driver can put these buffers on the HW ring and start to receive packets.

To characterize afxdp latency overhead and jitter using synthetic L2/Ethernet packets 64B payload traversing Linux BPF/XDP socket API ie. and Intel® Ethernet kernel XDP driver :

Tip

It is highly recommended to clean previous tc qdisc and tc filter (or ethtool flow-type) setting before any new egress traffic shaping or ingress traffic steering rules.

tc qdisc del dev $IFACE parent root
tc qdisc del dev $IFACE parent ffff:
tc filter del dev $IFACE parent ffff:
ethtool -U $IFACE delete $ID
ip addr flush dev $IFACE
ip neigh flush all dev $IFACE
ip link delete dev $IFACE.vlan

Intel® I210/igbIntel®I225-LM/igcIntel®TGL/stmmacIntel® EHL/stmmac

Important

Intel Ethernet Controller I210 [Springville] XDP/eBPF hw-offload can be set any combined 4 TX/RX queues.

It is highly recommended to turn-off VLAN striping, turn-on ingress steering hwoffload, set 4 hw-queues combine rx/tx queues, and disable eee.

ethtool -K enp1s0 rxvlan off hw-tc-offload on
ethtool --set-eee enp1s0 eee off
ethtool -L enp1s0 combined 4
ethtool -G enp1s0  tx 1024 rx 1024

Application on POSIX SOCKET API w/ .xdp_bind_flags	Linux Traffic Control (TC)	Comments
MODE_TX sock cyclic write XDP_SKB mode VLAN-tag L2-packets ./txrx-tsn -v -i enp1s0 -X -l 64 -q 1 \ --transmit --skb -n 10000 -y 1000000 \ -d 22:bb:22:bb:22:bb > /tmp/igb-afxdp1a-skb.txt & # Assign SCHED_FIFO policy to CPU2 TX_PID=$! taskset -p 2 $TX_PID chrt --fifo -p 90 $TX_PID	802.1Q Egress packets XDP Queue mapping tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 ip link add link enp1s0 name enp1s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done	igb tc...mqprio map is reverted see Earliest TxTime First (NET_SCHED_ETF) QDisc Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map
MODE_TX sock cyclic write XDP_COPY and VLAN-tag L2-packets ./txrx-tsn -v -i enp1s0 -X -l 64 -q 1 \ --transmit --native-copy -n 10000 -y 1000000 \ -d 22:bb:22:bb:22:bb > /tmp/igb-afxdp1a-copy.txt & # Assign SCHED_FIFO policy to CPU2 TX_PID=$! taskset -p 2 $TX_PID chrt --fifo -p 90 $TX_PID	802.1Q Egress packets XDP Queue mapping tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 ip link add link enp1s0 name enp1s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done	igb tc...mqprio map is reverted see Earliest TxTime First (NET_SCHED_ETF) QDisc Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map
MODE_RX cyclic socket read XDP_COPY mode ./txrx-tsn -v -i enp1s0 -X -q 3 \ --receive --native-copy > /tmp/igb-afxdp1b-copy.txt & # Assign SCHED_FIFO policy to CPU2 RX_PID=$! taskset -p 2 $RX_PID chrt --fifo -p 90 $RX_PID	802.1Q Ingress PTP and VLAN prior 1 and 3 packets steering hw-filters tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 tc qdisc replace dev enp1s0 ingress ethtool -U enp1s0 flow-type ether \ proto 0x8100 vlan 0x2000 \ m 0x1FFF action 1 ethtool -U enp1s0 flow-type ether \ proto 0x88f7 queue 2 ethtool -U enp1s0 flow-type ether \ proto 0x8100 vlan 0x6000 \ m 0x1FFF action 3 ip link add link enp1s0 name enp1s0.vlan type vlan id 3	igb tc...mqprio map is reverted see Earliest TxTime First (NET_SCHED_ETF) QDisc igb tc...flower is limited see Packet Classifier (CONFIG_NET_CLS_FLOWER) Set VLAN ID to 3, all traffic fixed

IEC-62541 OPC-UA PubSub (Part4) Linux RX/TX latency measurement

/opt/intel/iotg_tsn_ref_sw/opcua-server workload transmit and receive IEC-62541 OPC-UA PubSub standardized packets using AF_PACKET or AF_XDP sockets API to exercise IEEE 802.1Q-2018 Enhancements for Scheduled Traffic (EST) netlink hw-offloading functions.

The open62541 (commit-id a77b20ff940115266200d31d30d3290d6f2d57bd) application framework and Linux runtime library are used to provide the underlying IEC-62541 (Part4) network layer abstraction.

Important

open62541-iotg_git.bb is a Intel IOTG open62541-forked with extra XDP features and some worked-around into Yocto ROOTFS build recipes.

• Add UADP ETH on AF_XDP UAPI with LaunchTime support:

  • 0001-fix-PubSub-Enable-dynamic-compilation-of-pubsub-exam.patch

  • 0009-fix-PubSub-Fix-ETF-XDP-plugin-buffer-overflow.patch

  • 0002-feature-PubSub-Use-libbpf-for-AF_XDP-receive-update-.patch

  • 0003-feature-PubSub-add-support-for-AF_XDP-transmission.patch

  • 0004-fix-PubSub-XDP-dynamic-compilation.patch

  • 0005-fix-PubSub-update-example-to-set-XDP-queue-flags.patch

  • 0006-test-PubSub-Configuration-used-for-compile-test.patch

  • 0007-feature-PubSub-Add-ETF-LaunchTime-support-for-XDP-tr.patch

  • 0008-fix-PubSub-AF_XDP-RX-release-mechanism-AF_PACKET-com.patch

  • 0010-fix-PubSub-xdp-socket-cleanup-routine.patch

  • 0011-fix-PubSub-fix-null-checking-possible-memleak-klocwo.patch

  • 0012-fix-PubSub-remove-hardcoded-etf-layer-receive-timeou.patch

The multicallback_server.c (commit-id 12e444a2aa40602c606f4e7d93f7ddc8a499aee1) User space programs spawning IEC-62541 Part4 Publisher and/or Subscriber threads creating Linux socket with specified VLAN priority and continuously loops to either send or receive UADP ETH (EtherType 0xb62c) synthetic packets.

Below all JSON entries for IEC-62541 parametric test-specific using UADP ETH (EtherType 0xb62c) L2/Ethernet frame format with traffic-class QoS :

• IEC-62541 mandatory opcua_server parameters JSON entries :

  • publisher_interface [type = String] : Device specific <iface> or [iface2] name

  • subscriber_interface [type = String] : Device specific <iface> or [iface2] name

  • use_xdp [type = Boolean] : False=``AF_PACKET`` or True=``AF_XDP`` socket

  • use_xdp_zc [type = Boolean] : False = XDP_COPY or True = XDP_ZEROCOPY

  • packet_count [type = Int] : number of TX RX packet between 1 to 10000000

  • cycle_time_ns [type = Int] : publishingInterval (ie net cycletime)between 100000 and 5000000

  • polling_duration_ns [type = Int] : polling duration ns between 0 and 10000000

• IEC-62541 rt-threaded (ie. prio-99 SCHED_FIFO) publishers user-parameters JSON entry list:

  • url [type = String] : ie. opc.eth://<dst_mac>

  • pub_id [type = String] : egress testframes pub[id]

  • dataset_writer_id [type = Int] : egress testframes dataset_writer[id] between 0 and 99999

  • writer_group_id [type = Int] : egress testframes writergroup[id] between 0 and 99999

  • early_offset_ns [type = Int] : writer task sampling offset in ns between 0 and 1000000

  • publish_offset_ns [type = Int] : writer task publish offset in ns between 0 and 10000000

  • socket_prio [type = Int] : L2 frame VLANPCP field set between 0 and 7

  • two_way_data [type = Boolean] : single or round-trip scenario

  • cpu_affinity [type = Int] : cpu core ie taskset between 0 and 3

  • xdp_queue [type = Int] : queue[id] xdp_prog bpf map. between 1 and 3

• IEC-62541 rt-threads (ie. prio-99 SCHED_FIFO) subscribers user-parameters JSON entry list :

  • url [type = String] : ie. opc.eth://<dst_mac>

  • sub_id [type = String] : sub_id between 0 and 99999

  • subscribed_pub_id [type = Int] : ingress testframes pub[id] between 0 and 99999

  • subscribed_dataset_writer_id [type = Int] : ingress testframes dataset_writer[id] between 0 and 99999

  • subscribed_writer_group_id [type = Int] : ingress testframes writergroup[id] between 0 and 99999

  • offset_ns [type = Int] : reader task sampling offset in ns between 0 and 10000000

  • subscriber_output_file [type = String] : Record timestamp output file between and

  • two_way_data [type = Boolean] : single or round-trip scenario between and

  • cpu_affinity [type = Int] : cpu core ie taskset between 0 and 3

  • xdp_queue [type = Int] : queue[id] xdp_prog bpf map between -1 and 3

A test engineer aiming to measure IEC62541 communication timing-accuracy in a Talker-Listener RT-traffic cyclic scenario over Device A and B each equipped with single-port Intel(R) Ethernet Controller either discrete PCIe I210 or I225-LM or integrated-TGL TSN endpoint, as illustrated below :

• Create a /opt/intel/iotg_tsn_ref_sw/json/tgl/opcua-pkt1a.json file on Device A.

• Run a single IEC62541 publisher cyclic 1ms communication rt-thread registering a AF_PACKET socket onto the enp0s30f4 linux netdev.

  /opt/intel/iotg_tsn_ref_sw# ./opcua-server json/tgl/opcua-pkt1a.json
  

• Create a /opt/intel/iotg_tsn_ref_sw/json/tgl/opcua-pkt1b.json file on Device B.

• Run a single IEC62541 subscriber cyclic 1ms communication rt-thread, registering a AF_PACKET socket onto the enp0s30f4 linux netdev.

  /opt/intel/iotg_tsn_ref_sw# ./opcua-server json/tgl/opcua-pkt1b.json
  

With Dual Intel(R) Ethernet Controller ECI nodes, a test engineer aiming to measure IEC62541 communication timing-accuracy in more complex Talker-Listener scenario cyclic RT-traffic for as illustrated below with loopback testing :

In the table below more sample configuration to characterize Unidirectional or Loopback E2E latency and jitter of IEC-62541 part4 UADP ETH packets (EtherType 0xb62c) traversing Linux AF_PACKET or AF_XDP Socket API, netlink and Intel® Ethernet kernel driver :

Tip

It is highly recommended to remove previous tc qdisc, tc filter (or ethtool flow-type) and xdp to before setting any new egress traffic shaping or ingress traffic steering rules.

tc qdisc del dev $IFACE parent root
tc qdisc del dev $IFACE parent ffff:
tc filter del dev $IFACE parent ffff:
ip link set dev $IFACE xdp off
ethtool -U $IFACE delete $ID
ip addr flush dev $IFACE
ip neigh flush all dev $IFACE
ip link delete dev $IFACE.vlan

Intel® I210/igbIntel®I225-LM/igcIntel®TGL/stmmacIntel® EHL/stmmac

Important

Intel Ethernet Controller I210 [Springville] Traffic Shaping hw-offload is ONLY available on hw-queues 0 and 1, where as XDP/eBPF hw-offload can be set any combined 4 TX/RX queues.

It is highly recommended to turn-off VLAN Stripping and set hw-queues combined queue, Minimum is 4 combined rx/tx.

ethtool -K enp1s0 rxvlan off hw-tc-offload on
ethtool --set-eee enp1s0 eee off
ethtool -L enp1s0 combined 4
ethtool -G enp1s0  tx 1024 rx 1024
ethtool -K enp2s0 rxvlan off hw-tc-offload on
ethtool --set-eee enp2s0 eee off
ethtool -L enp2s0 combined 4
ethtool -G enp2s0  tx 1024 rx 1024

IEC-62541 Application JSON configurations samples	Linux Traffic Control (TC)	Comments
single-port AF_PACKET 1ms cyclic SCM_TXTIME Talker VLAN TCI header (Prio=3, id=3) { "opcua_server": { "publisher_interface": "enp1s0", "subscriber_interface": "enp1s0", "use_xdp": false, "packet_count": 1000000, "cycle_time_ns": 1000000, "polling_duration_ns": 0, "publishers": { "pub1": { "url": "opc.eth://22-bb-22-bb-22-bb:3.3", "pub_id": 2234, "dataset_writer_id": 62541, "writer_group_id": 101, "early_offset_ns": 400000, "publish_offset_ns": 900000, "socket_prio": 3, "two_way_data": false, "cpu_affinity": 3, "xdp_queue": -1 } }, "subscribers": {} }, }	802.1Q Egress packets Timed-Based Sched.(TBS) on ETF Queue-disciplines tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 tc qdisc replace dev enp1s0 parent 100:1 etf \ clockid CLOCK_TAI delta 400000 offload ip link add link enp1s0 name enp1s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done	igb tc...mqprio map is reverted see Earliest TxTime First (NET_SCHED_ETF) QDisc Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map igb tc...taprio map is reverted see Time Aware Priority (NET_SCHED_TAPRIO) QDisc igb tc...flower is limited see Packet Classifier (CONFIG_NET_CLS_FLOWER)
single-port AF_PACKET 1ms cyclic TIMER_ABSTIME Talker VLAN TCI header (Prio=3, id=3) { "opcua_server": { "publisher_interface": "enp1s0", "subscriber_interface": "enp1s0", "use_xdp": false, "packet_count": 1000000, "cycle_time_ns": 1000000, "polling_duration_ns": 0, "publishers": { "pub1": { "url": "opc.eth://22-bb-22-bb-22-bb:3.3", "pub_id": 2234, "dataset_writer_id": 62541, "writer_group_id": 101, "early_offset_ns": 200000, "publish_offset_ns": 350000, "socket_prio": 3, "two_way_data": false, "cpu_affinity": 3, "xdp_queue": -1 } }, "subscribers": {} }, }	802.1Q Egress packets w/ TAPRIO tx-assisted GCL on ETF Queue-disciplines BASE=$(expr $(date +%s) + 5)000000000 tc -d qdisc replace dev enp1s0 parent root handle 100 taprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@0 1@0 1@0 \ base-time $BASE \ sched-entry S 01 500000 \ sched-entry S 0e 500000 \ flags 0x1 txtime-delay 100000 clockid CLOCK_TAI tc qdisc replace dev enp1s0 parent 100:1 etf \ clockid CLOCK_TAI delta 50000 offload skip_sock_check ip link add link enp1s0 name enp1s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done
single-port AF_PACKET 1ms cyclic Listener rt-thread { "opcua_server": { "publisher_interface": "enp1s0", "subscriber_interface": "enp1s0", "use_xdp": false, "packet_count": 1000000, "cycle_time_ns": 1000000, "polling_duration_ns": 0, "publishers": {}, "subscribers": { "sub1": { "url": "opc.eth://22-bb-22-bb-22-bb", "sub_id": 0, "subscribed_pub_id": 2234, "subscribed_dataset_writer_id": 62541, "subscribed_writer_group_id": 101, "offset_ns": 1000, "subscriber_output_file": "afpkt-rxtstamps.txt", "two_way_data": false, "cpu_affinity": 3, "xdp_queue": -1 } } }, }	802.1Q Ingress packets VLAN TCI steering hw-filters tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 tc qdisc replace dev enp1s0 ingress ethtool -U enp1s0 flow-type ether \ proto 0x8100 vlan 0x2000 \ m 0x1FFF action 1 ethtool -U enp1s0 flow-type ether \ proto 0x88f7 queue 2 ethtool -U enp1s0 flow-type ether \ proto 0x8100 vlan 0x6000 \ m 0x1FFF action 3 ip link add link enp1s0 name enp1s0.vlan type vlan id 3
dual-port AF_PACKET 2ms cyclic SCM_TXTIME Talker and loopback Listener.VLAN TCI header (Prio=3, id=3) { "opcua_server": { "publisher_interface": "enp1s0", "subscriber_interface": "enp2s0", "use_xdp": false, "packet_count": 500000, "cycle_time_ns": 2000000, "polling_duration_ns": 0, "publishers": { "pub1": { "url": "opc.eth://22-bb-22-bb-22-bb:3.3", "pub_id": 2234, "dataset_writer_id": 62541, "writer_group_id": 101, "early_offset_ns": 400000, "publish_offset_ns": 100, "socket_prio": 3, "two_way_data": false, "cpu_affinity": 2, "xdp_queue": -1 } }, "subscribers": { "sub1": { "url": "opc.eth://aa-11-aa-11-aa-11", "sub_id": 0, "subscribed_pub_id": 2234, "subscribed_dataset_writer_id": 62541, "subscribed_writer_group_id": 101, "offset_ns": 1050000, "subscriber_output_file": "afpkt-rxtstamps.txt", "two_way_data": true, "cpu_affinity": 3, "xdp_queue": -1 } } }, }	802.1Q Egress packets Timed-Based Sched.(TBS) on ETF Queue-disciplines 802.1Q Ingress packets VLAN TCI steering hw-filters tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 tc qdisc replace dev enp1s0 parent 100:1 etf \ clockid CLOCK_TAI delta 400000 offload tc qdisc replace dev enp2s0 ingress ethtool -U enp2s0 flow-type ether \ proto 0x8100 vlan 0x2000 \ m 0x1FFF action 1 ethtool -U enp2s0 flow-type ether \ proto 0x88f7 queue 2 ethtool -U enp2s0 flow-type ether \ proto 0x8100 vlan 0x6000 \ m 0x1FFF action 3 tc qdisc add dev enp2s0 parent root handle 100 mqprio \ num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \ queues 1@0 1@1 1@2 1@3 hw 0 ip link add link enp2s0 name enp2s0.vlan type vlan id 3 ip link add link enp1s0 name enp1s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done	igb tc...mqprio map are reverted see Earliest TxTime First (NET_SCHED_ETF) QDisc igb tc...flower is limited see Packet Classifier (CONFIG_NET_CLS_FLOWER) Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map
dual-port AF_PACKET 2ms cyclic Listener and loopback Listener.VLAN TCI header (Prio=3, id=3) { "opcua_server": { "subscriber_interface": "enp1s0", "publisher_interface": "enp2s0", "use_xdp": false, "packet_count": 500000, "cycle_time_ns": 2000000, "polling_duration_ns": 0, "publishers": { "pub1": { "url": "opc.eth://aa-11-aa-11-aa-11:3.3", "pub_id": 2234, "dataset_writer_id": 62541, "writer_group_id": 101, "early_offset_ns": 400000, "publish_offset_ns": 1000100, "socket_prio": 3, "two_way_data": true, "cpu_affinity": 2, "xdp_queue": -1 } }, "subscribers": { "sub1": { "url": "opc.eth://22-bb-22-bb-22-bb", "sub_id": 0, "subscribed_pub_id": 2234, "subscribed_dataset_writer_id": 62541, "subscribed_writer_group_id": 101, "offset_ns": 50000, "subscriber_output_file": "afpkt-rxtstamps.txt", "two_way_data": false, "cpu_affinity": 3, "xdp_queue": -1 } } }, }	802.1Q Egress packets Timed-Based Sched.(TBS) on ETF Queue-disciplines 802.1Q Ingress packets VLAN TCI steering w/ hw-filters tc qdisc add dev enp2s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 tc qdisc replace dev enp2s0 parent 100:1 etf \ clockid CLOCK_TAI delta 400000 offload tc qdisc replace dev enp1s0 ingress ethtool -U enp1s0 flow-type ether \ proto 0x8100 vlan 0x2000 \ m 0x1FFF action 1 ethtool -U enp1s0 flow-type ether \ proto 0x88f7 queue 2 ethtool -U enp1s0 flow-type ether \ proto 0x8100 vlan 0x6000 \ m 0x1FFF action 3 tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 ip link add link enp1s0 name enp1s0.vlan type vlan id 3 ip link add link enp2s0 name enp2s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done
dual-port AF_XDP (XDP_COPY) 400us cyclic TIMER_ABSTIME and loopback Listener. VLAN TCI header (Prio=2, id=3) { "opcua_server": { "publisher_interface": "enp1s0", "subscriber_interface": "enp2s0", "use_xdp": true, "use_xdp_zc": false, "packet_count": 1000000, "cycle_time_ns": 400000, "polling_duration_ns": 0, "publishers": { "pub1": { "url": "opc.eth://22-bb-22-bb-22-bb:3.2", "pub_id": 2234, "dataset_writer_id": 62541, "writer_group_id": 101, "early_offset_ns": 100000, "publish_offset_ns": 100, "socket_prio": 3, "two_way_data": false, "cpu_affinity": 2, "xdp_queue": 2 } }, "subscribers": { "sub1": { "url": "opc.eth://aa-11-aa-11-aa-11:3.2", "sub_id": 11, "subscribed_pub_id": 2235, "subscribed_dataset_writer_id": 62541, "subscribed_writer_group_id": 101, "offset_ns": 200000, "subscriber_output_file": "afxdp-rxtstamps.txt", "two_way_data": true, "cpu_affinity": 3, "xdp_queue": 2 } } }, }	802.1Q Egress packets VLAN affinity 802.1Q Ingress packets VLAN TCI steering w/ hw-filters tc qdisc add dev enp1s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 tc qdisc replace dev enp2s0 ingress ethtool -U enp2s0 flow-type ether \ proto 0x8100 vlan 0x2000 \ m 0x1FFF action 1 ethtool -U enp2s0 flow-type ether \ proto 0x88f7 queue 3 ethtool -U enp2s0 flow-type ether \ proto 0x8100 vlan 0x4000 \ m 0x1FFF action 2 tc qdisc add dev enp2s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 ip link add link enp2s0 name enp2s0.vlan type vlan id 3 ip link add link enp1s0 name enp1s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp1s0.vlan type vlan egress-qos-map $j:$j ; done	igb tc...mqprio map are reverted see Earliest TxTime First (NET_SCHED_ETF) QDisc igb tc...flower is limited see Packet Classifier (CONFIG_NET_CLS_FLOWER) Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map
dual-port AF_XDP (XDP_COPY) 400us cyclic TIMER_ABSTIME and loopback Listener. VLAN TCI header (Prio=2, id=3) { "opcua_server": { "publisher_interface": "enp1s0", "subscriber_interface": "enp2s0", "use_xdp": true, "use_xdp_zc": false, "packet_count": 1000000, "cycle_time_ns": 400000, "polling_duration_ns": 0, "publishers": { "pub1": { "url": "opc.eth://aa-11-aa-11-aa-11:3.2", "pub_id": 2235, "dataset_writer_id": 62541, "writer_group_id": 101, "early_offset_ns": 50000, "publish_offset_ns": 250000, "socket_prio": 2, "two_way_data": true, "cpu_affinity": 2, "xdp_queue": 2 } }, "subscribers": { "sub1": { "url": "opc.eth://22-bb-22-bb-22-bb:3.2", "sub_id": 22, "subscribed_pub_id": 2234, "subscribed_dataset_writer_id": 62541, "subscribed_writer_group_id": 101, "offset_ns": 5000, "subscriber_output_file": "afxdp-rxtstamps.txt", "two_way_data": false, "cpu_affinity": 3, "xdp_queue": 2 } } }, }	802.1Q Egress packets VLAN affinity 802.1Q Ingress packets VLAN TCI steering w/ hw-filters tc qdisc add dev enp2s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 tc qdisc replace dev enp2s0 ingress ethtool -U enp2s0 flow-type ether \ proto 0x8100 vlan 0x2000 \ m 0x1FFF action 1 ethtool -U enp2s0 flow-type ether \ proto 0x88f7 queue 3 ethtool -U enp2s0 flow-type ether \ proto 0x8100 vlan 0x4000 \ m 0x1FFF action 2 tc qdisc add dev enp2s0 parent root handle 100 mqprio \ num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 1@0 1@1 1@2 1@3 hw 0 ip link add link enp1s0 name enp1s0.vlan type vlan id 3 ip link add link enp2s0 name enp2s0.vlan type vlan id 3 for j in `seq 0 7`; \ do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done

Note

As latest ECI code-freeze being declared with multicallback_server.c (commit-id 12e444a2aa40602c606f4e7d93f7ddc8a499aee1) the patch to add use_xdp_zc

Latest to multicallback_server.c (master) already adds use_xdp_zc and use_xdp_skb.

Optionally, a test engineer may desire to redirect DUT "subscriber_output_file": "afpkt-rxtstamps.txt" recorded timestamps to a visual plot iec62541-opcua-plot.png on a remote host machine. Please find below a iec62541-opcua-plot.gnu multi XY gnuplot examples:

Click to toggle visibility

reset

  set terminal pngcairo size 1920,1080
  set output 'iec62541-opcua-plot.png'

  set size 1,1
  set datafile missing'?' # Skip invalid/missing data
  set locale "en_US.utf8" # Set to en_US locale for thousands seperator

  set tmargin 5
  set bmargin 5
  set lmargin 15
  set rmargin 5

  set yrange [0:YMAX] # up to 1ms latency

  set grid
  set key center tmargin
  set border

  set style fill solid 0.25
  set boxwidth 0.5
  # set logscale y 10

  set title PLOT_TITLE
  set xlabel "Packet count"
  set ylabel "Latency in nanoseconds"

  #set multiplot layout 1

  stats FILENAME u 1 name "a" nooutput

  min_us = a_min/1000
  max_us = a_max/1000
  avg_us = a_mean/1000

  set label front sprintf("Minimum: %d us", min_us) at graph 0.01, graph 0.95
  set label front sprintf("Maximum: %d us", max_us) at graph 0.01, graph 0.90
  set label front sprintf("Average: %d us", avg_us) at graph 0.01, graph 0.85

  plot FILENAME \
    using ($1) title "Legacy socket"  lc rgb "red" w points

  unset label


  unset multiplot
  unset output

  #Replot to GUI
  set terminal x11 size 1720,980
  set output
  #set multiplot layout 1
  plot FILENAME \
    using ($1) title "Legacy socket"  lc rgb "red" w points

pause 10 #in case some one forgets to add -p

gnuplot -e "FILENAME='afpkt-rxtstamps.txt"'; YMAX=10000000; PLOT_TITLE='Transmission latency from TX User-space to RX User-space (AF-PACKET - unidirectional)'" $DIR/iec62541-opcua-plot.gnu -p 2> /dev/null &

Optionally, a test engineer may desire to introduce noisy-neighbor client-server to stress DUTs traffic-class QoS robustness :

1. To setup a dummy TCP/IPv4 server

iperf3 -s -B <IPv4> -i 10 -A 0 -1 2&> /dev/null'

1. To instantiate a client that concurrently to stress the data link on Best-effort VLANPCP=0 traffic-class.

iperf3 -c <IPv4> -u -b 200M -l 1440 -f m -i 10 -t 30000 -A 0 2&> /dev/null'

Sanity-Check iotg_tsn_ref_sw Testing

To Check :

• Enhancements for Scheduled Traffic (EST) (formerly known as 802.1Qbv) Deterministic scheduling of Ethernet TSN Domain

Important

No PCI-400-TSN TSN switch can be used in Sanity-Checking ECI images with IoTG pre-canned ./run.sh testrunner scripts. ONLY Direct 802.1as time-sync (see Linuxptp stack 802.1AS gPTP Profile) is established such as only Linux TSN endpoint port act as Grand-master (GM) Clock.

Sanity-Check #1: IEEE 802.1AS Time Synchronization Quality Measurement

The following section is applicable to:

Note

For the remainder of the following steps, user shall substitute $PLAT and $IFACE with the mounted interface corresponding to Intel® tgl|i210|i225 Industrial Ethernet TSN devices.

1. [Device A] Run the setup script to configure IP and MAC address, start ptp4l, enable pulse-per-second and external timestamping.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE tsq1a setup
   

2. [Device B] Run the setup script to configure IP and MAC address, start ptp4l and enable external timestamping.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE tsq1b setup
   

3. [Device A] Start the talker and listener.

   $ ./run.sh $PLAT $IFACE tsq1a run
   

4. [Device B] Immediately after step 3, start the talker.

   $ ./run.sh $PLAT $IFACE tsq1b run
   

5. [Device A] The externally-triggered timestamps difference from Device A -vs- B shall be record successfully into a tsq-listener-data.txt text file.

   The two-column records should report a difference within the +/- 50 nanoseconds range, considering Device A & B time-sync record stops after 50 seconds.

Sanity-Check #2: Single-port transmission over AF_PACKET -vs- AF_XDP socket with IEEE 802.1Q-2018 EST

Optionally, a test engineer may desire to reuse/modify IoTG pre-canned two-stage testrunner ./run.sh shell scripts that calls single-ended vs1a.sh & vs1b.sh underlying scripts to automate the P2P time synchronization init/teardown condition, the Device A talker time-bounded packets to Device B listener timestamped UDP framed records :

1. Setup stage run.sh <plat> <iface> setup

2. Run stage run.sh <plat> <iface> run which run back-to-back both AF_PACKET and AF_XDP subtest

The following section is applicable to:

Note

For the remainder of the following steps, user shall substitute $PLAT and $IFACE with the mounted interface corresponding to Intel® tgl|i210|i225 Industrial Ethernet TSN devices.

1. [Device A] Run the setup script to configure IP and MAC address, start clock synchronization and setup TAPRIO qdisc.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE vs1b setup
   

2. [Device B] Run the setup script to configure IP and MAC address, start clock synchronization and setup ingress qdiscs.

   cd /opt/intel/iotg-tsn-ref-sw/
   ./run.sh $PLAT $IFACE vs1b setup
   

3. [Device B] Start listening for packets using AF_PACKET, it will automatically re-start the application using AF_XDP socket after it has completed. This duration will be printed on the shell line: Phase xxx (N-duration seconds)

   $ ./run.sh $PLAT $IFACE vs1b run
   

4. [Board A] Immediately after step 3, start transmitting packets using AF_PACKET. it will automatically re-start the application and transmit using AF_XDP socket after 30 seconds.

   $ ./run.sh $PLAT $IFACE vs1a run
   

5. [Device B] The latency report of every-packets sent from [Device A] to [Device B] shall be record successfully into afpkt-traffic.txt and afxdp-traffic.txt text file.

Sanity-Check #3: Single-port IEC-62541 OPC-UA PubSub (Part4) transmission over Linux AF_PACKET socket with IEEE 802.1Q-2018 EST

Optionally, a test engineer may desire to reuse IoTG pre-canned testrunner shell ./run.sh scripts that calls single-ended vs1a.sh and vs1b.sh underlying scripts to automate with preset opcua-* JSON templates files for IEC-62541 parametric test (json/<plat>/*.json.i) specific to a Intel® Ethernet Controller :

• opcua-pkt1 are templates for bounded latency transmission/reception testing over Linux AF_PACKET sockets APIs with IEC-62541 traffic L2 frames (the open62541 OPC-UA-based library) using Intel® Industrial Ethernet hw-offload capabilities on a single-port . This application uses JSON files to configure a single TSN Ethernet either egress or ingress port.

The following section is applicable to:

Note

For the remainder of the following steps, user shall substitute $PLAT and $IFACE with the mounted interface corresponding to Intel® tgl|i225|i210 Industrial Ethernet TSN devices.

1. [Device A] Run the init stage to configure cpu-core affinity.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   ./run.sh $PLAT $IFACE opcua-pkt1a init
   

2. [Device A] Run the setup script to configure IP and MAC address, start clock synchronization and setup TAPRIO qdisc.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE opcua-pkt1a setup
   

3. [Device B] Run the init stage to configure cpu-core affinity.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE opcua-pkt1b init
   

4. [Device B] Run the setup script to configure IP and MAC address, start clock synchronization and setup ingress qdiscs Successfully

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE opcua-pkt1b setup
   

5. [Device B] Successfully Start IEC-62541 traffic subscriber rt-thread and the Time-Aware traffic-shaper using AF_PACKET socket, with recorded into afpkt-rxtstamps.txt reporting every incoming packets latency in textual-format:

   $ ./run.sh $PLAT $IFACE opcua-pkt1b run
   

6. [Device A] Start IEC-62541 traffic publisher rt-thread and on Time-Aware traffic-shaper using AF_PACKET socket.

   $ ./run.sh $PLAT $IFACE opcua-pkt1a run
   

7. [Device B] The latency report of every-packets sent from Device A to Device B shall be record successfully into afpkt-rxtstamps.txt text file.

Sanity-Check #4: dual-port IEC-62541 OPC-UA PubSub (Part4) loopback over Linux AF_PACKET socket with IEEE 802.1Q-2018 EST

Optionally, a test engineer may desire to reuse IoTG pre-canned testrunner shell ./run.sh scripts that calls single-ended vs1a.sh and vs1b.sh underlying scripts to automate with preset opcua-* JSON templates files for IEC-62541 parametric test (json/<plat>/*.json.i) specific to a Intel® Ethernet Controller :

• opcua-pkt2*, opcua-pkt3* are templates for bounded E2E round-trip latency transmission/reception over Linux AF_PACKET sockets APIs with IEC-62541 traffic L2 frames (the open62541 OPC-UA-based library) using Intel® Industrial Ethernet hw-offload capabilities on a dual-port . This application uses JSON files to configure a dual TSN Ethernet egress and ingress ports. Test engineer may also choose to disable noisy-neighbors PCP=0 traffic respectively on opcua-*2 or opcua-*3 .

The following section is applicable to:

Note

For the remainder of the following steps, user shall substitute $PLAT, $IFACE and $IFACE2 with the mounted interface corresponding to Intel® dual-i210 Industrial Ethernet TSN devices.

1. [Device A] Run the init stage to configure cpu-core affinity.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-pkt2a init
   

2. [Device A] Run the setup script to configure IP and MAC address, start clock synchronization and setup TAPRIO qdisc.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-pkt2a setup
   

3. [Device B] Run the init stage to configure cpu-core affinity.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-pkt2b init
   

4. [Device B] Run the setup script to configure IP and MAC address, start clock synchronization and setup ingress qdiscs.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-pkt2b setup
   

5. [Device B] Successfully Start IEC-62541 rt-threads Subscriber and Publisherloopback copy with the Time-Aware traffic-shaper using AF_PACKET socket, including recorded into afpkt-rxtstamps.txt reporting every incoming packets latency in textual-format:

   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-pkt2b run
   

6. [Device A] Initiate IEC-62541 traffic Publisher(root) and Subscriber rt-threads and on Time-Aware traffic-shaper using AF_PACKET socket.

   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-pkt2a run
   

7. [Device B] The latency report of every-packets sent from Device A to Device B shall be record successfully into afpkt-rxtstamps.txt text file.

Sanity-Check #5: dual-port IEC-62541 OPC-UA PubSub (Part4) loopback over Linux AF_XDP with IEEE 802.1Q-2018 EST

Optionally, a test engineer may desire to reuse IoTG pre-canned testrunner shell ./run.sh scripts that calls single-ended vs1a.sh and vs1b.sh underlying scripts to automate with preset opcua-* JSON templates files for IEC-62541 parametric test (json/<plat>/*.json.i) specific to a Intel® Ethernet Controller :

• opcua-xdp2*, opcua-xdp3* verifies that bounded E2E round-trip latency transmission/reception achieved over Linux AF_XDP sockets APIs with IEC-62541 traffic L2 frames (the open62541 OPC-UA-based library) using Intel® Industrial Ethernet hw-offload capabilities on a dual-port . This application uses JSON files to configure a dual TSN Ethernet egress and ingress ports. Test engineer can choose to disable noisy-neighbors PCP=0 traffic respectively on opcua-*2 or opcua-*3

The following section is applicable to:

Note

For the remainder of the following steps, user shall substitute $PLAT, $IFACE and $IFACE2 with the mounted interface corresponding to Intel® dual-i210 Industrial Ethernet TSN devices.

1. [Device A] Run the init stage to configure cpu-core affinity.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-xdp2a init
   

2. [Device A] Run the setup script to configure IP and MAC address, start clock synchronization and setup TAPRIO qdisc.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-xdp2a setup
   

3. [Device B] Run the init stage to configure cpu-core affinity.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-xdp2b init
   

4. [Device B] Run the setup script to configure IP and MAC address, start clock synchronization and setup ingress qdiscs.

   $ cd /opt/intel/iotg-tsn-ref-sw/
   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-xdp2b setup
   

5. [Device B] Successfully Start IEC-62541 rt-threads Subscriber and Publisherloopback copy with the Time-Aware traffic-shaper using AF_XDP socket, including recorded into afxdp-traffic.txt reporting every incoming packets latency in textual-format:

   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-xdp2b run
   

6. [Device A] Initiate IEC-62541 traffic Publisher(root) and Subscriber rt-threads and on Time-Aware traffic-shaper using AF_XDP socket.

   $ ./run.sh $PLAT $IFACE $IFACE2 opcua-xdp2a run
   

7. [Device B] The latency report of every-packets sent from Device A to Device B shall be record successfully into afxdp-rxtstamps.txt" text file.