Attention

You are viewing an older version of the documentation. The latest version is v3.3.

Ethernet TSN Benchmark

About TSN TX/RX latency benchmark

The /opt/intel/iotg_tsn_ref_sw/txrx-tsn is a benchmark tool to check Ethernet endpoints ingress and egress Traffic-class QoS under IEEE 802.1Q-2018 Enhancements for Scheduled Traffic (EST) polices with various Intel® Ethernet Controller TSN hw-offloading capabilities.

It simulates control cell and monitoring applications establishing low-jitter Layer-2 communication through Intel® Ethernet Controllers via direct Ethernet cable connection, to minimize packet travel time impact on the benchmark results.

../../../_images/txrx-tsn-overview.png

This tool records each timestamps of an cyclical L2/Ethernet packets 64B payload (EtherType=0xb62c) to characterize latency overhead and jitter traversing Linux Socket network API netdev to and from the Intel® Ethernet controller kernel driver.

The AF_PACKET socket subtest txrx-afpkt.c Linux user-space programs spawn a communication thread which creates a socket on a specified priority and continuously loops to either send or receive packets, running main thread behavior depends of two operational modes:

  • in MODE_TX a SOCK_DGRAM socket is opened to transmit -n <total number> of egress packets each -l <in byte incl. headers> length in size to a given interface, by calling afpkt_send_thread() with cyclic-timer -y <nanoseconds tx-interval> under different configurations:

    • Optionally, with -T  -e <nanoseconds cyclic sampling-offset> -o <nanoseconds cyclic tx-offset> enabling the time-based per-packet tx scheduling (TBS), by calling afpkt_send_thread_etf() see Earliest TxTime First (NET_SCHED_ETF) QDisc

  • in MODE_RX a SOCK_RAW socket is opened to receive all ingress packets from a given interface. Optionally -h hw-timestamping for all ingress packets using time stamping policy at the network driver level with the SIOCSHWTSTAMP

The AF_XDP socket subtest txrx-afxdp.c Linux user-space programs spawn a communication thread which creates a AF_XDP socket set to specified priority queues and continuously loops to either send or receive packets (see AF_XDP Socket (CONFIG_XDP_SOCKETS) section) running process main thread behavior depends of two operational modes:

  • in MODE_TX a xsk_socket__create AF_XDP socket is opened to transmit -n <total number> of egress packets each -l <in byte incl. headers> length in size to a given interface, by calling afxdp_send_pkt() with cyclic-timer -y <nanoseconds tx-interval> under different configurations:

    • with -s fallback onto generic XDP sk_buff Ethernet driver support.

    • with -c activate native XDP_COPY bind flags

    • with -z activate native XDP_ZERO_COPY bind flags

    • with -zw activate native XDP_ZERO_COPY and XDP_USE_NEED_WAKEUP bind flag usually leads to better performance especially if you run the application and the driver on the same core, but also if you use different cores for the application and the kernel driver, as it reduces the number of syscalls needed for the TX path.

  • in MODE_RX a xsk_socket__create AF_XDP socket is opened to receive all ingress packets from a given interface under different XDP_COPY and XDP_ZERO_COPY bind flags configurations:

    • with -s generic fallback onto sk_buff Ethernet driver.

    • with -c activate XDP_COPY bind flags

    • with -z activate XDP_ZERO_COPY bind flags

    • with -zwp activate XDP_ZERO_COPY XDP_USE_NEED_WAKEUP bind flag usually leads to better performance especially if user space can put buffers on the FILL ring and then call poll() so that the kernel driver can put these buffers on the HW ring and start to receive packets.

Installing TSN TX/RX latency benchmark

This component can be installed from the ECI repository. Setup the ECI repository, then perform either of the following commands to install this component:

logo_debian logo_ubuntu

Install from meta-package
$ sudo apt install eci-connectivity-tsn
Install from individual Deb package
$ sudo apt install iotg-txrx-tsn

Prerequisites

  1. Make sure the minimal set of systemd services has configured the TSN end-station on the desired Linux network interface.

    $ systemctl status eci-tsn-wait-sync-8021as@enp1s0.service

    The expected output should be similar after < 15 seconds to the following:

    Created symlink /etc/systemd/system/multi-user.target.wants/eci-tsn-wait-sync-8021as@enp1s0.service ? /lib/systemd/system/eci-tsn-wait-sync-8021as@.service.
? eci-tsn-wait-sync-8021as@enp1s0.service - ECI: Wait for clocks synchronization
   Loaded: loaded (/lib/systemd/system/eci-tsn-wait-sync-8021as@.service; enabled; vendor preset: enabled)
   Drop-In: /usr/lib/systemd/system/service.d
            └─10-override-protect-proc.conf
   Active: active (exited) since Tue 2018-08-28 08:59:30 UTC; 27ms ago
   Process: 24138 ExecStart=/usr/bin/bash /usr/libexec/eci-tsn-wait-timesync.sh enp1s0 15 (code=exited, status=0/SUCCESS)
   Main PID: 24138 (code=exited, status=0/SUCCESS)
      CPU: 3ms

eci-intel-23f0 bash[24139]: rt tstamp:          1535446769376683711
eci-intel-23f0 bash[24139]: tai tstamp:         1535446806376683783
eci-intel-23f0 bash[24139]: phc tstamp:         1535446806376685317
eci-intel-23f0 bash[24139]: rt latency:         30
eci-intel-23f0 bash[24139]: tai latency:        30
eci-intel-23f0 bash[24139]: phc latency:        2080
eci-intel-23f0 bash[24139]: phc-rt delta:       37000001606
eci-intel-23f0 bash[24139]: phc-tai delta:      1534
eci-intel-23f0 bash[24139]: Clocks on this system are synchronized :)
eci-intel-23f0 systemd[1]: Finished ECI: Wait for clocks synchronization.

  2. Setup Network Manager configuration via systemd.network, systemd.link and systemd.device unit files using eci-tsn-netlink@.service - 802.1Q-2018 TSN Link-layer Service for Intel® Ethernet Controllers

    Note

    Alternatively, user manual setting for ref:TSN TX/RX latency micro-benchmark <iotg_tsn_ref_txrx> are available under the Linux Traffic Control (TC) column.

    You may also have to clean tc qdisc and ethtool flow-type (or tc filter) it is recommended in between each user-configured netlink configuration

    $ tc qdisc del dev $IFACE parent root
$ tc qdisc del dev $IFACE parent ffff:
$ tc filter del dev $IFACE parent ffff:
$ ethtool -U $IFACE delete $ID
$ ip addr flush dev $IFACE
$ ip neigh flush all dev $IFACE
$ ip link delete dev $IFACE.vlan

Execute TSN TX/RX latency benchmark

The IEEE 802.1AS generalized precision time protocol (gPTP) must be establish using either Direct 802.1as manual configuration or systemd time-syschronization ECI service such as one Ethernet port act as Grand-Master (GM) Clock (below enp1s0).

To characterize af_packet latency overhead and jitter using synthetic L2/Ethernet packets 64B payload traversing Linux Socket API, netlink QDisc, netdev Intel® Ethernet kernel driver :

Important

Intel® Ethernet Controller I225 [Foxville] Software User Manual .

It is highly recommended to turn-off VLAN stripping and set hw-queues combined queue, Minimum is 4 combined rx/tx. For example:

$ ethtool -K enp2s0 rxvlan off hw-tc-offload on
$ ethtool --set-eee enp2s0 eee off
$ ethtool -L enp2s0 combined 4
$ ethtool -G enp2s0  tx 1024 rx 1024

It is highly recommended to set Network Interrupts Affinity to CPU. For example:

$ network-irq-affinity -i enp2s0-TxRx-3/1 -i enp2s0-TxRx-2/1 -v

For example to affinitize Hw-Queue 3 (e.g. Real-Time Traffic-Class) to CPU 1, Hw-Queue 2 (e.g. PTPv2) to CPU 1 and remaining Hw-Queues (e.g. Best Effort) to core 0

network-irq-affinity: Assigning enp2s0-TxRx-3 on IRQ 136 to CPU 1
network-irq-affinity: Assigning enp2s0-TxRx-2 on IRQ 135 to CPU 1
network-irq-affinity: Assigning enp2s0-TxRx-1 on IRQ 134 to CPU 0
network-irq-affinity: Assigning enp2s0-TxRx-0 on IRQ 133 to CPU 0

Application w/ Linux POSIX SOCKET API

Linux Traffic Control (TC)

Comments

MODE_TX sock cyclic write SCM_TXTIME & VLAN tagged packet

./txrx-tsn -v -i enp2s0 -P -l 64 -q 1  \
--transmit -n 10000 -y 1000000 -T -e 400000 -o 20000 \
-d 22:bb:22:bb:22:bb > /tmp/igc-afpk1a_etf.txt &

 # Assign SCHED_FIFO policy to CPU1
 TX_PID=$!
 taskset -p 1 $TX_PID
 chrt --fifo -p 90 $TX_PID

802.1Q Egress packets Timed-Based Sched.(TBS) on ETF Queue-disciplines

tc qdisc add dev enp2s0 parent root handle 100 mqprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 hw 0


tc qdisc replace dev enp2s0 parent 100:1 etf  \
clockid CLOCK_TAI delta 400000 offload

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done
igc tc...mqprio map is reverted

see Earliest TxTime First (NET_SCHED_ETF) QDisc

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

MODE_TX sock cyclic write VLAN-tagged packet

./txrx-tsn -v -i enp2s0 -P -l 64 -q 1  \
--transmit -n 10000 -y 1000000 \
-d 22:bb:22:bb:22:bb > /tmp/igc-afpk1a_taprio-0x1.txt &

 # Assign SCHED_FIFO policy to CPU1
 TX_PID=$!
 taskset -p 1 $TX_PID
 chrt --fifo -p 90 $TX_PID

802.1Q Egress packets w/ TAPRIO tx-assisted GCL on ETF Queue-disciplines

BASE=$(expr $(date +%s) + 5)000000000
tc -d qdisc replace dev enp2s0 parent root handle 100 taprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 \
base-time $BASE \
sched-entry S 01 500000 \
sched-entry S 0e 500000 \
flags 0x1 txtime-delay 300000 clockid CLOCK_TAI

tc qdisc replace dev enp2s0 parent 100:1 etf  \
clockid CLOCK_TAI delta 400000 offload skip_sock_check

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done
igb tc...taprio map is reverted

see Time Aware Priority (NET_SCHED_TAPRIO) QDisc

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

MODE_TX sock cyclic write VLAN-tagged packet

./txrx-tsn -v -i enp2s0 -P -l 64 -q 1  \
--transmit -n 10000 -y 1000000 -T -e 400000 -o 20000 \
-d 22:bb:22:bb:22:bb > /tmp/igc-afpk1a_taprio-0x2.txt &

 # Assign SCHED_FIFO policy to CPU1
 TX_PID=$!
 taskset -p 1 $TX_PID
 chrt --fifo -p 90 $TX_PID

802.1Q Egress packets w/ TAPRIO tx-offload GCL on ETF Queue-disciplines

BASE=$(date +%s%N)
tc -d qdisc replace dev enp2s0 parent root handle 100 taprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 \
base-time $BASE \
sched-entry S 01 500000 \
sched-entry S 0e 500000 \
flags 0x2 txtime-delay 0

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done

igc base-time must set CLOCK_TAI initial epoch time prior GCL offload

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

MODE_TX sock cyclic write VLAN-tagged packet

./txrx-tsn -v -i enp2s0 -P -l 64 -q 1  \
--transmit -n 10000 -y 1000000 -T -e 400000 -o 20000 \
-d 22:bb:22:bb:22:bb > /tmp/igc-afpk1a_taprio-0x2.txt &

 # Assign SCHED_FIFO policy to CPU1
 TX_PID=$!
 taskset -p 1 $TX_PID
 chrt --fifo -p 90 $TX_PID

802.1Q Egress packets w/ TAPRIO tx-offload GCL and Frame Preemption

BASE=$(date +%s%N)
tc -d qdisc replace dev enp2s0 parent root handle 100 taprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 \
base-time $BASE \
sched-entry S 01 500000 \
sched-entry S 0e 500000 \
flags 0x2 txtime-delay 0 \
preempt 1110

ethtool --set-frame-preemption enp2s0 fp on min-frag-size 192

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done

igc base-time must set CLOCK_TAI initial epoch time prior GCL offload

preempt parameter configures which traffic classes are marked as preemptible

Enable frame preemption via ethtool

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

MODE_RX cyclic Listener rt-thread with SIOCSHWTSTAMP

./txrx-tsn -v -i enp2s0 -P -q 3  \
--receive --hw-timestamps > /tmp/igc-afpk1b.txt &

 # Assign SCHED_FIFO policy to CPU1
 RX_PID=$!
 taskset -p 1 $RX_PID
 chrt --fifo -p 90 $RX_PID

802.1Q Ingress PTP and VLAN prior 1 and 3 packets steering hw-filters

tc qdisc add dev enp2s0 parent root handle 100 mqprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 hw 0

tc qdisc replace dev enp2s0 ingress
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF queue 2
ethtool -U enp2s0 flow-type ether  \
proto 0x88f7 queue 1
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF queue 3

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
igc tc...mqprio map is reverted

see Earliest TxTime First (NET_SCHED_ETF) QDisc

igc doesn’t yet enable tc...flower

see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed

To characterize af_xdp latency overhead and jitter using synthetic L2/Ethernet packets 64B payload traversing Linux BPF/XDP socket API i.e. and Intel® Ethernet kernel XDP driver :

Important

Intel® Ethernet Controller I225 [Foxville] Software User Manual XDP/eBPF hw-offload can be set any combined 4 TX/RX queues.

It is highly recommended to turn-off VLAN stripping and set 4 hw-queues combined rx/tx queues. For example:

$ ethtool -K enp2s0 rxvlan off hw-tc-offload on
$ ethtool --set-eee enp2s0 eee off
$ ethtool -L enp2s0 combined 4
$ ethtool -G enp2s0  tx 1024 rx 1024

Application on POSIX SOCKET API w/ .xdp_bind_flags

Linux Traffic Control (TC)

Comments

MODE_TX sock cyclic write XDP_SKB mode VLAN-tag L2-packets

./txrx-tsn -v -i enp2s0 -X -l 64 -q 1  \
--transmit -n 10000 -y 1000000 --skb  \
-d 22:bb:22:bb:22:bb > /tmp/igc-afxdp1a-skb.txt &

 # Assign SCHED_FIFO policy to CPU1
 TX_PID=$!
 taskset -p 2 $TX_PID
 chrt --fifo -p 90 $TX_PID

802.1Q Egress packets XDP Queue mapping

tc qdisc add dev enp2s0 parent root handle 100 mqprio \
num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \
queues 1@0 1@1 1@2 1@3 hw 0





ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done
igc tc...mqprio map is reverted

see Earliest TxTime First (NET_SCHED_ETF) QDisc

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

MODE_RX cyclic socket read XDP_SKB mode

./txrx-tsn -v -i enp2s0 -X -q 3  \
--receive  --skb > /tmp/igc-afxdp1b-skb.txt &

 # Assign SCHED_FIFO policy to CPU1
 RX_PID=$!
 taskset -p 1 $RX_PID
 chrt --fifo -p 90 $RX_PID

802.1Q Ingress PTP and VLAN prior 1 and 3 packets steering hw-filters

tc qdisc add dev enp2s0 parent root handle 100 mqprio \
num_tc 4 map 3 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \
queues 1@0 1@1 1@2 1@3 hw 0

tc qdisc replace dev enp2s0 ingress
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF queue 1
ethtool -U enp2s0 flow-type ether  \
proto 0x88f7 queue 2
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF action 3

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
igc tc...mqprio map is reverted

see Earliest TxTime First (NET_SCHED_ETF) QDisc

igc doesn’t yet enable tc...flower

see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed

MODE_TX sock cyclic write XDP_COPY & VLAN-tag packets

./txrx-tsn -v -i enp2s0 -X -l 64 -q 1  \
--transmit --native-copy -n 10000 -y 1000000 \
-d 22:bb:22:bb:22:bb > /tmp/igc-afxdp1a-copy.txt &

 # Assign SCHED_FIFO policy to CPU1
 TX_PID=$!
 taskset -p 1 $TX_PID
 chrt --fifo -p 90 $TX_PID

802.1Q Egress packets XDP Queue mapping

tc qdisc add dev enp2s0 parent root handle 100 mqprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 hw 0





ip link add link enp2s0 name enp2s0.vlan type vlan id 3
for j in `seq 0 7`; \
do ip link set enp2s0.vlan type vlan egress-qos-map $j:$j ; done
igc tc...mqprio map is reverted

see Earliest TxTime First (NET_SCHED_ETF) QDisc

Set VLAN ID to 3, all traffic fixed to one VLAN ID, but vary the VLAN Priority that socket priority N qos-map

MODE_RX cyclic socket read XDP_COPY mode

./txrx-tsn -v -i enp2s0 -X -q 3  \
--receive --native-copy > /tmp/igc-afxdp1b-copy.txt &

 # Assign SCHED_FIFO policy to CPU1
 RX_PID=$!
 taskset -p 1 $RX_PID
 chrt --fifo -p 90 $RX_PID

802.1Q Ingress PTP and VLAN prior 1 and 3 packets steering hw-filters

tc qdisc add dev enp2s0 parent root handle 100 mqprio \
num_tc 4 map 0 1 2 3 0 0 0 0 0 0 0 0 0 0 0 0 \
queues 1@0 1@1 1@2 1@3 hw 0

tc qdisc replace dev enp2s0 ingress
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x2000 \
m 0x1FFF queue 1
ethtool -U enp2s0 flow-type ether  \
proto 0x88f7 queue 2
ethtool -U enp2s0 flow-type ether  \
proto 0x8100 vlan 0x6000 \
m 0x1FFF queue 3

ip link add link enp2s0 name enp2s0.vlan type vlan id 3
igc tc...mqprio map is reverted

see Earliest TxTime First (NET_SCHED_ETF) QDisc

igc doesn’t yet enable tc...flower

see Packet Classifier (CONFIG_NET_CLS_FLOWER)

Set VLAN ID to 3, all traffic fixed

Interpreting TSN TX/RX latency Results

To visualize the benchmark from result file ./txrx-tsn -P ... > /tmp/afpkt-traffic.txt and/or ./txrx-tsn -X ... > /tmp/afxdp-traffic.txt all recorded packets timestamps into a visual plot as in txrx-tsn-plot.gnu multi XY gnuplot examples:

Click to toggle gnuplot script

reset

  set terminal pngcairo size 1920,1080
  set output 'txrx-tsn-plot.png'

  set size 1,1
  set datafile missing'?' # Skip invalid/missing data
  set locale "en_US.utf8" # Set to en_US locale for thousands seperator

  set tmargin 5
  set bmargin 5
  set lmargin 15
  set rmargin 5

  set yrange [0:10000000] #Legacy socket plot YMAX

  set grid
  set key center tmargin
  set border

  set style fill solid 0.25
  set boxwidth 0.5
  # set logscale y 10

  set title "Transmission latency from TX User-space to RX User-space"
  set xlabel "Packet count"
  set ylabel "Latency in nanoseconds"

  set multiplot layout 1,2

  stats FILENAME u 1 name "a" nooutput
  stats FILENAME2 u 1 name "b" nooutput

  min_us = a_min/1000
  max_us = a_max/1000
  avg_us = a_mean/1000

  set label front sprintf("Minimum: %d us", min_us) at graph 0.01, graph 0.95
  set label front sprintf("Maximum: %d us", max_us) at graph 0.01, graph 0.90
  set label front sprintf("Average: %d us", avg_us) at graph 0.01, graph 0.85

  plot FILENAME \
    using ($1) title "Legacy socket"  lc rgb "red" w points

  unset label

  set yrange [0:2000000]  #XDP socket plot YMAX

  min_us = b_min/1000
  max_us = b_max/1000
  avg_us = b_mean/1000

  set label front sprintf("Minimum: %d us", min_us) at graph 0.01, graph 0.95
  set label front sprintf("Maximum: %d us", max_us) at graph 0.01, graph 0.90
  set label front sprintf("Average: %d us", avg_us) at graph 0.01, graph 0.85

  plot FILENAME2 \
    using ($1) title "XDP socket"  lc rgb "red" w points

  unset multiplot
  unset output

  #Replot to GUI
  set terminal x11 size 1720,980
  set output
  set multiplot layout 1,2
  plot FILENAME \
    using ($1) title "Legacy socket"  lc rgb "red" w points

  plot FILENAME2 \
    using ($1) title "XDP socket"  lc rgb "red" w points

pause 10 #in case some one forgets to add -p

$ gnuplot -e "FILENAME='afpkt-traffic.txt';FILENAME2='afxdp-traffic.txt';" $DIR/txrx-tsn-plot.gnu -p 2> /dev/null &

To visualize Timeview-UI representing **af_packet** user and kernel space trace record

To visualize Text-UI histograms representing **af_packet** kernel-space latency outliers