Attention

You are viewing an older version of the documentation. The latest version is v3.3.

Commercial Industrial Ethernet Switches (PCIE-0400-TSN)

The Kontron PCIE-0400-TSN PCIe card is a Intel® CycloneV FPGA based Low-Profile Time Sensitive Network PCIe Add-In Card with four 10/100/1000Base-T Ethernet Ports.

The PCIe-0400-TSN Hardware/Software as well as part of KBox TSN Starter Kit, allows to evaluate TSN functionality using pre-installed Software tools and utilities intended to familiarize the user with IEEE 802.1AS-2011 time synchronization, IEEE 802.1Q-2018Q Enhancements for Scheduled Traffic (EST) (formerly known as 802.1Qbv) scheduling policies setup and analysis using various industrial Ethernet traffic profiles.

All PCIE-0400-TSN collateral can be downloaded from Kontron website (https://emdcustomersection.kontron.com/?page_id=11481 - you may need to register and login).

The PCIE-0400-TSN switch onboard TTTECH® DE-IP and Linux driver exposing the internal 5-port switch to the linux user-space. Therefore, the TSN switch can be managed by using standard linux tools like ifconfig or``bridge vlan add``.

Interface

PCIE-0400-TSN Description

SE01

Service port, mapped internally to the TSE network interface

IE01

Switchdev device for SE01

CE01

Switchdev device for external Port 1

CE02

Switchdev device for external Port 2

CE03

Switchdev device for external Port 3

CE04

Switchdev device for external Port 4

The PCIE-0400-TSN driver also supports reception of timestamps for PTP frames and implements the operations to adjust the PTP hardware clock (PHC) inside the switch.

For further details about PCIE-0400-TSN please refer to userguide_pcie-0400-tsn_v0.14.pdf.

PCI-400-TSN Sanity-Check Testing

Connect two TSN Endpoints (Device A & Device B) and the Performance Monitor (ie: Ubuntu Desktop) to a single Kontron PCI-400-TSN switch as depicted in the following figure:

Important

Only connect Device A and Device B to the Kontron switch via RJ45 Ethernet jacks corresponding to TSN capable NICs. Non-TSN NICs cannot be used with the examples presented in this guide.

../../_images/0511.png

Sanity-Check #1: TSN Switch PCI-400-TSN

The following section is applicable to:

../../_images/switch.png

Step 1: Identify TTTECH® DE-IP kernel modules.

  1. Perform the following command to identify TTTECH® DE-IP:

    lspci -d 1059:a100 -vvv
    Copy to clipboard

    The expected output appears as follows:

    03:00.0 Ethernet controller [0200]: Kontron Device [1059:a100] (rev 10)
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at d0000000 (32-bit, non-prefetchable) [size=128M]
        Region 1: Memory at dc020000 (32-bit, non-prefetchable) [size=4K]
        Region 2: Memory at dc000000 (32-bit, non-prefetchable) [size=128K]
        Region 5: Memory at d8000000 (32-bit, non-prefetchable) [size=64M]
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [78] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #1, Speed 2.5GT/s, Width x4, ASPM L0s, Exit Latency L0s <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Kernel modules: tsnic_core
    Copy to clipboard

    The Kontron Device [1059:a100] mounted as 03:00.0 pci device in this example.

    Important

    There may be more than one PCIE-0400-TSN cards. Be sure to identify the interface which is wired to the switch device.

Step 2: run “ifconfig to identify mounted on example output

CE01: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
      ether fa:e7:74:ab:35:cf txqueuelen 1 (Ethernet)
      RX packets 31130 bytes 1889482 (1.8 MiB)
      RX errors 0 dropped 31130 overruns 0 frame 0
      TX packets 6123 bytes 426906 (416.9 KiB)
      TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

CE02: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
      ether 52:d2:f1:0e:a7:4a txqueuelen 1 (Ethernet)
      RX packets 0 bytes 0 (0.0 B)
      RX errors 0 dropped 0 overruns 0 frame 0
      TX packets 0 bytes 0 (0.0 B)
      TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

CE03: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
      ether 8e:78:a4:14:4e:2d txqueuelen 1 (Ethernet)
      RX packets 0 bytes 0 (0.0 B)
      RX errors 0 dropped 0 overruns 0 frame 0
      TX packets 0 bytes 0 (0.0 B)
      TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

CE04: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
      ether 22:cb:6e:a0:17:40 txqueuelen 1 (Ethernet)
      RX packets 0 bytes 0 (0.0 B)
      RX errors 0 dropped 0 overruns 0 frame 0
      TX packets 0 bytes 0 (0.0 B)
      TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

IE01: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
      ether 3a:1f:d3:39:05:30 txqueuelen 1 (Ethernet)
      RX packets 0 bytes 0 (0.0 B)
      RX errors 0 dropped 0 overruns 0 frame 0
      TX packets 0 bytes 0 (0.0 B)
      TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

SE01: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
      ether 12:41:5a:ab:25:4c txqueuelen 1 (Ethernet)
      RX packets 0 bytes 0 (0.0 B)
      RX errors 0 dropped 0 overruns 0 frame 0
      TX packets 0 bytes 0 (0.0 B)
      TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Copy to clipboard

Sanity-Check #2: Set PCI-400-TSN switch as IEEE 802.1AS-2011 PTPv2 Grand-Master

The following section is applicable to:

../../_images/switch.png

Step 1: Disable existing PTP service.

  1. Perform the following commands:

    systemctl | grep ptp
    Copy to clipboard

    The expected output appears as follows:

    ptp.service loaded active running   Precision Time Protocol (PTP) service
    Copy to clipboard

  2. Disable each of the services identified in the previous step. For each output line with format ${service_name}.service, perform the following command:

    systemctl disable ${service_name}
    Copy to clipboard

Step 2: Customize and Start PTP service.

  1. Create or edit /etc/systemd/system/ptp4l-tsn.service and modify to be as follows:

    [Unit]
Description=Precision Time Protocol (PTP) service
After=copytime.service

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/ptp4l_tsn
ExecStart=/bin/bash -c "exec /usr/sbin/ptp4l ${OPTIONS}"

[Install]
WantedBy=multi-user.target
    Copy to clipboard

  2. Create or edit /etc/sysconfig/ptp4l_tsn and modify to be as follows:

    OPTIONS="-f /etc/ptp4l-tsn.cfg"
    Copy to clipboard

  3. Create or edit /etc/ptp4l-tsn.cfg and modify to be as follows:

    #
#       ptp4l 802.1AS config (minimal)
#
#

[global]
gmCapable               1
priority1               248
priority2               248
logAnnounceInterval     1
logSyncInterval         -3
syncReceiptTimeout      3
neighborPropDelayThresh 800
min_neighbor_prop_delay -20000000
assume_two_step         1
path_trace_enabled      1
follow_up_info          1

# Generic MAC to broadcast L2 PTP to many NICs (ie. diff MACs)
ptp_dst_mac             01:80:C2:00:00:0E
network_transport       L2
delay_mechanism         P2P

# Additional Config Parameters
tx_timestamp_timeout    100
summary_interval        0

[CE01]
transportSpecific 0x1

[CE02]
transportSpecific 0x1

[CE03]
transportSpecific 0x1

[CE04]
transportSpecific 0x1
    Copy to clipboard

  4. Perform the following commands:

    systemctl enable ptp4l-tsn
systemctl start ptp4l-tsn
systemctl status ptp4l-tsn
    Copy to clipboard

    The expected output should be similar to the following:

    ● ptp4l-tsn.service - Precision Time Protocol (PTP) service
  Loaded: loaded (/etc/systemd/system/ptp4l-tsn.service; enabled; vendor preset: disabled)
  Active: active (running) since Thu 2020-01-16 00:46:36 UTC; 3 weeks 6 days ago
Main PID: 1779 (ptp4l)
   Tasks: 1 (limit: 4915)
  CGroup: /system.slice/ptp4l-tsn.service
          └─1779 /usr/sbin/ptp4l -f /etc/ptp4l-tsn.cfg
    Copy to clipboard

    The text ‘Active: active (running)’ is expected.

  5. Check port that MASTER and SLAVE states the following commands:

    # ./ck-ptp.sh -w
CE01: portState =  MASTER, peerMeanPathDelay =  0
CE02: portState =  MASTER, peerMeanPathDelay =  228
CE03: portState =  MASTER, peerMeanPathDelay =  2044
CE04: portState =  MASTER, peerMeanPathDelay =  0
grandmasterId   = 1ca0d3.fffe.200618
master_offset   = 0
ingress_time    = 0
    Copy to clipboard

Sanity-Check #3: Configure Traffic Class Table and Gate Control List on PCI-400-TSN switch

The following section is applicable to:

../../_images/switch.png

Step 1: Create or edit ~/enable_tsn.sh and modify to be as follows:

#!/usr/bin/env bash

# setup Kontron PCIe
function setup_vlan () {
  bridge vlan add dev CE01 vid 2
  bridge vlan add dev CE01 vid 3

  bridge vlan add dev CE02 vid 2
  bridge vlan add dev CE02 vid 3

  bridge vlan add dev CE03 vid 2
  bridge vlan add dev CE03 vid 3

  bridge vlan add dev CE04 vid 2
  bridge vlan add dev CE04 vid 3

  bridge vlan add dev IE01 vid 2
  bridge vlan add dev IE01 vid 3
}

function setup_mtc() {
  tsntool edge mtc set 1 CE01
  tsntool edge mtc set 1 CE02
  tsntool edge mtc set 1 CE03
  tsntool edge mtc set 1 CE04
  tsntool edge mtc set 1 IE01
}

function generate_qbv_conf () {
  FILE=$1

cat <<'EOF' > $FILE
sgs 100000 0x01
sgs 100000 0x04
sgs 300000 0x02
sgs 100000 0x01
sgs 100000 0x04
sgs 300000 0x02
EOF
}

function configure_port () {
  PORT=$1

  CFG="/tmp/${PORT}_qbv.cfg"
  DUMP="/tmp/${PORT}_qbv.cfg.dump"

  generate_qbv_conf ${CFG}

  echo 1 > /sys/class/net/${PORT}/ieee8021ST/GateEnabled

  # Get the current time for port
  cur_time=$(cat /sys/class/net/${PORT}/ieee8021ST/CurrentTime | awk -F. '$0=$1')
  # Convert current time to the right format for tsntool
  set_time=$(echo $cur_time+0.0000000 | bc)
  # Write the configuration to the interface descriptor
  tsntool st wrcl ${CFG} ${PORT}
  # Configure the interface using:
  # * Current time
  # * Cycle time (in seconds -> 1 ms)
  # * Cycle offset
  tsntool st configure $set_time 1/1000 0 ${PORT}
  # Check configuration
  tsntool st rdacl ${DUMP} ${PORT}
  diff ${CFG} ${DUMP}
  rm ${DUMP}
}

function print_configuration() {
  echo "VLAN configuration"
  bridge vlan show

  echo "GateEnabled CE01"
  cat /sys/class/net/CE01/ieee8021ST/GateEnabled
  echo "GateEnabled CE02"
  cat /sys/class/net/CE02/ieee8021ST/GateEnabled
  echo "GateEnabled CE03"
  cat /sys/class/net/CE03/ieee8021ST/GateEnabled
  echo "GateEnabled CE04"
  cat /sys/class/net/CE04/ieee8021ST/GateEnabled

  echo "Current tsntool configuration:"
  echo "CE01:"
  tsntool st rdacl /dev/stdout CE01

  echo "CE02:"
  tsntool st rdacl /dev/stdout CE02

  echo "CE03:"
  tsntool st rdacl /dev/stdout CE03

  echo "CE04:"
  tsntool st rdacl /dev/stdout CE04

  tsntool edge show CE01
  tsntool edge show CE02
  tsntool edge show CE03
  tsntool edge show CE04
}

#
# Main script
#

setup_vlan

configure_port "CE01"
configure_port "CE02"
configure_port "CE03"
configure_port "CE04"

print_configuration
Copy to clipboard

Step 2: Perform the following commands:

chmod +x ~/enable_tsn.sh
~/enable_tsn.sh
Copy to clipboard

Step 3: Perform the following commands:

bridge vlan show
Copy to clipboard

The expected output is as follows:

port    vlan ids
IE01     1 PVID Egress Untagged

IE01     1 PVID Egress Untagged

CE01     1 PVID Egress Untagged
         3

CE01     1 PVID Egress Untagged
         3

CE02     1 PVID Egress Untagged
         3

CE02     1 PVID Egress Untagged
         3

CE03     1 PVID Egress Untagged
         3

CE03     1 PVID Egress Untagged
         3

CE04     1 PVID Egress Untagged
         3

CE04     1 PVID Egress Untagged
         3

deipce0  1 PVID Egress Untagged
Copy to clipboard

Step 4: Perform the following command:

tsntool st rdacl /dev/stdout ${interface}
Copy to clipboard

The ${interface} value can be CE01, CE02, CE03 and CE04. For each value, the expected output is as follows:

sgs  100000  0x01
sgs  100000  0x04
sgs  300000  0x02
sgs  100000  0x01
sgs  100000  0x04
sgs  300000  0x02
Copy to clipboard

Step 5: Perform the following command:

tsntool edge show ${interface}
Copy to clipboard

The ${interface} value can be CE01, CE02, CE03 and CE04. For each value, the expected output is as follows:

Port '${interface}':
   Management Traffic Class: : 1
   Cut-through                 OFF
   Port-Mirroring              OFF
Copy to clipboard