The IEEE TSN Task group specifies multiple enhancements to the IEEE802.1Q standard in order to improve the timing performances of Ethernet. Standard-conformant solutions are very attractive to the industry and offer an investment security. Multiple Time Sensitive Networking prototypes and demonstrators have been presented in different fairs the last few years.

For integration in the automotive and industrial sectors, the TSN features and the hardware-prototypes need to be evaluated. This article presents a test-plan to evaluate the following TSN-Sub-Standards: IEEE802.AS, IEEE802.1Q, IEEE802.1Qbu, IEEE802.3br and IEEE802.1Qbv.

Deterministic applications

Deterministic industrial applications can be classified into five main areas:

• Condition Monitoring
• Process Automation
• Machine Tool
• Packaging Machines
• Printing Machines

The performance of these applications is given by a set of indicators such as cycle time, number of communicating devices and the synchronization accuracy of their clocks, payload per device, etc. The typical timing constraints of these applications are illustrated in the table below.

Evaluated features & KPIs

The table illustrates deterministic IEEE802 standard-enhancements and their appropriate key performance indicators (KPIs).

Accuracy: The mean of the time or frequency error between the clock under test and a perfect reference clock over an ensemble of measurements.

Worst-Case (End-To-End) delay: The time duration for an Ethernet frame from its transmission start time point at the sender egress port to the reception start time point at the receiver ingress port.

Forwarding Delay: The time duration for a frame in a forwarding hop: From Reception start time point at the ingress port to the transmission start time point on the egress port.

Interference delay: The delay caused by the frame that was selected for transmission an arbitrarily small time before frame X became eligible for transmission selection, plus the delay caused by queued-up frames from all stream frames with higher priority than frame X’s class [IEEE802.1Q – 2014 L3.1 Interference delay]

Delay Variation (Jitter): The variation of the End-To-End delay in a sample duration (e.g. 1000 cycles).

Typical Requirements of Industrial Applications

Network design

The diagram above illustrates the network design to evaluate TSN standard enhancements using four TSN-switches. The TSN-switches are the Devices Under Test (DUTs) and are physically connected in a line topology. Four industrial PCs that generate multiple data traffics are distributed all-over the network.

The Time-Critical (TC) and the Non-Time-Critical (NTC) Sender 1 are connected to TSN-switch 1. NTC Sender 2 and 3 are respectively connected to TSN-switch 2 and 3. A Time-Critical (TC) Receiver is connected to TSN-switch 4. All four TSN-switches and four traffic senders are configured through the Development Host. A Standard-Ethernet Switch is used to distribute the configuration files (TFTP).

A network TAP with four channels is used to measure the End-To-End delay, single switch forwarding delays, interference delays and the jitter of the time-critical traffic. The captured frames are time-stamped and uploaded to the development host.

Network Configuration

The time critical (TC) sender is an industrial PC that transmits each one millisecond a 1400 Byte scheduled frame with the VLAN priority PCP 7 (IEEE802.1Q) over all TSN-switches to the Time-Critical receiver. Like in real-networks time-critical traffics might interfere with concurring background traffics. Therefor three Non-Time-Critical traffics are generated. Each of the traffics is causing a bandwidth utilization of 96% (~100%).

Depending on the activated Transmission Selection Algorithm, the TC traffic might face multiple interferences before reaching the receiver. Based on the activated transmission selection algorithm the interference can be reduced or even avoided. A set of time-critical and non-time-critical traffics have been assigned to each of the transmission selection algorithms [Nsaibi, Leurs: Worst-Case Timing Analysis of Integrating TSN in the Field-Level – KommA Conference 2017].

End-to-end delay distribution in [μs] using different transmission selection algorithms.

Experiment Procedure

This section explains a TSN evaluation procedure. Based on certain features such as link speed, network load, number of forwarding hops, traffic sizes, etc., multiple scenarios can be constructed.

• Step 1: Connect the Pulse Per Second (PPS) output pins of the TSN-capable devices to an oscilloscope in order to measure the synchronization accuracy. Power on all devices.
• Step 2: Configure the TSN switches by activating one of the Transmission Selection Algorithms. Time-Critical Traffics need to be identified on the TSN-network (e.g. through the MAC addresses and the PCP field within the VLAN-tag)
• Step 3: Start recording all data-traffics on Wireshark using the network TAP(s).
• Step 4: The gPTP synchronization status (Master, Slave, Passive, etc.) of each port on the TSN-switches can be read on wireshark. Measure the synchronization accuracy using an oscilloscope. Based on the gPTP daemon implementation, the synchronization might take few seconds to few minutes until the accuracy goes below 1 µs. Only then proceed to step 5.
• Step 5: Start transmission of time-critical traffic (without any background traffic).
• Step 6: Start the transmission of non-time critical traffics.
• Step 7: Stop the transmission of all data-traffics and the recording.
• Step 8: Save the Wireshark pcap file.
• Step 9: Activate another Transmission Selection Algorithm and repeat all steps from 2 to 8. A particular TSA can also be evaluated for different network loads (e.g. 0%, 20%, 40%, etc.).

Minimum, average & maximum end-to-end delay over 4x TSN-switches for IEEE802.1Qbu & IEEE802.1Qbv.

Results Analysis

Time Synchronization: An industrial control system might consist of hundreds of sensors, actuators, and controllers. The challenge for a deterministic network is to guarantee an upper bound delay of the time-critical traffic exchanged between the devices. Failure for an Ethernet frame to reach its destination in time can result in uncoordinated mechanical movements, or wrong correlation of sensor data. The best approach to avoid such a problem is to schedule the time-critical traffic throughout its path from the sender to the receiver. This requires a synchronized network. For highly deterministic applications, such as motion-control, synchronization accuracy in sub-microsecond is required.

The IEEE 802.1AS time-synchronization standard enables clock synchronization across TSN-aware devices and is therefore evaluated for use in industry.

The TSN-switches, Time-Critical Sender as well as the Time-Critical Receiver, support the IEEE802.1AS time synchronization standard. The PPS output pins of the Time-Critical Sender and Receiver are connected to the oscilloscope. The graphic above shows the two clock-signals of the end-devices. A synchronization accuracy below 100 nanosecond over four hops has been measured. The traces drift between ±50 nanoseconds.

For use in the deterministic applications, a measurement of the clock drift using IEEE802.1AS over a higher number of devices (e.g. 100 devices in daisy chain) is required.

A measurement of the clock accuracy over 11 TSN-switches from 4 different vendors connected in a daisy chain with 1Gbps link speed gave a synchronization accuracy of ±250 nanoseconds. A time-error, below 100 nanoseconds, between the clocks of the two end-devices (Time-Critical Sender and Receiver) can be illustrated as an oscillation graph for a sample duration of ~120 seconds.

Timing Behavior: end-to-end delay

The end-to-end delay of the time-critical traffic from the sender (TC-sender) to the receiver has been captured for a sample duration of 30 seconds. For the purpose of clarity we show only 1.5 seconds ~ 1500 Cycles. The experiment has been repeated three times. Each time a transmission selection algorithm is activated over the path, followed by time-critical traffic.

• Blue: End-To-End delay using the non-preemptive Strict-Priority Algorithm (IEEE802.1Q) and is in the range of [680 µs to 820 µs]
• Red: End-To-End delay using the preemptive Strict-Priority Algorithm (IEEE802.1Qbu) and is in the range of [508 µs to 519,12 µs]
• Green: End-to-end delay using the non-preemptive Time-Aware Shaper, and is in the range of [503,84 µs to 504,4 µs]

For a better comparison of the end-to-end delays using IEEE802.1Qbu and IEEE802.1Qbv, the red and green graphs are zoomed and the vertical axis is reduced.

Activating the frame preemption strongly reduces the worst-case End-To-End delay and the jitter of the time-critical traffic. The range is reduced from 140 µs [680 µs to 820 µs] using IEEE802.1Q to 10 µs [508 µs to 518 µs] using IEEE802.1Qbu. The results are optimistic since the network has only one time-critical traffic. The interference with other express data-traffics (e.g. alarms, other control-data traffics), that cannot be preempted, would highly increase the delay and thus jitter. This has not been covered in the experiment.

A further reduction of the End-To-End delay and the jitter for the same link speed is only possible if the time-critical frames are scheduled throughout the whole path from the sender to the receiver. This requires the activation of the Time-Aware Shaper IEEE802.1Qbv.

For a lower number of forwarding hops, the delay optimization is minimal compared to the reduction of the jitter range (below 1 microsecond). It is proven that the timing requirements of highly deterministic industrial applications, such as Motion-Control, can only be met with IEEE802.1Qbv Time-Aware Shaper. However the TSN scheduling approach might “highly” increase the configuration complexity for big networks.

IEEE802.1Qbu – Upper-Bound Worst Case delay: The time-critical traffic can be protected from interfering with background traffic(s) by reserving transmission time-slots on the egress ports of the forwarding hops as defined in IEEE802.1Qbv. For IEEE802.1Q and IEEE802.1Qbu the upper-bound worst case interference delay depends on the size of the interfering frame need to be computed.

The upper-bound worst-case interference delay for non-preemptive strict-priority algorithms is always given by an interfering frame with the maximum Ethernet frame size 1530 Byte. If frame-preemption is activated the interference delay can be strongly reduced.

The upper-bound worst case interference delay for IEEE802.1Qbu is either given by an interfering express frame of the size 1530 Byte or by a preemptable frame that cannot be preempted. That are preemptable frames with the size 64 – 131 Byte, for which the preemption does not result into two fragments that matches the minimum Ethernet frame size 64 Byte.

In order to measure the upper-bound worst-case interference delay for the preemptive strict-priority algorithm, the experiment is repeated with frame sizes between 64 – 131 Byte for the preemptable non-time-critical traffics 1, 2 and 3. The time-critical traffic is not modified.

It is shown that the size of the interfering preemptable frames has an impact on the End-To-End delay. The range of the delay variation over four hops increased from ~ 10 µs [508.84 – 519.72] to ~27 µs [507.24 – 534.92]. The delay variation (jitter) might increase with the number of forwarding hops.

The minimum, maximum and average End-To-End delay with IEEE802.1Qbu and IEEE802.1Qbv using 4 TSN-switches included different link speeds (100Mbps and 1Gbps).

The results proves that Gigabit-Ethernet can strongly reduce the end-to-end delay and the jitter. For a fair comparison between IEEE802.1Qbu and IEEE802.1Qbv, the background traffic needs to include other time-critical frames, such as alarms, safety-critical frames or other control-data traffics (Sercos III, EtherNet/IP or Profinet), that cannot be preempted. This would result in a high “undeterministic” timing behavior for the IEEE802.1Qbu.

Throughput/bandwidth usage

The channel throughput is an important key performance indicator for IT networks. Industry 4.0 and Internet of Things (IoT) aim to merge the IT and OT networks, which makes the channel throughput more important for the OT networks. The throughput is the rate of successful message delivery over a communication channel in bits per second and is limited by the channel capacity given by the data-rate or link speed.

A bad traffic scheduling in case of time-aware shaper (IEEE802.1Qbv) can lead to a poor bandwidth utilization and thus to an inefficient throughput. The worst-case scenario for the channel throughput or the bandwidth utilization has two bounds: the minimum bound is losing an amount of the reserved time-slot if this is much bigger than the transmission duration of the scheduled frame(s). The maximum bound is reached if a non-scheduled frame with the maximum Ethernet frame size (1530Byte) does not fit within the guard-band because it just missed the start of the guard-band.

Unlike the non-preemptive time-aware shaper (IEEE802.1Qbv), a non-preemptive strict-priority-shaper (IEEE802.1Q) offers a high bandwidth utilization whenever there are frames ready for transmission. Thus there is no bandwidth loss. Activating the preemption (IEEE802.1Qbu) causes an additional overhead: 24 Bytes by each preemption process. However for a single time-critical frame not more than one interference with preemptable traffic can occur. The lost bandwidth per forwarding hop is about ~2μs for 100Mbps and 192ns for 1Gbps.

Activating the frame-preemption increases the channel throughput of IEEE802.1Qbv by 6% for 100Mbps. Combining IEEE802.1Qbu and IEEE802.1Qbv leads to a better utilization of the guard-band to send non-scheduled traffics even if their transmission would not end before the start of the reserved time-slot.

Measurement precision

Transmission and reception time points of the time-critical frames are time-stamped with network taps. Each network TAP introduces an additional forwarding delay of 500-600 nanoseconds. The time-critical traffic is forwarded over 5 TAPs, which results in a total additional delay of 2.5 to 3 microseconds for each measured value. This has no effect on the synchronization accuracy. Only End-To-End delay as well as the jitter are affected.

Conclusion

The manual TSN-evaluation, introduced in this article, can deliver similar results as the “expensive” TSN-Evaluation-Boards but is too complex, can be error-prone and requires a deep TSN- and Ethernet knowledge. On the other side much more scenarios can be covered, since the data-traffics can be easily modified (e.g. more traffics, size, priority).

These test scenarios can be used to do an in-depth evaluation of the TSN-standards and prototypes. The results proved that only IEEE802.1Qbv can guarantee the end-to-end ultra-low jitter as required by the highly deterministic industrial automation applications.

But scheduling the whole network might introduce a lot of complex computations. The challenge is to choose the appropriate configuration approach (fully distributed, centralized or fully centralized as shown in IEEE802.1Qcc) to the Industrial-Ethernet networks using TSN.

Combining IEEE802.1Qbu with higher link speed (e.g. 1 Gbps) can strongly further reduce the delay but cannot guarantee an ultra-low jitter (<< 1 µs) in the coexistence of other time-critical (express) data-traffics.

Seifeddine Nsaibi, PhD Student at Bosch Rexroth AG and Ajitesh Mishra, Master Student at Bosch Rexroth AG