DETERMINISTIC ETHERNET IS WIDELY USED in industrial control systems. Many Industrial protocols solve the problem of determinism over Ethernet using proprietary layer 2 solutions. The new IEEE 802.1 TSN standards are aimed at the same class of problems encountered in industrial control and promise to supplant proprietary solutions in favor of a standards-based approach. EtherNet/IP has always relied upon standard, commercially available Ethernet technologies to deliver deterministic performance and is therefore well-positioned to leverage these emerging standards.

Deterministic Ethernet overview

Deterministic Ethernet refers to an extended set of capabilities that allow standard Ethernet to be used in real-time, mission critical applications such as Factory Automation, Process Control, and Automobile Networks.

Ethernet has traditionally been a “best-effort” network. To allow Ethernet to be deployed in mission critical applications, it is necessary to add specific features including time synchronization, scheduled traffic, ingress policing, seamless redundancy and others. These features allow network designers to ensure that certain classes of traffic can be delivered on time, every time throughout the entire network topology.

Deterministic Ethernet started in the factory automation market, where large OEMs defined their own methods for adding these capabilities to Ethernet and this has spawned a number of “open standards” that are used in Industrial Ethernet today. ODVA has deployed deterministic Ethernet networks, with time-synchronization and quality of service for critical traffic, for years.

The IEEE standards organization is currently working to add Time Sensitive Networking (TSN) features to standard 802.1 and 802.3 Ethernet to provide deterministic performance. When this work is completed, it will become practical to deploy standard TCP/IP Ethernet (with the TSN extensions) in real-time, mission critical applications. However, it is unlikely that the existing industrial protocols will be replaced by TSN. It is more likely that they will be adapted to make use of the inherent deterministic capabilities provided by TSN.

The goal behind the new TSN standards is to achieve a truly converged network where all classes of traffic can seamlessly coexist. This would allow mission critical real-time traffic to coexist on the same network as traditional QoS prioritized traffic and best-effort traffic.

A major driving force behind the development of these new TSN standards is the emerging Automotive Ethernet market. However, Industrial Automation market share many common requirements with automotive control applications and can therefore leverage the standard technologies and economies of scale inherent in this very large market.

It is important point to note that the new IEEE standards are based on some of the same basic techniques that have been used in the Industrial Ethernet protocols for many years. However, many organizations have relied upon proprietary layer 2 techniques to achieve determinism. In contrast, ODVA and EtherNet/IP have relied exclusively on widely available, ubiquitous standards from IEEE and other organizations. For this reason, ODVA is uniquely poised to successfully leverage the emerging TSN standards.

ODVA tasked its Distributed Motion and Time Synchronization special interest group (SIG) with evaluating how to merge these TSN standards into the existing CIP standards. It is a common misconception that TSN is a single standard. In reality, TSN is a set of new standards and enhancements to exiting standards. In other words, TSN is a basket of new Ethernet features including time synchronization and scheduled traffic.

IEEE802.1Qbv Queuing Structure.

Time synchronization

In the context of TSN, time synchronization refers to proposed modifications to the existing IEEE P802.1AS standard that was defined for Audio Video Bridging. To understand the proposed changes, first we should describe IEEE 802.1AS.

IEEE802.1AS is the Audio-Video Bridging (AVB) profile of the IEEE1588 Precision Time Protocol. IEEE1588, and therefore IEEE802.1AS uses a master-slave protocol to synchronize real-time clocks in the nodes of a distributed system that communicates using a network. In simple terms, PTP ensures that every node on the network knows what time it is. It does not specify what a given node does with its knowledge of time.

The AVB profile (IEEE802.1AS), has features tailored to the “plug and play requirements” of AVB components. It does not use “transparent clocks” to compensate for bridge latency because transparent clocks violate IEEE802 layering conventions. Instead, each node accepts time information from the best available master clock and produces a slave clock, in a manner similar to a boundary clock, to compensate for latency. By nature, this approach is peer-based.

This approach is not compatible with end-to-end transparent clock defined as default IEEE-1588 profile and used in CIP Sync. For this reason, time-bridging mechanisms need to be developed to allow technologies and installations to migrate into the newer TSN domains. By developing time gateways, brownfield installations can be included in the larger TSN eco-system and benefit from many of the new features and capabilities that TSN has to offer.

The End-to-End TC function does not require all nodes to be time aware which is important in brownfield installations. While time accuracy will be compromised if there are non-time-aware nodes, a common understanding of time is maintained among the time-aware nodes. In contrast, the Peer-to-Peer mechanism utilized by IEEE802.1AS requires that every node will be time-aware. Obviously, this requirement is not practical for brownfield uses cases.

IEEE802.1AS-REV introduces new features needed for time-sensitive applications. These features include the ability to support a multiple time domains to allow rapid switchover, should a Grandmaster fail, and a more precise measurement of time.

While time synchronization is a foundational component of most TSN features (AVB, ingress policing, scheduled traffic), none of these features require that specific PTP profile be utilized. For instance, ingress policing requires an understanding of time that can come from the default IEEE1588v2 profile, the IEEE802.1AS profile, or some other PTP profile.

Scheduled traffic

Applications implementing control loops over Ethernet (factory automation, robotics, automotive control systems) require delivery of control data at precise times with minimal latency and jitter. Existing priority mechanisms provide for prioritization of traffic or guarantees of bandwidth (AVB), but the time of delivery is unpredictable. IEEE802.1Qbv supports this requirement by divided Ethernet traffic into different classes thus ensuring that, at specific times, only one traffic class (or set of traffic classes) has access to the network.

This division, in effect, creates a protected “channel” that is used by that traffic class alone. For clarity, this traffic class is scheduled traffic in all cases, and scheduled traffic is given the highest priority on the wire.

IEEE802.1Qbv accomplishes this division by introducing time-aware “transmission gates”. These gates are used to enable separate transmission queues. The Qbv shaper provides a time-based circular schedule which opens and closes the transmission gate at specific times. Note that once a gate is open, existing IEEE802.1Q transmission selection algorithms function normally. A given queue may operate on strict priority (best effort) basis while another may use the credit-based shaper defined for AVB applications. However, it is worth noting that IEEE provides no guidelines regarding which QoS/traffic-shaping mechanism to use in a given situation.

Such use cases are application-specific. It will fall to industry bodies such as Avnu and ODVA to develop TSN profiles for these use cases. This task is vitally important in providing system models and behaviors that are consistent across applications and even within the same application. Without consistent models and behaviors, one CNC may configure a network in one way – while another CNC may configure the same network in a different way.

Preemption

Basics of preemption.

Today’s infrastructure components are designed to complete transmission of an entire packet before the next packet can be transmitted, if the initial packet has already been placed in the egress port. Therefore, a 1500 byte packet of lower priority can “hold off” a higher priority packet (~120 us at 100Mb/s wire speed). Preemption (also called Interspersing Express Traffic per IEEE802.3br/IEEE 802.1Qbu) defines a mechanism that allows the switch to stop a transmission in mid-stream in order to allow a higher priority packet to move through the system. Note that only one level of traffic is defined as preemptive.

Seamless redundancy

Frame Replication and Elimination for Reliability (IEEE P802.1CB)

To ensure robust and reliable communication, control systems must be tolerant to packet loss due to congestion, link failures, cable breakage and other faults. To minimize the impact of such faults, P802.1CB aims to send duplicate copies of critical traffic across disjoint paths in the network. If both frames reach their destination, the duplicate copy is discarded. If one copy fails to reach its destination, the duplicate message is still received, effectively providing seamless redundancy.

To minimize network congestion, packet replication can be selected based upon address/traffic class and path information. Likewise, duplicate frame elimination can be based upon address/traffic class and timing. In other words, only critical traffic need be replicated. Best effort and other traffic tolerant to congestion loss can still be transmitted normally.

Ingress policing

Ingress policing generically refers to methods used to prevent traffic overload conditions (e.g., Distributed Denial of Service or DDoS or erroneous delivery) from affecting the receiving node or port. These methods may be used to protect against software bugs on endpoints or switches/bridges but also against hostile devices or attacks. P802.1Qci proposes to provide filtering on a per stream (traffic class) basis by providing an input gate for each stream. These gates would be responsible for passing or blocking a given stream or streams based upon a policing function.

Supported functions include a time window (only allowing streams to pass at a certain time), only allowing specific streams to pass on specific ingress ports, a maximum burst size, a leaky bucket algorithm and others. Note: a leaky bucket algorithm checks that streams conform to defined limits on bandwidth and burstiness (a measure of the unevenness or variations in the traffic flow). Thought of another way: each talker has a contract with a respective listener (excess bandwidth, burst sizes, packet sizes, misuse of labels, etc.). The input gate serves to enforce that contract.

Network utilization of Centralized Configuration.

Centralized configuration

IEEE P802.1Qcc (Stream Reservation Protocol (SRP) Enhancements & Performance Improvements) enhances the existing Stream Reservation Protocol with the addition of a User Network Interface (UNI) which allows for a centralized network configuration (CNC) entity. This CNC can then provide a centralized means for performing network calculus, scheduling and other configuration via a remote management protocol such as NETCONF or RESTCONF. A Centralized User Configuration (CUC) entity communicates to the CNC via a standard API. The CUC may be used to discover end stations, retrieve end station capabilities and user requirements, and configure TSN features in end stations.

Industrial control overview

A Simple Control Model.

Modern industrial networks combine the disciplines of both information technology (IT) and operational technology (OT) to meet the requirements of industrial applications The applications served by the industrial control sector are sophisticated processes requiring complex and modularized designs that often require modification and augmentation during runtime. These applications require high performance, determinism, and predictability; their infrastructures demand highly robust and reliable designs both in hardware durability and in resiliency features to support high-availability needs.

The networking legacy in the industrial control market is long established with roots that extend to multiple industrial technology standards as well as to multiple vendor-specific, proprietary technologies. The industrial installed base that has settled on these technologies is a conservative community of manufacturers that move carefully and slowly from one generation of technology to the next. Solutions need to be proven and ROI is carefully calculated before investments are made. The move from any current position is always done through migration and evolution in order to protect installed assets and to maximize profitability for any given change in architecture.

Prior to the advent of TSN technologies, there hasn’t been much motivation to drive these differing industrial solutions to a more unified approach. The TSN conversation, however, has driven an awareness that future networking solutions will require a holistic approach and a true system perspective. The future network requires a comprehensive answer that includes all elements – infrastructure and end stations alike – to be included in the final solution.

Moreover, in order to solve the complete Industrial Internet of Things (IIoT) problem statement, this comprehensive solution will require the collaboration of the existing, differing, technologies and standards to morph toward solutions that can allow their inclusion in this new eco-system. If any specific technology, vendor, or product does not participate in this new paradigm, there is no way for “the system” to accommodate or plan for the traffic that needs to be managed to or from these components. The result is that these components will either not be properly served in the overall design – or that they will interfere with the overall design in a non-productive fashion.

Conclusions

TSN technologies offer a scalable, predictable approach to deterministic networking. Because Ethernet/IP products have always relied upon standardized technologies, ODVA is in an excellent position to leverage these emerging standards. However, significant challenges remain. The integration of various PTP profile and the convergence of EtherNet/IP traffic with a scheduled TSN network are chief among these challenges.

Jordon Woods, Director, Deterministic Ethernet Technology Group, Analog Devices, and Steve Zuponcic, Technology Manager, Rockwell Automation