Critical system applications are often required to maintain high availability for all transmission, generation, distribution, and communication network components. For critical infrastructures and time sensitive processes, downtime is never allowed. These systems may incorporate redundancy to cope with points of failure in their infrastructure. The key performance factor of redundancy is the recovery time. Recovery time is the time needed to restore normal operation after a disruption.

A key characteristic of recovery time is its determinism, i.e., the certainty that the desired recovery time is met. The time required for a dynamic network control protocol to recover from an equipment failure may be unacceptable in certain applications. Stringent system applications require zero recovery (switchover) time. FRER provides zero recovery time using frame replication with multiple paths called compound streams.

This article is an introduction to developing high availability for networks in EtherNet/IP systems using the Frame Replication and Elimination for Reliability (FRER) protocol for demanding high availability applications in industrial scenarios.

High availability

Dynamic network redundancy example. High availability achieved economically by detecting points of failure and avoiding service interruptions in the system.

High availability is based on the concept of availability. The availability of a network is the probability (in percent) that the network is in service and available for use at any instant in time.

High availability is represented as a percentage, usually referred to as the 9s. If the availability metric is specified as five nines, it is understood to mean that the network should be functional for 99.999% of the desired duty cycle (24 hours/day).

The availability of a network can be increased by designing network elements that are highly reliable (high MTTF), and/or by reducing the time required to repair the network and return it to service (low MTTR).

Since it is impossible to create networks that never fail, the key to high availability is to make recovery time as brief as possible. Availability is increased in networks by introducing redundancy.

Redundancy

Static network redundancy example.

High availability can be achieved economically by using techniques that detect points of failure and avoid service interruptions through redundancy in the system. There are two forms of redundancy, dynamic and static.

In dynamic (standby) redundancy the replicated components activate after a failure has been detected. Dynamic redundancy does not actively participate in the control. Switchover logic determines when to insert and activate redundancy. This requires detection and/or recovery.

In static (parallel) redundancy the replicated components are active concurrently. Static redundancy usually participates in the control. No special processing is needed on errors. This provides bumpless (0 ms) switchover, with continuously exercised redundancy and increased point-of-failure detection with fail-safe behavior. Static redundancy is provided at the cost of duplication.

The Frame Replication and Elimination for Reliability (FRER) is a static (parallel) redundancy high availability capability as defined the the IEEE 802.1CB-2017 standard.

The FRER solution

The Frame Replication and Elimination for Reliability (FRER) is a static (parallel) redundancy high availability capability as defined the the IEEE 802.1CB-2017 standard. This solution will be illustrated by transforming an EtherNet/IP Network into a FRER supported network.

FRER provides increased reliability (reduced packet loss rates) for a Stream by using a sequence numbering scheme, and by replicating every stream packet in the source. FRER also eliminates those replicated stream packets in the destination. FRER provides the following:

• Packet replication: sending replicated frames on separate paths, and then using inserted identification information to eliminate replicates, reducing the probability of frame loss.
• Multicast or unicast: A path on which a Stream is sent can be a point-to-point path or a point-to-multipoint tree.
• Latent error detection: some means of detecting a failure to deliver copies of each packet is provided at the point that the replicated packets are discarded.
• Interoperability: a small number of controls are provided that that make interoperation with other standards possible.
• Backward compatibility: To provide the ability to be connected to a network that is not aware of FRER, and for a network of conformant relay systems to offer these benefits to unaware end systems.
• Zero congestion loss: provide a Stream with zero (or very low) packet loss due to congestion.

A FRER supported network consists of several device types through the combination of defined FRER component and equipment elements.

The FRER protocol provides increased reliability (reduced packet loss rates) for a Stream by using a sequence numbering scheme, and by replicating every stream packet in the source end system and/or in relay systems in the network.

FRER also eliminates those replicates in the destination end system and/or in other relay systems. The devices types described in the standard are described below.

End systems

End Systems may contain a Talker component, a Listener component, or both. The Talker component is the originator of single or redundant streams.

When generating redundant streams, the Talker component performs the following functions: supplies sequencing information in frames; replicates each frame, assigning each replicate a different stream handle, at most one of which can be the same as the original passes unchanged; and encodes the sequencing information into the frame in a manner such that it can be decoded by its peer.

The Listener component consumes single or redundant streams. When receiving redundant streams, the Listener component performs the following functions: extracts and decodes the sequencing information from a received frame; and examines this sequencing information in received frames packets and discards frames indicated to be a duplicate of a frame previously received; and also monitors the variables to detect latent errors of Streams.

Relay systems

Relay Systems will either transfer packets belonging to redundant streams, or act as a proxy Talker or Listener for End Systems not capable of handling redundant streams. As a Proxy the Relay system performs the following functions.

The Talker proxy will generate redundant streams from single streams:

• Supplies sequencing information in frames;
• Replicates each frame passed to it, assigning each replicate a different stream handle, at most one of which can be the same as the original passes unchanged;
• Encodes the sequencing information into the frame in a manner such that it can be decoded by its peer.
• When the Listener proxy receives redundant streams:
• Extracts and decodes the sequencing information from a received frame.
• Examines this sequencing information in received frames packets and discards frames indicated to be a duplicate of a frame previously received and forwarded;
• Also monitors the variables to detect latent errors of Streams;

Relay-End systems

Relay-End Systems are not defined in the IEEE Standard but are elements within the EtherNet/IP Network. The Relay-End System is created by combining the FRER End system and Relay System capabilities. Relay-End Systems are represented by the Multi-Port IO Device(s) and one Multi-Port Sensor.

FRER in a ring topology

This section describes the use of FRER in the Ring network topology portion of an EtherNet/IP Network example. The FRER protocol is defined to interoperate with any of the Network Control Protocols defined in the IEEE 802.1Q-2018 Standard (i.e. RSTP, MSTP, SBB, etc.). The FRER protocol is described to also work with the IEC 62439-3 (PRP and HSR) network control protocols, illustrated later in this article.

In this scenario the Multi-Port IO Devices is a Relay-End System (RES) acting as a Listener/Talker that receives/transmits redundant streams to the Sngl-Port Controller, which is and End System (ES).

The Multi-Port IO Device (RES), acting as a Talker generating a redundant stream to the One-Port Controller (ES), performs the following FRER functions:

• Supplies sequencing information in frames;
• Replicates each frame, assigning each replicate a different stream handle, at most one of which can be the same as the original passes unchanged;
• Encodes the sequencing information into the frame in a manner such that it can be decoded by its peer.

The Multi-Port IO Device (RES), acting as a Listener receiving a redundant stream from the One-Port Controller (ES), performs the following FRER functions:

• Extracts and decodes the sequencing information from a received frame.
• Examines this sequencing information in received frames packets and discards frames indicated to be a duplicate of a frame previously received and forwarded;
• Also monitors the variables to detect latent errors of Streams.

The other Multi-Port IO Devices (Relay-End Systems) and Ethernet switch (Relay System) are acting as relay systems which transfer frames belonging to redundant streams.

One Ethernet Switch (RS), acting as a relay system for the Sngl-Port Controller, performs the following FRER functions. The Talker proxy will generate redundant streams from single streams:

• Supplies sequencing information in frames;
• Replicates each frame passed to it, assigning each replicate a different stream handle, at most one of which can be the same as the original passes unchanged;
• Encodes the sequencing information into the frame in a manner such that it can be decoded by its peer.

When the Listener proxy receives redundant streams:

• Extracts and decodes the sequencing information from a received frame.
• Examines this sequencing information in received frames packets and discards frames indicated to be a duplicate of a frame previously received and forwarded;
• Also monitors the variables to detect latent errors of Streams.

Use of FRER in a Parallel Redundancy Protocol (PRP) and a High-availability Seamless Redundancy (HSR) network. Control protocols are defined in IEC 62439-3 standard.

FRER in a mesh topology

This section describes the use of FRER in the Mesh network topology portion of the EtherNet/IP Network example. The Mesh network is supported any of the Network Control Protocols defined in the IEEE 802.1Q-2018 Standard (i.e. RSTP, MSTP, SBB, etc.). The network recovery time of these control protocols is irrelevant due to the seamless redundancy nature of the FRER protocol.

In this scenario, the IP Camera is an End System (RES) acting as a Talker that transmits a stream to both the Camera Recorder and the Monitor, both are End Systems (ES).
The Ethernet Switch is a Relay System (RS) connected to the Monitor is acting as a translator from the single stream to a redundant stream for the IP Camera End System (ES):

The Talker proxy will generate redundant streams from single stream:

• Supplies sequencing information in frames;
• Replicates each frame passed to it, assigning each replicate a different stream handle, at most one of which can be the same as the original passes unchanged;
• Encodes the sequencing information into the frame in a manner such that it can be decoded by its peer.

The Multi-Port IO Devices (Relay-End Systems) and Ethernet switch (Relay System) are acting as relay systems which transfer frames belonging to redundant streams.
Two Ethernet Switches (ES) are connected to the Monitor and Camera Recorder are acting as Listeners for the Redundant stream from the IP Camera.

When the Listener proxy receives redundant streams:

• Extracts and decodes the sequencing information from a received frame.
• Examines this sequencing information in received frames packets and discards frames indicated to be a duplicate of a frame previously received and forwarded;
• Also monitors the variables to detect latent errors of Streams.

In this scenario, the Multi-Port Sensor is an End System (ES) acting as a Listener/Talker that receives/transmits redundant streams to the Sngl-Port Controller, which is and End System (ES).

The Multi-Port Sensor (ES), acting as a Talker generating a redundant stream to the One-Port Controller (ES), performs the following FRER functions:

• Supplies sequencing information in frames;
• Replicates each frame, assigning each replicate a different stream handle, at most one of which can be the same as the original passes unchanged;
•  Encodes the sequencing information into the frame in a manner such that it can be decoded by its peer.

he Multi-Port Sensor (ES), acting as a Listener receiving a redundant stream from the One-Port Controller (ES), performs the following FRER functions:

• Extracts and decodes the sequencing information from a received frame.
• Examines this sequencing information in received frames packets and discards frames indicated to be a duplicate of a frame previously received and forwarded;
• Also monitors the variables to detect latent errors of Streams.

The Multi-Port Sensor (Relay-End System) and Ethernet switches (Relay System) are acting as relay systems which transfer frames belonging to redundant streams.

One Ethernet Switch (RS), acting as a relay system for the Sngl-Port Controller, performs the following FRER functions:

The Talker proxy will generate redundant streams from single streams:

• Supplies sequencing information in frames;
• Replicates each frame passed to it, assigning each replicate a different stream handle, at most one of which can be the same as the original passes unchanged;
• Encodes the sequencing information into the frame in a manner such that it can be decoded by its peer.
• When the Listener proxy receives redundant streams:
• Extracts and decodes the sequencing information from a received frame.
• Examines this sequencing information in received frames packets and discards frames indicated to be a duplicate of a frame previously received and forwarded;
  • Also monitors the variables to detect latent errors of Streams.

FRER in PRP and HSR topologies

This section describes use of FRER in a Parallel Redundancy Protocol (PRP) and a High-availability Seamless Redundancy (HSR) network. The PRP and HSR control protocols are defined in the IEC 62439-3 standard.

There are additional devices in this network that need to support the FRER protocol. These devices are:

PRP and HSR Redundancy Boxes [Redbox] (RS): These are Relay Systems will need perform the standard FRER Relay System functions as well as their respective PRP or HSR function. The redboxes will substitute the FRER protocol tag with either the PRP or HSR protocol tag.

In this scenario the Multi-Port IO Devices is a Relay-End System (RES) acting as a Listener/Talker that receives/transmits redundant streams to the Sngl-Port Controller, which is and End System (ES).

The Multi-Port IO Device (RES), acting as a Talker generating a redundant stream to the One-Port Controller (ES), performs the following FRER functions:

• Supplies HSR sequencing information
• Replicates each frame, assigning each replicate a different stream handle, at most one of which can be the same as the original passes unchanged;
• Encodes the sequencing information into the frame in a manner such that it can be decoded by its peer.
  The Multi-Port IO Device (RES), acting as a Listener receiving a redundant stream from the One-Port Controller (ES), performs the following FRER functions:
• Extracts and decodes the HSR sequencing information from a received frame.
• Examines this sequencing information in received frames packets and discards frames indicated to be a duplicate of a frame previously received and forwarded; Also monitors the variables to detect latent errors of Streams;

The other Multi-Port IO Devices (Relay-End Systems) within the HSR network and one HSR Redbox (RS) are acting as relay systems which transfer frames belonging to redundant streams.

Two HSR Redboxes (RS) are acting as relay systems which transfer frames belonging to redundant streams to the PRP Network, converting the sequencing information to the PRP Trailer tag. One HSR Redbox will generate a stream for the red network, the other will generate the stream for the blue network. The Ethernet switches (Relay System) in the PRP network are acting as relay systems which transfer frames belonging to redundant streams.

One PRP Redbox (RS), acting as a relay system for the Sngl-Port Controller, performs the following FRER functions.

The Talker proxy will generate redundant streams from single streams:

• Supplies PRP sequencing information in frames;
• Replicates each frame passed to it, assigning each replicate a different stream handle, at most one and of which can be the same as the original passes unchanged;
• Encodes the sequencing information into the frame in a manner such that it can be decoded by its peer.

When the Listener proxy receives redundant streams:

• Extracts and decodes the PRP sequencing information from a received frame.
• Examines this sequencing information in received frames packets and discards frames indicated to be a duplicate of a frame previously received and forwarded; latent errors of Streams;

Conclusion

With the addition of FRER, an existing network can be converted to a seamless redundant network by changing the core network equipment and adding some additional interconnections. Most on the interconnection wiring and Edge Equipment can be reused.

This is less expensive than having the duplicate the network or replacing the Edge Equipment as well. This will allow the convergence of existing/legacy network with new TSN deterministic (Guarantee-of-Service vs Quality-of-Service) capabilities.

George A. Ditzel III, Ethernet Architect, Schneider Electric