As companies with critical infrastructures invest in digital transformation to improve operational efficiency, cyber risks have significantly increased, leading to unscheduled downtime, negative corporate brand perception, as well as data and safety concerns.

Securing and monitoring your network is a core goal for most companies. To accomplish this goal, teams utilize ICS security solutions designed to respond to and manage threats in OT environments efficiently. To properly identify, detect, and respond to security threats and breaches, most ICS security tools focus on threat detection and monitoring, and asset visibility and management.

Implementing these security solutions, OT teams face complex challenges when it comes to architecting connectivity throughout these large and sometimes aging infrastructures. Many  weren’t initially designed with network security in mind, like having to rely on legacy switch SPAN ports for visibility, that aren’t secure, reliable or available.

Manufacturing Topology.

According to SANS State of OT/ICS Cybersecurity Survey, “visibility is critical for managing OT/ICS systems. According to survey respondents, increased visibility into control system cyber assets and configurations is the top initiative organizations are budgeting for in the next 18 months.”

Security and performance strategies start with 100% visibility into network traffic. Security tools need to see every bit, byte and packet or they could miss a threat, and that visibility starts with the packets traversing the network.

A common access point for packet visibility in OT environments has been SPAN ports on a network switch. Many times an engineer will connect a SPAN directly to intrusion detection systems (IDS) or network monitoring tools. But today, in modern ICS networks, network TAPs (test access points) are considered an industry best practice as a more reliable and secure option to access network packets for security and monitoring solutions to properly analyze threats and anomalies.

This high level network topology diagram, following the Purdue model, illustrates how an ICS network monitors various segments. From Level 1 control networks of DCS and PLCs, Level 2 process networks of HMI and engineer workstations, and Level 3 DNS operations to Level 4 data center and security control centers. Instead of mirroring traffic directly from the various switches, this diagram showcases how to properly access the packets with network TAPs and unidirectional Data Diode TAPs, providing complete and reliable visibility to ensure the monitoring solutions are seeing every bit, byte, and packet.

TAP vs SPAN in OT Environments

Determining when you use SPAN ports or network TAPs comes down to a multitude of issues. And many times a combination of both is a visibility architecture reality. But there are some significant differences which affect the integrity of the traffic that is being analyzed, as well as the performance of the network traffic. Let’s review some of the pros and cons of each to help you decide what works best for your network.

OT SPAN.

1. Switch SPAN ports

A common visibility use case is to route mirrored traffic from a SPAN port on the switch to a security or monitoring tool. Port mirroring, also known as SPAN (Switched Port Analyzer), is a designated port on a network switch that is programmed to mirror, or send a copy of, network packets seen on specific ports where the packets can be analyzed.

• SPAN ports provide access to packets for monitoring
• SPAN sessions do not interfere with the normal operation of the switch
• SPAN ports are configurable from any system connected to the switch

The concept is simple enough — the switch is already architected into the environment. Just hook up your security solution. Done. But many times the seemingly simplest path isn’t the best path.

High-level SPAN challenges include:

• SPAN takes up high value ports on the switch
• Some legacy switches do not have SPAN ports even available
• SPAN ports can drop packets, an additional risk for security and regulatory solutions

One of the fundamental reasons security teams do not like to use SPAN is because of dropped packets. In IT environments, this usually happens when the port is heavily utilized or oversubscribed. In OT environments, network switches tend to run 10M, 100M, up to 1G so you may think this will never happen. Unfortunately, ICS switches are prone to drop packets at a lower speed, even when network links are not saturated. This can happen for a variety of reasons:

• Packets sometimes can’t be stored because of a memory shortage
• ‘PAUSE’ frame attack – a bad actor can flood the SPAN disguised as a loopback, hiding bad data and forcing dropped packets
• Packets showing a broken cyclic redundancy check (CRC) will be dropped
• Frames smaller than 64 bytes or bigger than the configured maximum transmission unit (MTU) can be dropped because of an ingress rate limit

If dropping the packets isn’t an eye opener, SPAN also:

• Will not pass corrupt packets or errors
• Can duplicate packets if multiple VLANs are used
• Can change the timing of the frame interactions, altering response times

The SPAN concept may have sounded easy because it was available, but after weighing packet loss and altered frames, additional SPAN security considerations include:

• Bidirectional traffic opens back flow of traffic into the network, making the switch susceptible to hacking
• Administration/programming costs for SPAN can get progressively more time intensive and costly

2. Network TAPs

The industry best practice for packet visibility is network TAPs (test access points). Network TAPs are purpose-built hardware devices that create an exact full duplex copy of the traffic flow, continuously, 24/7 without compromising network integrity.

Instead of connecting two network segments, such as routers and switches directly to each other, the network TAP is placed between them to gain complete access to traffic streams. TAPs transmit both the send and receive data streams simultaneously on separate dedicated channels, ensuring all data arrives at the monitoring or security device in real time, without affecting the traffic between the segments.

• Network TAPs make a 100% full duplex copy of network traffic
• Network TAPs do not alter the data or drop packets
• Network TAPs are scalable and can provide a single copy, multiple copies (regeneration), or consolidate traffic (aggregation) to maximize the production of your monitoring tools

TAP v SPAN, which wins?

The two methods for network access both provide ICS security tools packets to monitor and protect the network. Understanding the options and challenges allows teams to better architect and secure the network.

Network TAPs

• 100% full duplex copy of network packets
• Enables faster troubleshooting
• Ensures no dropped packets, passing physical errors and supports jumbo frames
• Does not alter the time relationships of frames
• Are passive or failsafe, ensuring no single point of failure (SPOF)
• TAPs are secure, do not have an IP address or MAC address, and cannot be hacked.
• Data Diode TAPs provide unidirectional traffic to protect against back flow of traffic into the network

SPAN

• Provides access to packets for monitoring
• Can take up high value ports on the switch
• SPAN traffic is the lowest priority on the switch
• Some legacy switches do not have SPAN available
• SPAN ports drop packets, an additional risk for security and regulation solutions
• Will not pass corrupt packets or errors
• Can duplicate packets if multiple VLANs are used
• Can change the timing of the frame interactions, altering response times.
• Bidirectional traffic opens back flow of traffic into the network, making switch susceptible to hacking
• Administration/programming costs for SPAN can get progressively more time intensive and costly

Putting TAP vs SPAN to the test

In a test conducted by a 3rd party (Packet Pioneer), with the goal to see the difference between a data stream captured on a network TAP versus a SPAN port.

The test connected two PCs to a basic Cisco Catalyst Switch at 100Mbps. A throughput test using iPerf was configured and run between the two machines. On one of the PCs, placing a 100Mbps TAP, and a hardware analyzer to capture. Lastly, they configured a SPAN on the switch to forward all traffic to and from this port to another hardware analyzer.

The throughput test finished with a result of 93.1Mbps sustained for 10 seconds between the two PCs.

The SPAN data capture showed almost 8,000 packets missing from the trace. This represents almost 8% of the total packets that were captured by the analyzer from the network TAP. We should also point out that this was on a 100Mbps interface, not a Gigabit interface, and there were no errored frames. The switch bus was not in a near overloaded state.

Also, the difference in the timing between the TCP SYN and SYN ACK in the two traces shows us that the switch is not treating both the SPAN and destination ports the same. In fact, it was forwarding traffic to the SPAN port faster than the true destination. While the difference is only 21 uSec, it shows that the switch is affected when SPAN is enabled. It is not as seamless as it would appear, and this delay was under no load test. With the switch loaded with traffic, the losses and timing will show greater differential and also dropped packets. The results are clear that the network TAP outperformed the SPAN in a head to head packet test, which can mean missing a threat or successfully containing a breach.

Following critical infrastructure’s guiding principles — OT teams build their networks to last, to ensure minimal to no network downtime. These concepts rest on the network infrastructure and visibility architecture. Incorporating best practices like network TAPs in the network will help achieve these goals.

Critical infrastructure visibility solutions for OT provide industry the most reliable Network TAP, Data Diode, Network Packet Broker and cloud visibility solutions, and deliver packet visibility while ensuring the secure connectivity needed.

Chris Bihary, CEO and Co-founder, Garland Technology