Based on our experience visiting and touring multiple customer plants, we can safely say that whether you are building an airplane or making a chocolate-chip cookie, the processes (with slight modifications) are similar, the safety and compliance challenges are almost the same.

In addition, the network and security challenges are almost identical. Security at the manufacturing zone has always been a tough equation to solve. It requires a delicate balance among performance (with automation), compliance (and visibility), internal polices (traffic movement), and the various “personas” that operate the environment (process ownership).
In this article, we introduce a security and collaboration model that utilizes Cyber Vision technology for monitoring on industrial networks and subsequently uses the power of WebEx Teams to create “virtual spaces” or communities of interest to react to a certain reported event. Cyber Vision provides full visibility into the Industrial Network that includes asset inventory, real-time monitoring, and threat intelligence.

Problem Statement

Manufacturers need to protect plants from external threats and protect employees from accidents due to unplanned events. The introduction of Ethernet based protocols within Industrial control networks has brought tremendous operational efficiency by allowing control devices like Programmable Logic Controllers (PLCs) and field devices like Robots to be automatically upgraded, continuously monitored to predict future failures and proactively schedule maintenance to avoid unnecessary downtimes. The efficiency is made possible due to the ability to communicate with the control devices through remote network connections and to stream telemetry events from field devices in the plant to the Robotics vendor.

This network connectivity also increases the potential threat vectors and poses a serious security challenge to manufacturers. The challenge is exacerbated by rich ecosystem of control and field devices from different vendors, talking different protocols and running different software versions. The key question is how to detect, manage and mitigate a security threat in a multi-vendor industrial control network?

Detect and inform

Cyber Vision provides a unique solution to this security challenge by providing real-time insights into communication flows within an industrial control network. The network switches such as Cisco IE3400 and Cisco Catalyst 9300 series run a Sensor application which captures the traffic from OT components such as PLCs, Supervisory Control and Data Acquisition systems (SCADA), decodes the packets, analyzes the communication exchanges and sends metadata to Cyber Vision.

Secure Collaboration Spaces for OT and IT teams.

The network sensors automatically discover the type, software version, vendor details of the device through deep inspection of the packets and are able to understand protocol message flows that correspond to activities such as reading a variable value from a PLC, writing a variable value to a PLC, file transfers and upgrade processes. The flow metadata sent to Cyber Vision enables it to form an understanding of the normal operation of the industrial control network in a given plant.

Cyber Vision continuously analyzes the traffic streams from sensor applications and compares the identified activities against known communication patterns and detects anomalies in the network as and when an unexpected known or unknown event happens in the network. These anomalies are brought to OT (operational technology) admin’s attention through critical, high, medium and low events in Cyber Vision Center web-based dashboard.

The detection of an anomaly event doesn’t necessarily mean that it is a security threat. Some of these events could be triggered by an unplanned change by process control engineers working in the plant while some of them could be real threats triggered by malware running in the OT and IT devices. The next key question that arises is how do we differentiate between unplanned change management events and security threat events and take swift action to mitigate the risks and protect the control network?

It is essential to inform plant experts about anomaly events and facilitate collaboration among the experts to discuss and determine next steps such as acknowledge and ignore the event, invoke an emergency safety procedure etc. The event notifications displayed in a Cyber Vision center dashboard may get unnoticed due to the sheer volume of events.
Supervisors and Experts who are responsible for a particular event (also called as Community of interest) need to engage with their peers manually through out of band communication channels such as email, push-to-talk, walkie-talkie, phone call or call for a meeting. The API based integration between Cisco Cyber Vision and Webex Teams platforms automate the engagement of the “community of interest” through secure collaboration virtual spaces.

An example deployment illustrates how the Monitoring application periodically checks for anomaly events using the Cyber Vision APIs. When events of interest such as critical, high severity event types happen, the monitoring application determines the list of resource roles to be engaged based on an Event category to Resource roles mapping table. It then retrieves information about plant employees with the identified role and who is currently on duty/shift.

Cyber Vision dashboard.

The application requests Collaboration Orchestrator to create a secure collaboration space and adds the on-shift OT experts as participants. IT experts and external vendors are added as required depending on the type and criticality of the event. The monitoring application also posts relevant information of the event to the collaboration space for expert’s review. The application could also add bots to the space to augment the review process. The bots can be used by experts to get quick information about control devices and machines right within the collaboration space instead of manually retrieving information from multiple systems.

Supervisors and process control experts get notified through their mobile, desktop based Webex Teams clients and start to review the event. They collaborate and decide on next steps swiftly. They could even use the bot participant to execute administrative next steps such as acknowledge and mark the event as an unplanned, no harm event. In other scenarios, they could invoke safety procedures to protect the industrial control network from the identified threat or trigger troubleshooting processes to isolate the source of the issue, execute a task to mitigate the impact and perform root cause analysis in order to prevent future threats.

The combination of security and collaboration platforms paves the way to accelerate engagement of “community of experts” to address anomalies in an industrial control network of a manufacturing plant faster and more effectively.

Chidambaram Arunachalam, Principal Engineer and Hazim Dahir, Distinguished Engineer, Cisco Systems.