IN 2015, THE BLACK ENERGY TROJAN attacked the Ivano-Frankivsk power station in West Ukraine and knocked out electricity in 80,000 homes. The bug was delivered via spear phishing emails, disguised as a Microsoft Office attachment from the Ukrainian parliament, and had the ability to delete critical system files and specifically sabotage industrial systems.

With the increasing frequency of malware outbreaks, cyber security for industrial control systems (ICSs) such as energy, chemical, oil and gas, and other manufacturing markets has become a pressing issue. The idea that malicious attacks in the digital world can damage people and equipment in the physical world is a bit sobering.

And according to an extensive and detailed study done by the Kaspersky Lab, the threat has only increased, nearly 40% of industrial computers experienced cyberattacks in the second half of 2016.

Defense-in-depth cyber security

It’s going to take more than your average firewall to stop something that is specifically made to take advantage of all the holes and gaps in established factories and plants. And with the growing popularity of new smart manufacturing tech and the Industrial Internet of Things (IIoT), it may seem like an even bigger headache to figure out what exactly is going on with your network, much less protect it.

Rather than put your trust in one really, really strong firewall (and hope it’s bulletproof against a wide range of attacks), the key is to build a multi-layered, multi-strategy defense system that can prevent security breaches from many different fronts, as well as slow down attackers to buy you the time to respond and counteract.

Defense in depth is a well-known cyber security philosophy, and the U.S. Department of Homeland Security and numerous national organizations have released publications that go into painstaking detail about recommended ICS security programs and architectures.

Based on the defense in depth model and our extensive hands-on experience with ICS networks, we’ve put together our short list of very practical, high-level action items for the control systems engineer or superintendent in charge of managing cyber security. The goal of this white paper is to present you with low-hanging fruit (and some high-hanging ones) that you can tackle right now in order to strengthen your ICS.

Understand ICS network

You can’t protect what you don’t understand, so the first step to creating a defense system is to map everything out. Make sure all your documentation is up-to-date before you get started, and look at the network diagrams to study the system as a whole.

Know exactly what is running in your current environment, such as network devices switches, routers, servers, PLC’s, OIT’s VFD’s, and other plant Ethernet devices. This also includes software components, like firewalls and applications accessing the network. Find out if your hardware firmware and software patches are up-to- date and if they can even be upgraded. Some older products may no longer be actively supported by the vendors, which can leave security weaknesses.

Since one of our goals is to plug up any holes in our wall, you’ll need to be able to tell what is a necessary pathway for data flow (e.g., sending data for reporting purposes) versus what is an unnecessary gap (e.g., an old device with backdoor access that is no longer needed).

By understanding the system’s purpose and its end product, you can then decide what data flow should be occurring and what devices are allowed to communicate in order accomplish the overall goal.

Understand network segmentation

Network segmentation usually refers to the splitting of a primary network into subnetworks for security and other purposes. It’s like splitting your house into two living areas, with a locked door or security system in between. If a thief gets into the first half of the house, they don’t have automatic free entry into the second half. In the same way, many companies have a corporate network that runs day-to-day business activities and an industrial network for the factories, usually separated by firewalls that control traffic between the two. Determine how your network is segmented (or not).
Assess current network status
Once you have a good idea of the current status of your network, it’s time to identify any key liabilities and develop an action plan. How do we sufficiently protect the system while still letting it accomplish the intended objectives? Your vulnerability assessment should include the following questions:

Know your data

In the previous section, we briefly touched on knowing your network well enough to make educated decisions about data flow. What kind of data is flowing outbound of your ICS, and is it secure? Is there any data flowing inbound to the ICS, and if so, is it also secured? If you see data flow occurring that may be a gray area, it might be worth doing a little risk analysis to justify its presence.

Review firewalls & segmentation

Let’s take a closer look at your network segmentation. First, does segmentation between your ICS and business networks even exist? If the answer is yes, does that firewall need to be updated to the latest firmware? Plus, have you reviewed your firewall rules lately?

Your firewall is only as good as its security policies, so identify any legacy rules that should be removed or revised. Furthermore, set regular review times in your calendar so you won’t forget to reassess those firewall policies on a defined schedule. Document the reviews and record any changes that are made.

Security technology is always getting better and better, so you may want to upgrade your network devices to something with more intelligent features – such as malware scanning, deep packet inspection (DPI), intrusion detection systems (IDS), intrusion prevention systems (IPS), and more.

Finally, have you considered installing more firewalls? Micro- segmentation with multiple firewalls is a nice technique that can be used to protect specific areas of your environment by maintaining tighter control on certain kinds of traffic. Just like how the military has different levels of security access, you may decide to deploy different sets of firewall rules that are laxer or stricter depending on the type of traffic.

Reconsider air gapping

Many people might think that air gapping an ICS network, physically isolating a secure network from unsecured ones, is the be-all and end-all of cyber security. However, air gapping makes system updates, patches and design changes, and remote support very cumbersome. Someone will have to manually download all software revisions onto a USB drive (or other portable device) and carry it into the ICS network every time the slightest update is needed. Plus, what if that USB drive becomes infected malware?

You have now introduced malware to critical systems that may not be up to the latest patch level because there is no avenue to efficiently manage those updates. The reality is that almost all ICS networks require at least a limited amount of information from the outside world to help keep them secure. Fortunately, modern-day network edge security devices make it much easier to effectively and securely manage that information coming into your ICS network.

Scrutinize remote access

Firewalls are meant to make unauthorized access difficult for external sources, so most companies allow remote access via virtual private network (VPN), remote desktop services, gateways, and such. However, is your remote access secure? Is there a defined policy for remote access, and is that policy the same for everyone? Perhaps you need to put specific policies in place for different groups that restrict access for unnecessary areas of the network. For example, certain employees may have full access whereas contractors have a limited view of what’s on the network.

In 2016, the U.S. Justice Department announced charges that the Iranian government had sponsored a coordinated campaign of cyberattacks and obtained illegal access to the ICS of the Bowman Dam in Rye, New York. The SCADA system was attached to the internet via a cellular modem (probably to allow technicians to remotely access the equipment) and unfortunately had no firewall protection or authentication access controls in place. The hacker actually had the ability to release water from the dam through his remote access, but by a stroke of luck, the sluice gate happened to be offline for maintenance at the time.

Legacy hardware and backdoors

Time to dig out the skeletons in the closet, or rather those legacy devices collecting dust in the corner but still connected to the network for some reason. Can any of them be decommissioned? These relic pieces of the system are never updated or patched and no one’s paying attention to them, so the easiest thing to do is retire them.

If the legacy devices are still needed, are they running any outdated or unused protocols? For example, sometimes HTTP or Telnet is enabled but is not actually used to manage the device. These insecure protocols often do not have authentication or encryption options available and therefore pose a serious security risk.

Another often forgotten security loophole to watch for is backdoors into your network placed by vendors of specific systems. For easy installation and customer support, vendors may use a cellular modem to bypass your network and directly access the equipment to provide instant troubleshooting (just like the New York dam). Unfortunately, you have little to no control over the security procedures of these backdoors, so you need to shut them down before they become a problem.

Stay up-to-date with software

How current is your software? It may seem like an obvious must- do, since most security vulnerabilities are covered with the latest patches. But in reality, probably 90% of ICSs aren’t on the latest operating system, firmware, or patches for their equipment.

Auto-updates for Windows Server and other software are turned off because the new versions may be incompatible with equipment, bringing your factory to a screeching halt. So instead, people just repeatedly ignore software update notifications, leaving your ICS open to the next big security threat.

Rather than skirting the issue forever, develop a systematic plan to upgrade software on your all devices. For example, schedule a quarterly software update where you check with the vendors, have them run internal tests to make sure nothing will break, and then correctly patch your machines. Depending on the specifics of the system, this can be done more frequently as needed.

Formalize backup procedures

Do you have a formal plan for backup and disaster recovery of your ICS? This means having a documented strategy for how the systems are backed up and where the backups are kept for disaster recovery. If you do, do you save your backups offsite in remote storage and is it secure? Have you tested your backups to verify integrity? The last thing you want after months of faithfully saving your files is to find out that the backups are corrupted or unusable for some reason.

Vulnerability assessment baseline

A vulnerability assessment completed by a cyber security expert prior to any action can provide deep insight into your network, as well as a baseline to measure the success of your action plan. A vulnerability assessment consists of thoroughly identifying, classifying and prioritizing the vulnerabilities in your network. The assessment may also include a penetration test by a white hat ethical hacker to simulate various types of attacks and determine what systems are potentially open to exploit.

Form an actionable plan

After you’ve answered all the previous questions to identify the potential weaknesses in your ICS, it’s time to form an actionable plan that makes sense for your business and execute it. Much of your plan will match the areas surveyed in the vulnerability assessment. This process of eliminating security risks and strengthening your defenses is also known as system hardening or device hardening.

Physically secure devices

All of the attacks we’ve talked about so far have been digital in nature, but don’t forget that you need to physically secure your network as well. You may have an area of your system that is easily accessible by the general public, and if these devices are not physically secured, the entire system may be open to an attack.

Remove legacy devices & software

For existing parts of the system, complete the necessary changes by decommissioning legacy devices and updating the current ones. This may be as simple as refreshing the firmware or as big as installing a whole new operating system.

Install new devices and firewalls

Oftentimes legacy devices are replaced with new releases that are actively supported by the vendor and can still be updated. Purchase and install any new devices as needed. Also, if you need to segment your network for greater security, install new firewalls with the appropriate security policies.

Turn off unused services & protocols

A service is a type of application that runs in the background but usually doesn’t have a user interface. Some of these behind- the-scenes programs provide core functionality, but others are unnecessary to your ICS.

These nonessential services will just sit there and run by default, so go through all those devices and understand what services are being used. For example, a switch may have Telnet remote access automatically turned on, so you may need to disable it and configure SSH (Secure Shell) as a more secure, encrypted network protocol.

The bottom line is to turn off unnecessary services and packages, configure port security, and shut off unused ports. If you’re not as familiar with services and ports used in SCADA systems, it may be easier to get help from a knowledgeable expert rather than potentially break something. Create a secure data flow plan for inbound and outbound traffic.

There are many ways to get data in and out of a secure ICS network, such as a data-diode, configuring application mirrors, or installing data replication servers. To determine what type of controls need to be installed, make a full assessment of the data flow that traverses the edge of the ICS network.

Develop an incident response plan

In an ideal world, your cyber security defenses will be impenetrable to any and all forms of malicious attacks. However, what happens if your ICS is indeed compromised and you’re seeing strange things happen? Don’t panic! Instead, it is of paramount importance that you are prepared with a step-by-step emergency plan. Who is part of the cyber security response team? Who should be contacted if you start to see anomalies?

What are the next actions to take? Your company is equipped with workplace emergency and evacuation procedures in case of a fire or other crisis, so in the same way you should have a robust incident response plan for network-borne threats.

In March 2016, the Kemuri Water Company (Verizon’s pseudonym for an unnamed U.S. water utility plant that they investigated) was unknowingly breached by hackers who had access to hundreds of programmable logic controllers (PLCs). According to Verizon, “they managed to manipulate the system to alter the amount of [water treatment] chemicals that went into the water supply and thus handicap water treatment and production capabilities so that the recovery time to replenish water supplies increased.” Fortunately, the threat actors weren’t as familiar with the SCADA system and the water company’s well-trained crew was able to react quickly and minimize downtime using manual procedures (even if they didn’t realize the cause was from hacking).

Given how automated and smart ICSs are becoming in today’s world, it’s all the more vital to have both manual and digital emergency procedures in place. If parameters are starting to run amok, you need to have a plan in place to determine the root cause and respond accordingly, whether it’s a machine failure, user error, or security breach.

Post-action network assessment

After you have finished executing the action plan and eliminating vulnerabilities, don’t forget to reassess the system and complete a network health checkup. This will allow you to determine how effective your action plan has been and determine possible new threats. The timeline for executing your security plan will depend on the size and complexity of the network and your company’s needs.

In the fastest cases, a business may choose to do a quick first pass with system hardening and finish making the most important changes within a week. In other cases, the corporation may only allow one portion of the ICS to be taken down at a time or may only allow system shutdown to occur once a year on Christmas. Depending on how stringent the plant’s needs are, it may take more than 6 months to complete the changes within small time slots here and there.

Monitoring

In June 2016, the highly sophisticated SFG malware was detected on the networks of a European energy company. Its goal is to intentionally create a backdoor on ICSs that will allow the program to “extract data or potentially shut down the energy grid,” according to security firm SentinelOne Labs. The malware is especially scary due to its knack for bypassing traditional Windows-based antivirus software and firewalls and for rendering itself temporarily inactive to evade detection by security analysts.

The reality is that bad stuff might eventually be able to get through your cyber security walls, so will you have the ability to catch them? As seen above, hackers have begun reverse engineering popular antivirus software to take advantage of their weaknesses, so you cannot just rely on techniques that block attacks. A truly layered defense system also needs the capability to detect and stop the breaches if and when they occur.

Log and correlate data

Basic network monitoring 101 means you need some kind of logging in place. Many people set up a central log collector using tools that range from freeware all the way up to enterprise level software with 24/7 customer assistance.

Does your company have security information and event management (SIEM) or other security-focused software that can make sense of the logged data? A SIEM is a good first line of defense because it tracks and manages the logging, correlates to other events happening on the network, and creates alerts for suspicious anomalies. Plus, the SIEM software can recommend next steps and even automatically execute some of them.

Regularly scheduled maintenance

Finally, we’ve reached the last step. After having made all this upfront effort to bolster the cyber security of your ICS, you can’t just cross it off your checklist and forget about it. As with any good defense, regular maintenance is needed to make sure you are protected against the latest and greatest malicious threats.

Updates and reviews

Everything we talked about so far must be kept up-to-date on a frequent basis. That means maintaining current network documentation and evaluating any configuration changes. It also involves scheduling consistent times for installing software patches on all devices. And don’t forget to regularly review firewall rules and SIEM data for relevancy.

Practice the incident response plan

Maintain your incident response plan to make sure it’s updated and relevant to what’s installed on the ICS network. Run a test at least once a year by holding “cyberattack drills” (just like fire drills) to make sure people have a chance to practice their roles.

Train employees

While the manager or team in charge of cyber security may be wise to the latest computer viruses and scams, it does no good if the rest of the company is uninformed. Train all employees on proper standards for avoiding security threats and consider implementing occasional “practice drills” for them as well. For example, some businesses have a required online cyber security course for new hires. A few days after completion, IT sends a fake email (with lots of malware tells) that seems to come from a manager of another department. If the recipient clicks the link in the message, they receive a notification that their behavior could have resulted in the download of malware.

Audit vulnerabilities regularly

Remember the basic vulnerability assessment you completed in Part 2? These audits are an ongoing task that needs to be repeatedly scheduled – a common theme you’ll find with many security best practices. If you have the tools and resources, you can do this internally on a quarterly basis, but it is recommended that you hire a third party company to complete an annual assessment that gives you a second opinion and makes sure you’re not missing anything.

Developing a solid security system is extremely important in this day and age, especially as ICS networks are expanding to include greater automation and IIoT technology. You need someone dedicated to driving these cyber security initiatives or else they’ll quickly get forgotten in favor of other plant priorities and initiatives. So who is the right person for the job?

Some businesses choose to use their own IT departments or hire external companies that specialize in IT security, but just remember that corporate networks and ICS networks are two entirely different beasts with their own set of priorities. Regular IT focuses on privacy and confidentiality – how do we protect our data and keep our employees working efficiently?

ICSs focus on process integrity and safety. As a result, security measures that are acceptable in IT systems may not be acceptable in an ICS.

Given the specialized knowledge required to run, maintain, and ultimately protect SCADA systems, choose someone who knows all the ins and outs of ICS networks. If no one internally has this kind of experience, get the help you need from an industrial cyber security expert that approaches the problem from a controls perspective first.

Josh Blume, Network Engineer, Vertech