Engenuity, Inc. provides control automation and data integration solutions across several industries, with a primary focus on advanced technologies for oil & gas customers. Through their work in this industry, they identified specific deficiencies in validation pressure testing of blowout preventers (BOPs) and well control equipment.

Pressure tests must be completed every few weeks in order to maintain the safety and integrity of drilling facilities. They are typically executed and recorded manually, taking hours to complete. Since it can cost as much as $6.00 per second to operate the associated valve arrays in an offshore drilling application, testing can cost millions of dollars annually.

In collaboration with customers like Shell International Exploration and Production Company, Engenuity developed a set of fully automated hydrostatic test execution and reporting products. Their high-reliability solution uses Opto 22’s groov EPIC edge programmable industrial controller for precise process control, automatic text and email notification, and process history storage and replication.

Example BOPX system architecture

Grace under pressure

Using Engenuity’s BOPX testing software, users can map out valve arrays and identify validation pressure limits for each component in the system.

Using a patented constant-pressure/variable-volume pumping method, BOPX brings the system to pressure while precisely measuring the injected fluid and objectively comparing measurements against validation criteria.

EZ Valve retrofittable actuator.

Engenuity’s EZ Valve retrofittable actuator works with the BOPX test manager to add automated monitoring and control to manual valve arrays. The system can actuate up to 70 valves through as many as 30 test sequences.

Finally, Engenuity’s EZ Vision readout uses an innovative acoustical leak detection system to pinpoint the location of leaks in large-manifold systems.

Collected data can be mapped between tests for comparison and reliability purposes and used to generate PDF reports. Depending on an installation’s size and operations, Engenuity’s BOPX pressure testing system can save 10-20 hours for each full test with a fully integrated system.

Engenuity’s BOPX testing software lets users map valve arrays and set limits.

How it works

BOPX offers sophisticated modeling, analytics, and presentation functions. But while Engenuity has added many features over the years, Ted Royer, a controls engineer with Engenuity, stresses that “all the test software is on EPIC. [BOPX] is just an interface now.”

Let’s see how Engenuity uses groov EPIC to bring this awesome system together.

Secure configuration

BOPX’s primary role in test execution is to send configuration data to the groov EPIC edge controller over a Modbus/TCP connection.

“We could do other things, but we needed a way to show where the data was,” explained Royer. “Customers want it their way, so you need to be able to say, OK, we can do that.”

And typically, Engenuity’s customers want to work with traditional protocols like Modbus so they can easily verify that configuration parameters are transmitted correctly.

However, an unsecured Modbus connection could pose a risk to operations, so Engenuity uses groov EPIC’s configurable firewall to protect critical control data. One of the edge controller’s two network ports is configured as a read-only customer interface joined to an untrusted network called the Manufacturing Demilitarized Zone (DMZ). The other network port is joined to a separate trusted network, named the Manufacturing Zone, where all I/O and control communication takes place.

The EPIC’s design prevents routing traffic between its two ports, so there is no risk of untrusted connections tampering with configuration, execution, or process history data on the trusted network.

Pressure tests performed every few weeks maintain the safety and integrity of drilling facilities.

Process control and alarm notification

With configuration data securely transmitted, the edge controller begins executing test scenarios. Generally, automated valve manifolds and EZ Valve and EZ Vision connections are integrated directly into the EPIC’s local I/O modules, but if a large amount of high-speed processing is required (typically in the 3-4 ms range) Engenuity sometimes uses a separate groov EPIC as remote I/O. In that case, the remote EPIC handles all of the control independently and simply reports back to the primary edge controller.

As test execution proceeds, the primary controller monitors the process.

“We have a really well-developed alarm system we‘ve developed over decades,” Royer added. “It runs in the controller, not the HMI, so it’s available to other systems,” including external mail and messaging services, which the control program (written in PAC Control) contacts to generate email and text notifications for a range of conditions.

Process history storage

The primary controller also stores any process data generated during testing to an internal MariaDB server. A popular and proven open-source database management system, MariaDB is available in a cryptographically signed package from Opto 22’s Linux repository. Using the free secure shell (SSH) license for groov EPIC, Engenuity can download and install this and many other applications for execution directly on their edge controller.

To transfer data from the control engine to the database, Engenuity uses Node-RED, an open-source IoT platform originally designed by IBM, which is pre-installed and integrated into groov EPIC’s management layer. Node-RED provides thousands of functions for connecting and processing data from different sources, including devices, databases, and web services.

Engenuity configures Node-RED to monitor I/O, process variables, and events, then transmit these data points to the internal MariaDB server.

Data integrity and availability

Test report (PDF).

Because testing data is used to validate the safety and efficacy of the system, customers need frequent access to process history and often require additional protections to ensure data integrity. Engenuity accomplishes this using MariaDB’s built-in replication function.

“Typically, on an oil rig, customers just have a WiFi connection to the equipment, which isn’t super reliable,” Royer explained, “and they want to be able to log in and see data from previous runs. Sometimes they want to pull massive amounts of records, even entire jobs, into an Excel spreadsheet. It doesn’t have to be real-time, though, so we will set up another controller here in the office that serves as a replica for the one that’s in the field.”

Periodically, the MariaDB server housed in the on-shore replica requests database replication from the off-shore primary controller. In response, the primary MariaDB server sends a transaction log to the replica, which it uses to update itself.

Instead of burdening the primary EPIC with large data requests, customers can connect to the replica, providing them with data redundancy and high availability. “We have plenty of bandwidth here [in the office],” added Royer, “so it gives them the best of both worlds.”
Of course, transmitting sensitive data between networks over long distances creates security concerns, and Engenuity’s network architecture again plays an important role.
.
To facilitate secure data exchange with the primary edge controller, Engenuity joins the replica controller to the off-shore Manufacturing Zone network using a point-to-point VPN connection.

Engenuity then grants customers read-only credentials to this private network. In fact, every replica controller in Engenuity’s offices connects to the outside through a separate, dedicated VPN appliance. “It’s simple and as bulletproof as we could make it,” said Royer.

Enterprise connectivity

In the case that a customer wants to distribute process data beyond a single site, Engenuity takes advantage of groov EPIC’s integration with Ignition Edge® from Inductive Automation®. Like Node-RED, this application comes pre-installed on all EPICs. It is used most often to provide OPC UA integration of third-party devices and PLCs.

However, the Ignition platform has a modular architecture that allows users to install a range of additional components, including connectivity solutions from third-party provider Cirrus Link® Solutions. Engenuity uses Cirrus Link’s cloud injector modules, which allow groov EPIC to send OPC tag data directly to cloud services like AWS, Azure, Google Cloud, or IBM Cloud. Once there, customers can access testing data around the world.

Why EPIC?

Engenuity’s BOPX system uses intelligent, integrated automation to reduce the costs associated with validated pressure testing. Testing is required for regulatory compliance, and the cost of downtime may be $15-20k per hour, so BOPX must execute quickly and consistently in order to deliver value to the customer.

Engenuity has standardized on groov EPIC over traditional PLC control, ultimately, because it helps them provide more reliable operation and efficient support.

“EPIC I/O is high quality, and you need to have very stable I/O for these applications,” noted Royer. “[But] there are so many advantages to having a full control system that can tell you what’s wrong…. I’ll get a text [from the control system] in the middle of the night, and I’m already logged in by the time they call me. I can tell them exactly where the problem is, and, usually, it’s something simple.”

Naturally, Engenuity’s customers reap the benefits as well. With high uptime and automated testing, they are able to reduce on-site staff and have adapted more easily to remote operations.

However, Royer is also quick to point out the value that off-shore sites place on equipment that can be installed in the field.

“[Customers] don’t have space for racks of servers. These [IT rooms] are non-classified areas in the middle of hazardous areas. They need all this air conditioning and so on to keep this IT stuff going. They are designed to be half full and [typically] they are overfull. So now you are going to replace that with stuff that can go outside in a NEMA 4 box. [Our groov EPIC’s] got nine servers on it and we haven’t even taken up one foot of space! If you want another one, it’s only another $1,500.”

“You cannot do this with a traditional controller,” Royer argued. “That’s been the big leap with Opto 22: the hybrid controller allows you to do it all.”

With groov EPIC, Engenuity eliminates complex multiproduct integration requirements and delivers seamless operation and instant data access either in SCADA, IoT, or hosted data applications.

Jeff Hilpert, president at Engenuity, added, “The integration of data and control on a single backplane has been key to dramatically lowering cost, improving uptime, accessing data, and expanding utilization.”

Josh Eastburn, Director of Technical Marketing, Opto 22