In response to the increasing demand for hand sanitizer in 2020, Emerald 66 Enterprises (E66) set up shop in an empty denim processing plant in Seminole, Oklahoma, U.S.A.
In only three months, with the help of system integrator Northeast Automation Company, Inc. (NACI), E66 had automated packaging lines producing up to 1 million bottles of hand sanitizer a week in a cGMP-compliant facility, and it continues to expand its core capabilities at a rapid clip.

Let’s examine the technologies and techniques used to achieve competitive advantage in a challenging market environment.

Getting down to business

When E66 hired NACI to automate its bottling and packaging process, the company understood it was competing against low-paid, high-volume workforces operating manually and believed it could use technology to do more with a smaller, better-paid workforce.

“Each piece of equipment needs to be intelligent … because management is so keen on information,” explained Thomas Coombs, principal engineer at NACI. “We’re going to make every conveyor and every device smart.”

To achieve this, Coombs planned to employ edge computing—a design technique that adds general-purpose data processing and connectivity capabilities to traditional real-time control and sensing applications—to build an information management system at the same time that he scaled up production capabilities. He did this using Opto 22’s groov family of industrial edge controllers and I/O.

But E66 had also determined that the quickest way to build a new packaging process was by acquiring a variety of equipment at auction. The state of equipment on arrival varied widely, and NACI had to get creative in order to design a cohesive system at the speed that Emerald 66 needed.

Layered distributed control

To address the different circumstances the team faced, NACI employed a unique architecture that enabled separate control systems to function together and also laid a foundation for E66’s data acquisition goals.

At the top level, NACI used a groov EPIC edge programmable industrial controller, which combines PLC control with embedded HMI, OPC UA, and secure gateway functions in a single backplane. The EPIC supervised the process lines and connected disparate devices through REST APIs. Any equipment that arrived with a defunct control system was integrated directly into this network as simple remote I/O.

Functional controllers, on the other hand, were left in place and loosely coupled to the main process using groov RIO edge I/O modules. These modules provide software-configurable, multi-signal I/O channels and are powered over Ethernet (PoE), making them quick to deploy. NACI placed a module in each piece of equipment upon arrival, connected any I/O wires, and identified the types of signals the equipment provided. These I/O signals were then integrated into the groov EPIC network in parallel with the existing PLC I/O connections, which continued to function independently.

Coombs noted, “The ease with which you can do this, you know, you’re talking about a half-hour of wiring. Your biggest problem is finding the documentation from the original manufacturer.”

NACI also engineered an additional layer of control independent of the groov EPIC by building limited local control into each groov RIO module through Node-RED, an embedded, open-source IoT platform from IBM. Living up to the ambition to make every device smart, NACI added motors, photo eyes, load cells, and other instrumentation to many pieces of semi-automated and dumb equipment, connected these to local groov RIO modules, and added Node-RED logic to make them work together and report process data up to the supervisory level.

E66 COO Robert Bodnar explained, “The top-level process [in the EPIC] is turning on two lines or three lines. If you have a line coming in and you have a line going out, they may not be running at the same speed. … so it’s kinda neat to be able to say, okay, what if we use the groov RIOs to control just the lines and the belt and case packers and things like that? … They’re kind of little islands of point automation.”

This loosely coupled, distributed architecture allowed NACI to assemble their production line without modifying any of the existing control systems that came as part of their purchased equipment. This strategy ultimately saved them development time, and in three months, Coombs and his team had 15 pieces of equipment up and running.
But this wouldn’t be their last challenge.

The big pivot

Because it had built its business around packaging and distribution for a single large purchaser, Emerald 66’s process was originally designed to maximize throughput. But the situation changed significantly when that customer suffered a financial setback and had to close production. Then, the whole business had to pivot to allow E66 to become a multi-product facility.

Automation grew from processing a high volume of single-formulation, one-gallon containers to working with a variety of sanitizer chemistries in different batch sizes and packaging form factors: from small two-, four-, six-, and eight-ounce containers, hand pumps, and spray bottles, to large jugs in excess of one gallon.

Fortunately, NACI’s decision to use a loosely coupled production line made it easy to modify individual segments without interrupting their data collection and process integration. In combination with on-site panel building and 3D printing capabilities, their investment in edge-oriented automation allowed them to retool very quickly and break even on their initial investment within six months.

Data-ready automation

In addition to using Node-RED to cement process control, E66 has started moving data from each groov device into relational and time-series databases, financial software, and other connected systems. And since Node-RED is a free, open-source application, E66 is adding it to their back-end systems as well so they can push data down into the control system.

“You might want to trip a lot number forward on a device or increment something based on a date that’s somewhat arbitrary, based on a business event, not necessarily on a machine event,” Bodnar explained. “[Node-RED] gives you a very lightweight way to run an operational bus or hub.”

For example, Emerald 66 purchased a standalone pallet wrapping machine that used a proprietary circuit board design and offered only a limited operator interface. However, by adding load cells, connecting them to a groov RIO module, and feeding that data to Node-RED, E66 could sanity check the weight based on the known pallet contents and communicate the pallet number, lot number, shipping weight, and date to its central database to create bills of lading automatically.

More to come

As demand grows, Emerald 66 continues to expand its automation and diversify its business. It is adapting to accommodate new functions like on-site container molding, bulk product blending, and additional quality control procedures.

Bodnar says he is defying the industry norm of investing in a multi-million dollar build for an automated facility and instead “putting in just enough automation to double my business, literally, which is going to make the company’s year. And then I can grow into that. And I’m not going to have to go back and rip and replace everything …. We’ll tweak things, but I’m not going to have to throw away anything that I do because I can scale it.”

“The fluidity and dynamics of modern manufacturing requires extremely fast response to changing market demands,” added NACI’s Coombs. “With groov EPIC and groov RIO, Opto 22 puts dynamic manufacturing data at the edge of the production line and into enterprise systems simultaneously in real-time.”

Josh Eastburn, Director of Technical Marketing, Opto 22