Machine vision systems can recognize superficial aberrations in manufactured goods—like lines, spots, holes, color discrepancies, and more—that would typically require time-consuming manual inspection. However, these systems require specialized programming and maintenance, which can make them difficult to implement.

California-based Intrinsics Imaging solves this problem with its Heijunka Vision analytics-as-a-service software. Instead of programming specialized cameras, Heijunka provides a library of image processing and machine learning algorithms running in the cloud that allow any IP camera to perform intelligent defect detection. Intrinsics works with customers to tailor analytics to their specific objectives, then provides a concierge level of service so that customers no longer need to train, adapt, or maintain their vision system.

Typically, Heijunka integrates with SCADA systems to create analytics dashboards, alarms, and quality control actions. But when Intrinsics was approached by a customer hoping to integrate Heijunka directly into its process controls, it decided to adopt an edge-oriented approach using Opto 22’s groov RIO.

Securing a path to the edge

Intrinsics’ hybrid architecture used two groov RIO modules on each production line to allow cloud-hosted analytics to securely control action in the physical process at the edge of the network.

For applications like manufacturing asphalt roofing and other kinds of building products, Heijunka not only finds defects in coatings, but also detects problems in fiberglass underlayment, color consistency, product wrapping, and even pallet counts. It can also continuously inspect in-process materials to ensure that specifications, such as dimensions, smoothness, straightness, and color, are met.

For this application, Heijunka would be looking at two production lines moving discrete boards at high speed.

• The primary line cuts large sheets of raw material to size. Cut sheets would need to be inspected for excess moisture as well as dents, debris, and scratches as small as a grain of rice. The customer runs hundreds of different product types through this conveyor, each being cut to a different size and configuration.
• The second line would be responsible for monitoring the quality of the milling process, specifically looking for chipping along the edges.

Unlike most Heijunka applications, the customer also wanted a PASS/FAIL I/O signal that it could integrate directly into the PLCs that would handle material rejection. By bypassing the SCADA and providing a direct path to PLC action, the customer hoped to simplify integration and reduce latency. However, Intrinsics hadn’t integrated their cloud-based software directly with a hardware system before.

“This was surprisingly difficult to figure out,” says Eric Cheng, CTO at Intrinsics. “I wasn’t sure what kind of device could provide a physical interface to our cloud-based system and also satisfy our technical and operating requirements.”

Besides needing a device that could tolerate an industrial environment and integrate with Heijunka’s existing software stack, Eric needed something that would operate with minimal latency. From the time a given video capture was sent to Heijunka, the customer would have a roughly five-second window in which to detect and reject a problematic part. Heijunka would need to return a pass or fail indication that consistently fell within that window of opportunity.

Finding the missing piece

According to Eric, Heijunka’s communication infrastructure is usually built on MQTT. “Generally, we set up an MQTT broker and publish to an agreed topic,” Eric explains. “It’s usually new for customers, but it’s typically the best way to meet their performance and security requirements. [MQTT is] lightweight and low-latency, there’s plenty of open-source support for it, and it doesn’t require us to reach into their network from the outside. Amazon Web Services (AWS) has MQTT managed services to host a broker, and in cases where we are hosting the system ourselves, it’s all [deployable] out of the box and in the cloud.”

In this case, Heijunka was hosted on AWS and published MQTT messages to a hosted broker, which was then bridged to an on-premises broker in the customer’s facility. (Bridging allows brokers to exchange data behind the scenes so that it’s available to clients on both systems.) This architecture proved to be the key factor in choosing edge I/O for the final piece of Intrinsics’ solution.

“The customer… made me realize that an MQTT device could work for what we were doing. [Edge I/O provided] exactly what we needed, and it was eye-opening to see everything we could do with it.”

Edge I/O modules provide traditional, industrial I/O sensing for real-time applications along with multiple options for programming, data processing, networking, and communication, including MQTT. Eric also appreciated that Opto 22’s groov RIO provided a web-based interface, a low-code Node-RED runtime, and an embedded firewall so that he wouldn’t need to rely on a Windows PC for configuration, programming, and management.

Each of the defects in the plot above indicates a product that triggered a reject signal in Heijunka that then made its way to the groov RIO modules via MQTT (source: Intrinsics Imaging).

Putting the cloud in control

Intrinsics built an isolated network to connect cameras and groov RIO MM1 modules (GRV-R7-MM1001-10) to the on-premises MQTT broker. A separate network connects that broker to the internet for video streaming to Heijunka Vision and data exchange with the hosted MQTT broker, both running on AWS.

The groov RIO MM1 module provides 8 channels of universal I/O with support for more than a dozen software-selectable signal types. I/O data can be shared via MQTT, REST, VPN, or traditional protocols like Modbus/TCP and secured with user authentication, a local firewall, and TLS X.509 certificates. Each module can then be connected to an industrial ethernet network with support for Power over Ethernet.

In this case, each edge I/O module makes an encrypted connection to the local broker, which has only port 8883 open, the standard port for MQTT TLS connections. The decision to bridge between two MQTT brokers is also for security. The local broker acts as a firewall for the OT side of the system while still allowing I/O data to be exchanged with Heijunka in the cloud. “The goal is to keep the [modules] inaccessible from the outside,” says Eric.

To satisfy another customer request, each production line uses two modules configured to provide 8 discrete inputs each. Production line PLCs encode the product ID for the specific part being examined by Heijunka as a 16-bit integer and send each bit to one of the inputs on the module pair. Custom logic that Eric added to each module using the Node-RED language publishes its 8 input channels as MQTT topics, which Heijunka combines to decode the product ID and select the appropriate set of algorithms for that product type.

The modules also use Node-RED to subscribe to quality indicators, which Heijunka publishes to the MQTT broker. One of the relay outputs in each pair of modules is used to indicate the PASS/FAIL decision returned by Heijunka for a given part. The production PLCs watch these outputs and use them to trigger a physical rejection of the product if needed.

Since Heijunka performs all the heavy computation and product identification, the groov RIO modules can run the same logic without regard for the product type, creating a clean interface between cloud and edge networks.

Fast, automated quality control

At this point, the customer has been automatically rejecting defects for several months and plans to introduce Heijunka in the rest of its facilities. Intrinsics confirmed a round trip time, from measurement to result, of less than two seconds.

“I’m impressed with how fast it is even though we are taking two or three steps,” says Eric. “Most of that latency is due to transmitting video over the network.”

Intrinsics’ customer is using Heijunka to save on labor costs and increase quality with an overall goal of avoiding material returns. The customer can review system performance through Heijunka’s built-in trending, which displays metrics like the number of defects detected per hour. Over time, the customer can monitor historical trends in defect rates and diagnose the root cause of elevated defect levels. A user can click on any bar in the histogram and immediately bring up replay images to see every defect the system has ever detected. Each one of the defects in the plot above indicates a product that triggered a reject signal in Heijunka, which then made its way to the edge I/O modules via MQTT.

“We were under the gun to do this quickly,” says Eric, “but we got it figured out in less than a month… Now we can provide a direct physical interface to low-level automation systems… Using [edge I/O] allowed us to own more of the ‘last mile’ between cloud software and physical action and allowed the customer to speak the language they were most comfortable with. That allowed a cleaner separation between our software expertise and their hardware expertise.”

Heijunka can now be adapted to many more applications, supporting both hardware and software interfaces, whichever produces the best performance.

For more on Intrinsics Imaging, visit www.intrinsicsimaging.com.

Opto 22