During the year 2020, dozens of companies worldwide worked to develop vaccines for the pandemic-causing virus SARS-CoV-2, known to the world as COVID-19. Four goals had to be met: vaccine efficiency, clinical trial completion, worldwide regulatory approval, and vaccine production.

Scaling up production to supply billions of doses per year would be a daunting task, and waiting for clinical trials to be completed first would take time. The one thing that the world did not have in 2020 was time.

So vaccine production needed to be started in parallel with clinical tests, with the risk that if the product was not authorized, all production would have to be dumped as unsellable.

The Galician company Biofabri, a subsidiary of Zendal, the Spanish international biopharmaceutical group, became responsible for industrial production of the antigen vaccine for most of Europe. Their task was daunting.

Zendal includes seven facilities focused on Human and Animal Health, including research, development, manufacturing, and marketing of vaccines and other biologics, pharmaceuticals, and probiotics. Based in O Porriño, in northwest Spain, the company employs over 650 professionals at five production sites across Spain and Portugal.

With more than 30 years of experience in the biopharmaceutical sector, Zendal is known for its flexibility in adapting existing installations for new animal and human vaccine products. But this flexibility would be tested beyond limits to produce the new Covid vaccine.

Fill and Finish Tanks during commissioning

Zendal’s challenge

Constructing and automating a new fill-and-finish facility would somehow have to be accomplished during a worldwide pandemic, in a country with one of the strictest lockdowns. From March 2020, apart from designated primary services, Spain’s population was not allowed out of their homes except for medical reasons or to buy food. For the following nine months, travel outside of their local state area without authorization was also prohibited.

Logistics were soon in chaos. Orders for anything associated with sanitary or pharmaceutical use became impossible to fill as both stock and production dried up. Delivery dates became meaningless. Sanitary certified stainless steel such as tanks, cabinets, and tubing were unavailable. Industrial supplies such as machinery, instrumentation, and valves were in short supply, and as global supply chains broke, control equipment such as PLCs, PCs, and even basic items like instrumentation wiring, cable ducting, and terminals became impossible to source.

A technician uses the groov EPIC processor’s integrated high-resolution color touchscreen.

System requirements

For production control, many existing plant areas used single-loop controllers and several generations of PLCs from various suppliers. In addition, Zendal had no control over the make or model of PLCs that package suppliers would provide with new equipment. There was no time to substitute these or migrate to a centralized control system.

So they decided to add a new control system that could communicate with standalone equipment using nonstandard protocols, sending setpoints and control commands, reading their process inputs and outputs, and storing data in a SCADA system.

The new SCADA platform was essential to meet strict regulatory requirements. For example, the U.S. Food and Drug Administration (FDA) regulation on electronic records required that even the name of anyone who modifies a process setpoint must be traceable and explainable to FDA inspectors.
Zendal also required a flexible data historian, integrating information not only from the SCADA and control systems, but also from many other data sources and databases.

Assuming the vaccine would pass clinical trials, centralized recording in the data historian would be vital in allowing the sale of the vaccines in production.

In addition, the new system would need to provide a common user interface to replace the incompatible individual HMIs throughout the plant and also be accessible by any authorized user on the IT network, or even off site though secure VPN connections. And perhaps most important was cybersecurity.

Air Handling Units (AHU) guarantee sterile conditions inside production areas.

Priority One: protect production control systems against cyberattack

Pharmaceutical companies involved with Covid-19 vaccine production became a target for cyberattacks. But remote connectivity to the system was required for staff working from home, for European suppliers who could not send specialists to Spain, for control system and SCADA access, and for software development by two separate teams, 575 kilometers (more than 350 miles) away.

While the advantages of connecting production networks with IT networks are indisputable, this integration also opens the possibility of organized cyberattacks, resulting in loss of production for hours, days, or weeks or even closing a company down.

At Zendal, production engineering, maintenance, and IT departments worked together to construct a more secure platform for production control based on five pillars:

• Eliminate any dependency on Microsoft Windows in the production environment. Windows attracts more interest from hackers than any other operating system and behaves badly after power outages.
• Use thin-client computers in the production process, connected to server-based software and replaceable in less than 15 minutes. Install no additional software and do not use them for historical data storage, which would be lost in a hardware failure or an attack on the file storage system.
• Physically separate production networks (OT) from the corporate computer network (IT), using managed switches and gateways to create both vertical and horizontal network segmentation.
• Build a control system architecture that can continue without the network if a cyberattack occurs at the corporate level.
• Use only process controllers with an internal firewall to avoid cyberattacks if either IT or OT networks are compromised.

Ground Level of Fill and Finish Storage Tanks during commissioning.

Team and approach

With all these challenges—pandemic limitations, system integration with old and new equipment (especially specialized equipment lacking current security and communication technology), SCADA and HMI requirements, more than 20 suppliers, and cybersecurity demands—Zendal knew they needed to work with a versatile, quick, and flexible technology integrator.

They found the perfect partner in Optomation Systems, based in Madrid. With extensive control experience for other pharmaceutical companies, they are well known for their flexibility. Optomation took a close look at the requirements and recommended the Opto 22 groov EPIC system and Inductive Automation’s Ignition SCADA. Opto 22’s official representative for Spain and Portugal since 1997, Optomation is also a certified Gold Level Integrator for Inductive Automation.

To meet committed start-up dates, Zendal formed a single integrated team that would meet only virtually. The team consisted of key staff from their own Engineering, Maintenance, and IT departments, and key staff from Optomation.

System hardware was installed, wired, and checked in suppliers’ workshops before being transported to the site where it was installed and connected to the company’s network, allowing specialist engineers to connect all the way from Madrid.

As Covid-19 continued to spread, key workers became infected and unavailable to work. It was the perfect storm of challenges for supplying a high-profile industrial automation project. For those involved, it would be the most difficult automation project of their careers.

One of the many groov EPIC distributed control system cabinets.

The choice: groov EPIC and Ignition

Opto 22 hardware and software are developed and manufactured by Opto 22 in Temecula, California, in the U.S. The company’s flagship product groov EPIC combines an advanced hardware platform, lifetime warranty on most I/O, security features, and embedded application software to handle any modern industrial automation or IIoT project.

The groov EPIC hardware price includes access to software with no hidden surcharges like number of tags, size of database, number of devices used, or annual license fees.

Opto 22 provides all software updates as new firmware without charge. Two main alternatives are included for programming real-time control applications: its own PAC Control and CODESYS, the IEC 61131-3 programming tool for more PLC-oriented applications.

Another reason for choosing Opto 22: the company was one of the few control suppliers still able to ship product during the pandemic. In its nearly 50 years of history, Opto 22 has never subcontracted production overseas to reduce costs. They continue to design, manufacture, and distribute from their factory headquarters in the U.S.

Due to the pandemic, Zendal was classified as a priority one customer, and batch shipments of required hardware were airfreighted to Spain in less than two weeks from order entry.

For Zendal, groov EPIC processors were supplied with 16-module chassis racks, allowing expansion up to 384 hardwired I/O, unlimited remote I/O, and data from externally connected PLCs or other intelligent devices.

All processors can share data using peer-to-peer communication over the Ethernet network.

Ignition software is developed and marketed by Inductive Automation in Folsom, California, in the U.S. It is regarded as the most complete and advanced distributed SCADA software in the market, used by over half of America’s Fortune 100 companies, in distributed manufacturing, industrial automation, process control, utilities, and transport infrastructures.

Unlike competitors’ products, Ignition can be scaled up without additional charges. The software runs on various operating systems and hardware platforms and includes drivers to connect virtually any industrial hardware or software. The company does not charge for connecting additional devices or tags to the servers.

groov EPIC also includes Ignition EDGE Panel software.

Similarly, Ignition’s Visualization HMI module has no charge for added users. Zendal currently has between 30 and 60 clients connected simultaneously to the system, which in time will probably increase to over 150 simultaneous users.

Opto 22 provides both Ignition EDGE and Ignition Standard software products embedded in groov EPIC controller firmware for optional activation and licensing. Network problems are avoided because the SCADA data collector is fully integrated in the same controller hardware, simplifying communication and creating a secure connection between them in a single hybrid system. For the pharmaceutical industry, the importance of this combined solution cannot be overestimated.

When Optomation and Zendal chose Opto 22 and Inductive Automation for the system, they knew that the hybrid solution and guarantees offered by groov EPIC and Ignition ticked all the boxes they needed:

• Security
• Network efficiency
• Ease of upgrading
• Rapid system growth
• Remote development and testing

Security requirements met

The groov EPIC process controllers use a crypto-signed industrial runtime version of Linux, supplied and
maintained by Opto 22. A crypto-signed operating system cannot be modified or substituted by a third party, eliminating the possibility of obtaining system access through contaminated updates or patches.

Traditionally, integrating any process control system with a SCADA product requires protecting not only both systems, but also the network connections between them. But because groov EPIC runs control software and SCADA software in the same hardware platform, communication between them is internal and protection is built in. Both Opto 22 and Inductive Automation promptly address vulnerabilities, and Opto 22 provides a free groov EPIC firmware revision to resolve them.

Communication between any client and the groov EPIC server requires only a web browser on the client and is encrypted using a secure channel to protect sensitive transaction data. groov EPIC’s configurable internal firewall defines which applications can be externally accessed and which network hardware interface each application uses.

groov EPIC also integrates the Water for Injectables (WFI) plant using ASi bus.

Network efficiency: the power of edge

In typical SCADA architectures, the system constantly scans PLCs and control equipment for data, creating unnecessary traffic. Any network problem results in data loss.

For Zendal, the Ignition EDGE software in the groov EPIC processor remotely collects real-time industrial data from the controller, data marked for historical storage, operator actions, and process alarm data. The software then uses secure protocols and reporting by exception to send the data to real-time relational databases. Without scanning and unnecessarily repetitive data, network traffic is reduced to a minimum.

If a network problem occurs, all data is automatically buffered in the Ignition EDGE gateway for up to seven days or 1,000,000 data samples. Once normal network communications are restored, this buffered data is transferred and merged into the centralized Ignition database.

A corporate-level cyberattack could compromise communication with the centralized SCADA system. Zendal provides plant operators a local user interface through the optional EDGE Panel in production-area groov EPICs, so that a client computer can still connect locally for control and regulatory data.

“In the pharmaceutical industry, losing historical data is not an option, as without access to batch manufacturing data, the product is unsellable,” notes George Mitchell of Optomation. “The full value of Ignition EDGE in groov EPIC only becomes apparent when you have network problems.”

Scaling up: planning for system growth of 2000% in three years

The initial scope at Zendal was to monitor and register working temperatures and alarms for 15 deep-storage freezers, where the early COVID-19 virus samples were stored at -80 °C. (-112 °F).

But Zendal knew they were not looking simply for a precision controller for batch recording and data logging. Instead, they needed a flexible control system that would cover all the company’s future requirements, not only for this installation, but for all the other production facilities on the same site, other production centers across Spain, and a new production center being built in Portugal.

“We knew this baby would grow, but we honestly had no idea how big it would get,” says David Vila, responsible for Installations in Zendal.
“We just knew we needed something that would have the capacity to scale up at the same speed as our business and would not need to be replaced with something bigger or better in a few years. We do not have the time or resources to start again from zero.”

Ignition Edge Panel software runs directly on the groov EPIC hardware.

So Zendal initially chose a single groov EPIC controller and chassis, with two 8-channel GRV-AIRTD analog input modules and an onboard Ignition Standard license.

The processor and its I/O met harsh industrial conditions and modern cybersecurity standards; the Ignition software complied with GMP and FDA guidelines for use in pharmaceutical installations, especially with respect to electronic records and electronic signatures.

A NAS drive was added to the controller to support Ignition’s Tag Historian module, and the VISION module with a single user license allowed the temperatures to be monitored from anywhere on the IT network.

Three years later, 19 groov EPIC controllers are on site, each covering a distributed production area or process. Zendal continues to use the same Ignition Standard Gateway license purchased with the first controller. At the time of writing, the gateway receives and manages data from 28,6000 OPC UA tags and a total of 186,900 atomic or derived tags, used to display, control, and store data for regulatory requirements from the entire company’s production systems, all without any additional surcharges for system expansion.

In 2022, Zendal inaugurated a new production facility in Portugal. They installed a new redundant Ignition Standard Server locally to handle the complete SCADA requirements of the production facility, communicate to multiple control packages (some based on Siemens and Allen Bradley PLCs in addition to groov EPIC controllers), and provide a user interface and a tag historian package for the local plant.

The sites in Portugal and Spain are interconnected through secure VPN connections, and authorized users connected to the Ignition servers in Spain can switch to the Ignition servers in Portugal for a real-time connection to processes running in another country. The Ignition distributed architecture model provides this functionality without any extra programming or costs.

Each new groov EPIC controller added to a production facility adds another distributed co-processor to Ignition’s total computing power. Ignition EDGE gateways running on the groov EPIC controllers scan and update all data in real time, performing local processing at the edge of the network before transferring post-processed data to the centralized Standard Servers.

groov EPIC also integrates the Water for Injectables (WFI) plant using ASi bus.

Remote development: accelerating software design

The entire software team developed the project from Madrid, connecting to the servers on site through customer-provided VPN connections. Due mainly to Covid restrictions, the team did not visit the site for commissioning and startup. File locking and Ignition’s web-based designer tool allow multiple users to simultaneously work on the same project without the risk of overwriting other team members’ work. The VPN client on groov EPIC offers remote connections for configuration, loading and updating software, and testing.

As Lead Engineer Fabio Alberini explains, “Both groov EPIC and Ignition software are designed from the ground up for remote connectivity, so once the equipment is connected to the customer’s IT network and switched on, you really do not have to be there locally. This saved us at least a day’s travel every week and onsite expenses. Whenever possible, I prefer to be at home with my family.”

As process design changed during the project, production and engineering staff needed to provide feedback to the programming team. For this reason, the team decided to use PAC Control, Opto 22’s own control programming environment. Based on visual flow diagrams, the program could be easily understood by everyone, software expert or not. The same flowcharts were later used to train new production staff, explaining every operational step in the process. The flow diagram format also served as documentation for developing design qualification (DQ), installation qualification (IQ), and operational qualification (OQ) documentation, avoiding delays in control system validation.

For valve, motor, PID controller, and other objects basic to most industrial automation projects, the team developed library subroutines. Once tested and approved, these software building blocks could be signed off and reused with the confidence that all other instances would perform identically, saving time during the software testing phase.

One such building block was an F0 algorithm, associated with microbiology analysis, to provide an FDA-approved estimate of adequate production autoclave sterilization based on steam temperature of 121 °C (250 °F) and a reduction of microbial population by 90%. This algorithm, which calculates Minutes of Accumulated Lethality, avoids unnecessary time delays in sterilizing operations during sequential batch production.

A powerful feature of Ignition software that helps replicate equipment and problem-free scale-up for similar installations is the use of UDTs (User Defined Types). UDTs allow the development of object-oriented design models in Ignition, dramatically reducing the amount of work necessary to create robust software by creating parameterized “data templates.”

With features like these, time spent designing, implementing, and testing software was notably reduced, meeting the tight time restrictions of the project. For each phase of the project, the hardware and software were ready before the mechanical and piping installation were finished.

Looking to the future

With the worst of the Covid 19 pandemic in the rearview mirror, Zendal is now immersed in other ground-breaking projects, in which groov EPIC and Ignition will be tested to new limits. The most exciting of these is the development of MTBVAC, a new and more efficient version of a tuberculosis vaccine.

Zendal also plans on adding additional standard Ignition modules, specifically for MES/ERP. Although different aspects of a manufacturing process, they are based on the common database created by Ignition’s SCADA model.

These software modules will offer detailed visibility into production through seamless data exchange, allowing for faster and more efficient execution, and will help optimize resources and plan for future demand.

Application case history article by Opto 22