The manufacturing industry has a large number of production lines and machines with old programmable logic controllers (PLCs) and custom controls that could benefit from an upgrade to newer technology. Many control and automation systems in current use were installed well over 30 years ago and are past their projected product life.
Automation has been key to industrial operation and efficiency for many years with most industrial automation and control systems functionally reliable for years. Many installed systems, however, are old and becoming obsolete, expensive to maintain, and in some cases, almost impossible to support. It is increasingly difficult to find people knowledgeable about old systems. Further compounding the problem is a lack of training for new people who try to maintain and troubleshoot these old systems.
Automation technology improves at a much higher rate than production machines, which have a much longer productive life. The advances and refinements of automation software and controllers over the past 10 years have been dramatically driven by developments in the broad computer industry and by vendor innovation. This provides an opportunity to improve production and efficiency with upgrades.
Several factors contribute to a tipping point where an upgrade is needed. Automation upgrades are complex business investment decisions that need to be carefully thought through. Reaching this tipping point does not happen on a specific date, but rather is a gradual process as an automation system ages, performance degrades, support cost and effort increase, and repair parts become harder to find and more expensive. It is better to plan upgrades in advanced rather than waiting for an upgrade to become a necessity.
As automation systems age, reliability typically decreases, affecting productivity. Reliability is defined as the probability that a component part, piece of equipment, or system will satisfactorily perform its intended function under given circumstances (such as environmental conditions, operating time, frequency, and preventative maintenance for a specified period of time). Reliability is dependent on several parameters, including components, environmental factors (e.g., temperature, humidity, and vibration), and electrical stressors, such as voltage and current fluctuations. Electronic systems, such as automation system devices, including controllers, computers, displays, and power supplies, are characterized as following the bathtub reliability curve, illustrating typical failure rates versus operating time (figure 1).
The initial steep slope from the start to where the curve begins to flatten is the early-failure period or infant mortality period. This period is characterized by a decreasing failure rate that occurs during the early life of a system. The weaker units fail, leaving a more robust population. The next period is termed the useful-life period, when failures occur randomly at a low rate. As systems age, the third period, the wear-out period, begins. The rate of failures increases rapidly as components begin to fatigue or wear out. Wear out in electronic assemblies is usually caused by the breakdown of electrical components that are subject to physical wear and electrical and thermal stress. It is in this area of the graph that the automation system manufacturer’s mean time between failures (MTBFs) no longer applies. When an automation and control system enters the wear-out range, the risk of failures increases dramatically. These are factors to consider:
Availability is the proportion of time a system is in a functioning condition. As reliability increases, so does availability. For example, a unit that is capable of being used 100 hours per week (168 hours) would have an availability of 100/168. However, typical availability values are specified in decimal (e.g., 0.9998). In high-availability applications, a metric known as nines, corresponding to the number of nines following the decimal point, is used. In this system, “five nines” equals 0.99999 (or 99.999 percent) availability.
The mean time to repair (MTTR) older systems increases, and repair can be difficult and time consuming. Downtime due to older system failures can have a major impact on production. Obsolete industrial control and automation components that fail are a challenge to replace, because they can be difficult to find. When they are found, they usually have a large price tag. Repair challenges include lack of hardware and software documentation, repair parts availability, lost manuals, lost and out-of-date drawings, and lost and out-of-date wiring diagrams. The most difficult problem is finding people who understand the system and can efficiently diagnose and solve problems. Typically people with the knowledge, know-how, and experience are in short supply. Suppliers, service companies, and system integrators also have a shortage of these people and are unlikely to train new people on old systems.
Total cost of ownership
Total cost of ownership (TCO) includes the initial costs to implement automation together with the continuing costs to maintain, modify, train staff, deploy, provide infrastructure, and any other cost associated with the project, including final decommissioning. Many automation systems are running past their initial projected life, and the real TCO increases beyond the original plan. Reliability issues also affect the TCO, because downtime lowers productivity and efficiency. All the factors, including spare parts and retaining people experienced in older systems, should be considered in calculating the TCO.
Many upgrades are done based on reliability considerations, but upgrading to increase productivity is a forward investment that can have greater business value. The investment in upgrading automation and controls can improve the productivity of the machine and, in many cases, overall production flow and efficiency.
These are perspectives on upgrades based on the experience of industry professionals.
I asked David Jenson, director of engineering at Gross Automation, about upgrades. He noted these clues that indicate an upgrade should be considered: out-of-date or obsolete controllers, human-machine interfaces (HMIs) running on unsupported versions of operating systems (e.g., Windows XP), obsolete hardware displays, and complaints that the system cannot be updated or is incapable of supporting newer technologies that would improve operations. He emphasized the way to approach an upgrade project is spending time with users to write a complete functional specification of the process or sequence. If it is a simple system, maybe just an outline will do. Tips for people approaching a machine upgrade include discussing the process or sequence of operation with more than the people in charge of the project, and talking with operators and maintenance people to gain valuable insight into the process or sequence.
Tye Long, plant engineer, Centria Coating Services, commented on a recent project. He started with justification of the upgrade based on many parts of the system being outdated, with unavailable replacement parts. The Centria Coating Services facility in Cambridge, Ohio, produces approximately 600,000 pounds (about 210,000 lineal feet) of coated steel and aluminum per day, with about 4 million pounds of metal running through the facility’s coating operations every week. Plant engineers also saw opportunities for higher operating efficiencies and reducing breakdown time by up to 50 percent. Line speed was increased by using a newer automation system and replacing decades-old motor generators and DC motors with modern variable speed drives. The paint line’s line speed was increased by almost 100 linear feet per minute, a 25 percent improvement.
Centria also added 24/7 remote monitoring application support from Rockwell Automation for 2,500 data points from the Cambridge facility, including all controllers and drives, select HMI activity, and some regulatory compliance parameters. If issues arise, the remote application support team can either address the issue remotely or immediately notify an on-site plant foreman or maintenance technician.
Upgrading 80 machines
I had a discussion with Mark Lewis, manager, technical services, at Beckhoff, who just worked with a client on the upgrade of 80 machines. A major factor was obsolescence of the existing PLC controls, because repair parts were becoming difficult to find. This created a looming problem but also an opportunity to improve operations. Lewis emphasized, “Don’t start until you clearly have in your mind what it will be.” Lewis described the process as defining expectations, knowing the goals for increased production and decreased costs, and understanding the whole picture before starting the project. For a successful project, he believes all the people involved with the operation should give input to the project to get a clear vision of the desired outcomes. In this project, for example, the facilities engineer was brought in as part of the retrofit planning team that led to finding energy savings opportunities.
In this application, new automation controls had more capability, more accurate control, and improved performance, resulting in a more than 15 percent increase in production. Lewis also noted the upgraded system enabled more production flexibility. The improved communication with operators and business systems gave immediate visibility into the production process that did not exist previously. Retrofitting the controls reduced energy costs, including compressed air, heat, and process steam.
The old program was 100 percent ladder logic, which was hard to understand. Programming the new automation and controls was efficiently done using existing sequence-of-operation descriptions and encapsulating functions in easy-to-understand IEC 61131-3 standard function blocks for their applications. Some basic sequencing was done with sequential function chart programming. The structured logic design and new software has modern troubleshooting tools, including watch lists, trace, and strip chart recorders, simplifying troubleshooting.
The controller platform in the application is an industrial PC (IPC) running a real-time operating system, PC-based automation software programmed using multiple IEC 61131-3 languages, and an OPC server. The upgrade took advantage of OPC Data Access to connect with other controls via an open, “vendor neutral” interface. The newer standard, OPC Unified Architecture, was used to interact with the production planning and production control system.
Mark Lewis emphasized, “If you understand the process, then you will understand the program. If you can’t understand the process, you will never understand the program; you will see bits, bytes, and contacts, but you will not understand. Once you understand the process, then you can write the program.”
Developing a formal migration strategy and plan for existing automation and control systems avoids surprises, rather than waiting until systems cannot be practically supported anymore. Planning ahead also provides time to identify ways to improve production efficiency and to enhance reliability and safety. Using past knowledge about the machine or process can help to lower the MTTR of the new system, reducing downtime.
Last word . . . compete or die
Simply upgrading automation and controls with a similarly functioning replacement without new features can be shortsighted. Manufacturers are in a competitive world. Implementing more automation and control functions to increase productivity, flexibility, and quality will help companies remain profitable.