Best ISA Webinars of 2017

Best ISA Webinars of 2017

This 2017 webinar roundup was edited by Joel Don, ISA’s community manager.

 

As 2017 comes to a close, we surveyed the year’s lineup of educational webinars to select five of the most popular presentations co-hosted with ISA’s partners. Scroll down and enjoy this roundup of “best of the best” webinars.

Large Project Execution – A Better Way

There are many challenges to effectively executing automation scope as overall project size grows to $100 million and beyond. Interfaces between various stakeholders and proper distribution of work become critical to define and manage properly. The traditional model of sole-sourcing an EPC to handle everything, including the automation scope, has inherent weaknesses that can be mitigated by an alternate approach. Join us as we review common problems with executing automation scope in large projects and present solutions proven to be effective.

Cybersecurity for Control Systems in Process Automation

Attacks to your production system may happen at any time and at any level – from outside as well as from inside. Which concepts and measures exist to protect your assets efficiently from software attacks? The ISA99 standards development committee brings together industrial cybersecurity experts from across the globe to develop the ISA-62443 (IEC 62443) standards on industrial automation and control systems security. German-based Siemens AG is a leading provider of automation equipment, global manufacturing company with close to 300 factories and provider of Industrial Security Services. In this webinar, ISA99 Committee Co-Chair Eric Cosman and Siemens Plant Security Services PSSO Robert Thompson will present the current threat landscape and key steps you can take to protect your critical assets in the production environment.

How to Avoid the Most Common Mistakes in Field Calibration

Field process calibration isn’t just about getting the job done, it’s about getting the job done right. It’s cliché, but true. Instrument calibration requires the proper process, tools and parameters for each instrument application to ensure valid results. If one of the three are lacking, your results could have little validity. In this webinar, experts will expose the most easy-to-make mistakes and how to avoid them, so you can be confident in your calibration results.

Unlocking the Truth Behind Alarm Management Metrics

Alarm Management is a well understood process, supported by global standards and operating best practices. But ultimately, the process of Alarm Management goes beyond Alarm Benchmarking and KPI Reporting. To drive alarm system improvement, action is required. So, what action should you take? What are the metrics really telling you about how your plant is operating? In this joint presentation, Honeywell’s Global Alarm Management Product Director Tyron Vardy and Manufacturing Technology Fellow Nicholas Sands unlock the truth behind your plant’s alarm management metrics.

Protecting Cyber Assets and Manifest Destiny from the Industrial Internet of Threats

During the 1800s, settlers saw it as their “Manifest Destiny” to settle the American West; but, found their lands under attack by the cattlemen surrounding them. The Manifest Destiny of industrial process and power generation companies is under similar assault. Bands of outlaws, or hackers, are cutting down perimeter-based defenses and successfully infiltrating process control networks (PCN). They are aided by growing attack surfaces created by the Industrial Internet of Things (IIoT) adoption; it is why IIoT is often referred to as the Industrial Internet of Threats. These and other factors put the highly complex, proprietary, and heterogeneous cyber assets in the plant at risk. Watch this webinar to listen to a discussion on the current landscape of ICS cybersecurity solutions. We will share how ISA advises companies to proceed and discuss “gotchas” that can derail an ICS cybersecurity initiative.

 

 

When and How to Use Derivative Action in a PID Controller

When and How to Use Derivative Action in a PID Controller

The following technical discussion is part of an occasional series showcasing the ISA Mentor Program, authored by Greg McMillan, industry consultant, author of numerous process control books, 2010 ISA Life Achievement Award recipient and retired Senior Fellow from Solutia Inc (now Eastman Chemical). Greg will be posting questions and responses from the ISA Mentor Program, with contributions from program participants.

Introduction

Derivative action is the least frequently used mode in the PID controller. Some plants do not like to use derivative action at all because they see abrupt changes in PID output and lack an understanding of benefits and guidance on how to set the tuning parameter (rate time). Here we have a question from one of the original protégés of the ISA Mentor Program and answers by a key resource on control Michel Ruel concluding with my view.

Hector Torres’ Initial Question

Is there a guideline in terms of when to enable the derivate term in a PID?

Michel Ruel’s Initial Answer

Derivative is more useful when dead time is not pure dead time but instead a series of small time constants; using derivative “eliminate” one of those small time constants.

You should use the derivative time equal to the largest of those small time constants. Since we usually do not know the details, a good rule of thumb is adjusting Derivative time to half the dead time.

Adding derivative (D) will increase robustness (higher gain and phase margin) since D will reduce apparent dead time of the closed loop.

A good example is the thermowell in a temperature loop: if the thermowell represents a time constant of 10 s, using a D of 10 seconds will eliminate the lag of the thermowell.

Hence, the apparent dead time of the closed loop is reduced and you can use more propositional, shorter integral time; the settling time will be shorter and stability better.

When you look at formulas to reject a disturbance, you observe that in presence of D, proportional and integral can be stronger.

We recommend using derivative only if the derivative function contains a built-in filter to remove high frequency noise. Most DCSs and PLCs have this function but some do not or there is a switch to activate the derivative filter.

Hector Torres’ Subsequent Question

What does having a higher phase margin increase the robustness?

Michel Ruel’s Subsequent Answer

Robustness means that the control loop will remain stable even if the model changes. Phase and gain margin represents the amplitude of the change before it becomes unstable, i.e. before reaching -180 degrees or a loop gain above one.

Ta analyze, we use open loop frequency response, the product of controller model and process model. On a Bode plot, gain are multiplied (or added if plot in dB) and total phase is the sum of process phase and controller phase.

Phase margin is the number of degrees required to reach -180 degrees when the open loop gain is 1 (0 dB). If this number is large (high phase margin), the system is robust meaning that the apparent dead time can increase without reaching instability. If the phase margin is small, a slight change in apparent dead time will bring the control loop to instability.

Adding derivative adds a positive phase, hence increases phase margin (compare to adding a dead time or a time constant that reduces the phase margin).

The ISA Mentor Program enables young professionals to access the wisdom and expertise of seasoned ISA members, and offers veteran ISA professionals the chance to share their wisdom and make a difference in someone’s career. Click this link to learn more about how you can join the ISA Mentor Program.

Greg’s Concluding Remarks

The use of derivative is more important in lag dominant (near-integrating), true integrating, and runaway processes (highly exothermic reactions). The derivative action benefit declines as the primary time constant (largest lag) approaches the dead time because the process changes become too abrupt due to lack of a significant filtering action by a process time constant.

Temperature loops have a large secondary time constant courtesy of heat transfer lags in the thermowell or the process heat transfer areas. Setting the derivative time equal to the largest of the secondary lags can cancel out almost 90 percent of the lag assuming the derivative filter is about 1/8 to 1/10 the rate time setting. Highly exothermic reactors can have positive feedback that causes acceleration of the temperature. Some of these temperature loops have only proportional and derivative action because integral action is viewed as unsafe.

If a PID Series Form is used, increasing the rate time reduces the integral mode action (increases the effective reset time), reduces the proportional mode action (decreases effective PID gain or increases effective PID proportional band) and moderates the increase in derivative action. The interaction factors moderates all of the modes preventing the resulting effective rate time from being greater than one-quarter the effective reset time. This helps prevent instability if the rate time setting approaches the reset time setting. There is no such inherent protection in the ISA Standard Form. It is critical that the user prevent the rate time from being larger than one-quarter the reset time in the ISA Standard Form. While in general it is best to identify multiple time constants, a general rule of thumb I use is the rate time should be the largest of a secondary time constant identified or one-half the dead time and never larger than one-quarter the reset time.

It is critical to convert tuning based on setting units and PID form used as you go from one vintage or supplier to another. It is best to verify the conversion with the supplier of the new system. The general rules for converting from different PID forms are given in the ISA Mentor Program Q&A blog post How Do You Convert Tuning Settings of an Independent PID with the last series of equations K1 thru K3 showing how to convert from a series PID form to the ISA Standard Form.

In general, PID structures should have derivative action on the process variable and not error unless the resulting kick in the PID output upon a setpoint change is useful to get to setpoint faster particularly if there is a significant control valve or VFD deadband or resolution limit.

A small setpoint filter in the analog output or secondary loop setpoint along with external reset feedback of the manipulated variable can make the kick a bump. A setpoint lead-lag on the primary loop where the lag time is the reset time and the lead is one-quarter of the lag or a two degrees of freedom structure with the beta set equal to 0.5 and the gamma set equal to about 0.25 can provide a compromise where the kick is moderated while getting to the primary setpoint faster.

See the ISA book 101 Tips for a Successful Automation Career that grew out of this Mentor Program to gain concise and practical advice. See the InTech magazine feature article “Enabling new automation engineers” for candid comments from some of the original program participants. See the Control Talk column “How to effectively get engineering knowledge” with the ISA Mentor Program protégée Keneisha Williams on the challenges faced by young engineers today, and the column “How to succeed at career and project migration” with protégé Bill Thomas on how to make the most out of yourself and your project. Providing discussion and answers besides Greg McMillan and co-founder of the program Hunter Vegas (project engineering manager at Wunderlich-Malec) are resources Brian Hrankowsky (consultant engineer at a major pharmaceutical company), Michel Ruel (executive director, engineering practice at BBA Inc.), Leah Ruder (process systems automation group manager at the Midwest Engineering Center of Emerson Automation Solutions), Nick Sands (ISA Fellow and Manufacturing Technology Fellow at DuPont) and Bart Propst (Process Control Leader for the Ascend Performance Materials Chocolate Bayou plant). 

Image Credit: Wikipedia

 

How to Manage Pipeline Valve Positioner and PID Tuning

How to Manage Pipeline Valve Positioner and PID Tuning

The following technical discussion is part of an occasional series showcasing the ISA Mentor Program, authored by Greg McMillan, industry consultant, author of numerous process control books, 2010 ISA Life Achievement Award recipient and retired Senior Fellow from Solutia Inc (now Eastman Chemical). Greg will be posting questions and responses from the ISA Mentor Program, with contributions from program participants.

Hiten Dalal’s Question #1

I have been trying to get a handle on small ripples in one of the pipelines by using a rule of thumb to successively reduce proportional action by 20 percent and integral action by 50 percent.  Using the same rule, I could stabilize the ripples on Friday. On Sunday, the product changed in the pipeline and with that back came those 4 percent ripples. There is one control valve that impacts line pressure. I could stretch ripples a bit but could not eliminate them. Output going to zero is natural scheduled shutdown of pipeline. I know it is a lot of information that I am providing but perhaps you can glance through and pinpoint something that stands out. I am learning since I started tuning the control valve that it is product sensitive as well.

Greg McMillan’s Answer #1

Since I don’t know if there is a trend of valve signal and valve flow, I am not sure what is happening. If the considerable decrease in gain does not help or makes it worse, I am wondering if there is some valve stiction or backlash, respectively. Is the valve the same for both products? Could a product be causing more stiction due to buildup or coating on valve seating or sealing surfaces or stem? Could the Sunday valve be closer to the shutoff where friction is greatest?

It sure looks like you have too much proportional (P) action for the new product. The integral action is already greatly reduced and most of the overcorrection is occurring very quickly due to proportional action. I would try decreasing the proportional mode action (proportional mode gain) by 50 percent (cut gain in half). If this helps, reduce the proportional gain again. Based on the very small integral (I) action, you may be able to increase integral action once you decrease proportional action. However, I reiterate that if decreasing the gain simply increases the period of the oscillation, you have backlash or stiction. If amplitude stays the same, you have stiction.

Please make sure there is no integral action in the digital valve controller.

Hiten Dalal’s Question #2

When you say no integral action, do you mean in valve positioner or in controller? I don’t think our positioner has any PID setup. Only PID action is in controller. Since it is liquid pressure and flow, we use P&I. Are you suggesting we use only P action in my controller?

Greg McMillan’s Answer #2

I meant no integral in the valve positioner that for Fisher is called a digital valve controller (DVC). You should use integral action in most process controllers (e.g., flow and pressure). Integral action in the process controllers is essential for the PID control of many processes. So far as tuning the process controller for pipeline control, the integral time also known as reset time (seconds per repeat) should generally be greater than four times the deadtime for an ISA Standard Form. You must be careful about what PID form, structure and tuning setting units are being used. If the integral setting is an integral gain, such as what is used in the “parallel” PID form depicted in textbooks and used in some PLCs, the integral setting may not just be a simple factor of the deadtime (e.g., four times deadtime) but will also depend upon other dynamics. Also, some integral settings are in repeats per minute instead of seconds.

The ISA Mentor Program enables young professionals to access the wisdom and expertise of seasoned ISA members, and offers veteran ISA professionals the chance to share their wisdom and make a difference in someone’s career. Click this link to learn more about how you can join the ISA Mentor Program.

Please make sure you extensively test any tuning settings by making small changes in the setpoint with the controller in automatic or in the controller output by momentarily putting the controller in manual. There should be little to no oscillation. The tests should be done at different valve positions particularly if the valve installed flow characteristic is nonlinear. Oscillations may be most prone near the shutoff positioner where stiction is greatest from seat/seal friction.

If there is interaction between loops, the least important loop must be made slower or decoupling used by means of a feedforward signal. If you are going to do some optimization via a controller that seeks to minimize or maximize a valve position, the proportional gain divided by the reset time for this controller doing optimization must be an order of magnitude smaller than process controller to prevent interaction. These PID controllers used for optimizing a valve position are called “valve position controllers” (VPC). I hesitated to mention this to avoid confusion because these are not valve positioners and are only used for optimization. Also, nonlinear or notch gains and directional move suppression via external reset feedback are used to keep the VPC from responding too much or too little so the process controller does not oscillate or run out of valve.

Many newer smart positioners have added integral action to positioners in the last two decades. In some cases, integral action is enabled as the default. This prompted me to write the Control Talk blog post “Getting the Most Out of Positioners.” This blog does not address setting integral action in process controllers (e.g., flow and pressure controllers).

Hiten Dalal’s Question #3

Do you teach a control valve tuning class? Is there a specific method you recommend for a pipeline control valve?

Greg McMillan’s Answer #3

I do not offer a class on tuning positioners. Supplier courses on tuning positioners are good but you will need to insist on turning off integral action. You can have them talk to me if they disagree. In general you should make sure you do not use integral action and that you use the highest valve positioner gain that does not cause oscillation since for pipeline flow and pressure control, oscillations are not filtered. If you have an Emerson Digital Valve Controller (DVC), I recommend “travel control” with no integral action and with the highest gain that still gives an overdamped response. The valve must be a true throttling valve and not an on-off valve posing as a throttling valve as discussed in the Control Talk blog “Getting the Most out of Valve Positioners”. Note that in this blog we are going for a more aggressive response than what you need. Because of the lack of a significant process time constant in a pipeline, you need a smooth valve response. In the blog, the valve positioner gain is described to be set high enough to cause a slight overshoot and oscillation that quickly settles out. Oscillations in the valve response are useful to get a faster response for vessels and columns since there is a a large process time constant to filter out oscillations. You want to still use a high gain and no integral action in the positioner but seek an overdamped (non-oscillatory) response of valve position.

Hiten Dalal’s Follow-up Reply

I have bought Tuning and Control Loop Performance Fourth Edition. I reference tables from there for suggested PID values. I have removed derivative from several pressure and flow loops and observed them to be equally efficient. In the process of tuning I have learned that operations installations have impact on loop tuning. I have made the following types of corrections,

(1) As installed, the logic had the PID getting initiated as soon as block valve #1 was fully opened but block valve #2 was getting commanded to open after #1 causing PID output to ramp off to high output limit since the control valve was not seeing full flow. We solved this by setting temporary upper clamp in PID output at safe limit to avoid overshoot until block valve #2 was fully opened.

(2) Transmitter range was high and margin of error was not acceptable by operations. Re-ranged transmitter to suitable range and brought error within acceptable margin.

(3) EIM Controls Electric and REXA electrohydraulic actuators have a limit on number of actuations. I added an acceptable dead band to reduce number of actuations.

See the ISA book 101 Tips for a Successful Automation Career that grew out of this Mentor Program to gain concise and practical advice. See the InTech magazine feature article “Enabling new automation engineers” for candid comments from some of the original program participants. See the Control Talk column “How to effectively get engineering knowledge” with the ISA Mentor Program protégée Keneisha Williams on the challenges faced by young engineers today, and the column “How to succeed at career and project migration” with protégé Bill Thomas on how to make the most out of yourself and your project. Providing discussion and answers besides Greg McMillan and co-founder of the program Hunter Vegas (project engineering manager at Wunderlich-Malec) are resources Brian Hrankowsky (consultant engineer at a major pharmaceutical company), Michel Ruel (executive director, engineering practice at BBA Inc.), Leah Ruder (process systems automation group manager at the Midwest Engineering Center of Emerson Automation Solutions), Nick Sands (ISA Fellow and Manufacturing Technology Fellow at DuPont) and Bart Propst (Process Control Leader for the Ascend Performance Materials Chocolate Bayou plant). 

Image Credit: Wikipedia

What Are Best Cutover Strategies for Upgrading an Industrial Plant?

What Are Best Cutover Strategies for Upgrading an Industrial Plant?

The following technical discussion is part of an occasional series showcasing the ISA Mentor Program, authored by Greg McMillan, industry consultant, author of numerous process control books, 2010 ISA Life Achievement Award recipient and retired Senior Fellow from Solutia Inc (now Eastman Chemical). Greg will be posting questions and responses from the ISA Mentor Program, with contributions from program participants.

Here we see how a general and open-ended question can lead to a very insightful, useful, and at times, humorous discussion on a critical phase of migration projects where automation is transferred from the old to a new system with many safety and performance implications. The question was posed by Marsha Wisely, a relatively new protégée. The answer is provided by Hunter Vegas, ISA Mentor Program co-founder, who has the most extensive practical experience of the program resources in automation project design and execution.

Marsha Wisely’s Initial Question

I am looking for general information on the different types of cutovers and what the impact of each type may be. If this topic is too big, I would appreciate even some quality documentation on the topic such that I can read up and maybe come back with more specific questions.

Hunter Vegas’s Initial Answers

It IS a pretty open-ended question, and the topic is rather huge. Books by practitioners are more useful because they reflect actual plant experience. Greg adds that academics who have partnered with industry have expanded our understanding and principles through theory supporting practice. You get good at cutovers by doing them – lots of them – and learning (sometimes the hard way) what works and what does not. Older folks in automation have lots of gray hair, or none at all, and plant startups are most likely to blame. There are several universal constants that you must know about startups:

a) Instrumentation and automation are almost always the last thing to install and check out because you can’t hook up instruments on pipes that don’t exist, and you can’t talk to instruments if there are no wires. Invariably the civil, mechanical, and electrical installations all get delayed on a cutover but the start date does not slide so instrumentation is ALWAYS critical path. You start off with three weeks to install your equipment and that usually gets whittled down to three hours. “You mean you aren’t DONE yet?!?!?!”

b) You can’t run the plant without instrumentation so if your manager keeps hounding you about when you’ll be done, tell him that you absolutely promise that you’ll be done before he starts up.

c) Regardless of who is on what shift, the night shift is almost always half as productive as the day shift.

d) Good, Fast, and Cheap (pick any two).

e) There is no such thing as too much coffee on a startup.

Cutover types:

The different “cutover types” isn’t nearly cut and dried or simple. I’ll start by talking about a control system retrofit as opposed to a new plant or a major plant expansion. Usually a retrofit has a couple of different types:

1)  Software upgrade – same instruments, same hardware (or maybe a few upgraded computers), and a new revision of software.

2)  Hardware upgrade – same field instruments, new hardware and software. (Maybe same vendor, maybe not)

3)  Total revamp – upgrade/replace a lot of field instruments and the entire control system.

All of those can be done in various ways that have different impacts on the production plant. Nearly all COULD be done without shutting down the plant. It would be expensive, and it would take a while, but it can certainly be done. I’ve converted three or four plants from pneumatic controllers on panelboards to full-blown distributed control systems (DCS) and never shut them down. Ultimately, the economics of a plant production drives the decision. If they are making a million dollars of product a day, they can afford to pay an extra $200,000 to cutover on the fly. Similarly, if they are barely running eight hours a day, a two- or three-day outage doesn’t matter. Often the cutover is a mix – you do as much work as possible prior to shutting down and then you come down and finish the rest.

The ISA Mentor Program enables young professionals to access the wisdom and expertise of seasoned ISA members, and offers veteran ISA professionals the chance to share their wisdom and make a difference in someone’s career. Click this link to learn more about how you can join the ISA Mentor Program.

HOW do you cutover?

That totally depends on the plant, the process, and the timing. When I am looking at a retrofit project, I start by talking to the plant and understanding a few things:

1)  Their budget

2)  Their schedule

3)  The financial justification for the project

4)  What their “pain points” are

Based on that, I then start going over the project – looking at panels, mulling my options, determining what I have to do, and how I might go about it. I also investigate what options I have and what is required to change over the system. Is the I/O compatible? Are there interconnect options and do they work? (Just because a company SAYS they offer interconnect options, does NOT mean they actually work as advertised!) Eventually, I pull it all together and hatch a plan, figure what it will cost, and present it to the client. If they can afford it and like it, we are off to the races. If they don’t, then we tweak it accordingly. Maybe they can afford to take a longer outage and save on installation costs. Maybe they are willing to pay more to bring the plant on line faster. Maybe there is a way to jury rig things to get the bulk of the benefit now and do the rest of the cutover later when they have more time.

Well, that’s a start… let’s see what questions this generates.

Marsha Wisely’s Subsequent Questions

Thanks for the information! When you have only seen one of something, it’s hard to know what the norm is or what the other options are. Also, I really like your universal constants section! After reading your comments, I started thinking about how the information applied to my experience. Below I have a summary of my experience to give you some background on me, and it inspired some questions as well.

  • Type of project: brownfield/retrofit
  • My role was strictly software-oriented
  • Moving from a combination of a programmable logic controller (PLC) and legacy system (Provox) to a modern DCS platform (DeltaV)
  • The idea was for it to be a software-only upgrade, temporarily
    • Profibus DP served as the interface between the PLC and modern DCS platform
    • A specialized interface (DeltaV Controller Interface for Provox I/O) was used to communicate with the legacy system’s I/O
  • To your point on the interconnect options: both solutions above were tested prior to startup, which saved us some headaches I’m sure
  • The long-term goal was a full conversion to electronic marshalling
  • The customer also had some existing control modules on the modern DCS platform they wanted to integrate once we were onsite.

The cutover and startup both took a good deal longer than previously anticipated. Loop checkouts were taking longer than anticipated, the plant discussed how to improve, but management never prioritized timeliness over safety and quality of work, which I respect and appreciate (I know that isn’t always the case). There was some unexpected equipment issues, namely valves were backwards (fail open when they should be fail closed and vice versa).

Startup was also slower due to some equipment issues – air lines to pneumatic valves got leaks and equipment that was working before startup needed to be repaired before being brought back up. This leads to some frustrations and finger pointing at the software – “The valves aren’t working; they worked before. It must be the software.” Definitely a good lesson in patience. In those cases, we basically did an additional loop check, which quickly found the issue. Shutdowns are in the hands of operations and can be pretty hard on equipment.

This leads to the following additional questions.

1) If you do a hardware and software cutover, are those types of equipment issues more likely to get caught? And are these types of equipment configuration errors common?

 2) Are there any common issues caused by shutting down equipment that are good to check before starting back up?

3) In your notes, you mention asking the customer about their “pain points.” Could you provide some examples?

4) My next couple projects are for new plants – one of which I will be leading. Do you have any advice specific to new plants? It looks like your “HOW to cutover section” covers a lot, but are there any additional challenges associated with new plants?

Thanks again for your help! I greatly appreciate your insight into the field since I haven’t had much field experience. My next couple projects, in addition to being new plants, also have a larger equipment and instrumentation component to my assignment, and I am looking forward to that exposure.

Hunter Vegas’s Subsequent Answers

Let me try to answer your questions:

1) If you do a hardware and software cutover, are those types of equipment issues more likely to get caught? And are these types of equipment configuration errors common?

If you are doing a hardware/software cutover you have to be fanatical about the details. Picking up failure modes of valves, ranges, square root versus linear, interlocks, etc. is pretty much a given. Only our most experienced people generally do the decoding of the existing system because it is so crucial to success. If you toss it to some inexperienced engineers, you will pay a wicked price on startup and rework.

2) Are there any common issues caused by shutting down equipment that are good to check before starting back up?

One thing I learned very early on was, if it all possible, obtain historical data of the running plant before you shut down and have it available for access during the start up. It is a very common trick for plant personnel to get the automation company to fix instrumentation that hasn’t worked for years. If you can point to the fact that the transmitter has been flat-lined since 2006, it is a pretty easy argument to say that fixing that transmitter is out of scope.

3) In your notes, you mention asking the customer about their “pain points.” Could you provide some examples?

Pain points are what keeps them up at night. Are they struggling with quality? Production? Throughput? Reliability? Does something break and it takes the techs 2 days to figure it out? Is there a particular area in the plant that is always in manual because the controls never work? Is the messaging awful so the batch stops and nobody knows why? Etc.

Often I can fix a lot of those things fairly easy and make them very happy. Or I can offer solutions that increase the work scope some but has very big payback.

4) My next couple projects are for new plants – one of which I will be leading. Do you have any advice specific to new plants? It looks like your “HOW to cutover section” covers a lot, but are there any additional challenges associated with new plants?

New plants can be very painful and difficult for a whole new set of reasons. Specifically:

a) How good is the engineering contractor? The large Architectural & Engineering (A&E) firms can be awful or wonderful; it all depends on what team you get. If you get the “A” team, things will be in pretty good shape. Unfortunately, they might bait you with the “A” team but swap you the “C” team later in the project or you end up with the “C” team due to turnover. Either way, the engineering is just wrong. Wrong pipe, wrong materials, wrong instruments, wrong drawings, wrong…wrong…wrong. And it’s your job to make it work using BIFF principles (Big Improvements for Free) because the money has already been spent.

b) Was anything reviewed by someone other than the engineering contractor? The best option is for the plant personnel to review and approve everything; however, the plant often lacks either the expertise or the time to do that and the engineering contractors are infamous for dropping 1,000 pages on your desk and demanding you approve it overnight or “the project will be delayed and it was your fault.” The second-best option is for a third party to at least look things over and catch the worst stuff. If nobody looks it over, then you better have the “A” team or it will be bad.

c) Does anyone even know how this is supposed to work? Is the plant a copy of an existing plant and the process is well understood (and ideally you have people from that plant helping you) OR is this Serial #1 and the only people who have a vague clue are some lab chemists who had a tiny pilot reactor running somewhere? Obviously, Serial #1 is going to be tricky because nobody knows what they don’t know and nobody has the answer.

d) What is the schedule/budget like? Was it stupid aggressive to start? If so, it will be tough, as nothing ever goes to plan on a new plant and both the budget and schedule are bound to suffer. I have had no-bid projects that I knew were badly estimated because failure was virtually assured, and I was much happier having my competitor get the black eye than me!

Now don’t get me wrong – large projects are executed successfully all the time and with the right team things can go very smoothly. But if the client goes with an unproven, low-bid entity, they often regret the decision. I once was part of a large project that was more than half-way completed but things were going so badly that the client fired the A&E firm mid project and went with a new one. They lost $10 million in engineering but probably saved $20 to $30 million in extra startup costs because the first firm was doing so poorly.

See the ISA book 101 Tips for a Successful Automation Career that grew out of this Mentor Program to gain concise and practical advice. See the InTech magazine January/February 2013 feature article “Enabling new automation engineers” for candid comments from some of the original program participants. See the May 2015 Control Talk column “How to effectively get engineering knowledge” with the Mentor Program protégée Keneisha Williams on the challenges faced by young engineers today.

Discussion and answers are provided by Greg McMillan, Hunter Vegas (co-founder of the ISA Mentor Program and project engineering manager at Wunderlich-Malec), Brian Hrankowsky (consultant engineer at a major pharmaceutical company), Michel Ruel (executive director, engineering practice at BBA Inc.), Leah Ruder (process systems automation group manager at the Midwest Engineering Center of Emerson Process Management), Nick Sands (ISA Fellow and Manufacturing Technology Fellow at DuPont) and Bart Propst (Process Control Leader for the Ascend Performance Materials Chocolate Bayou plant).

How to Eliminate Lagging Plant Construction and Upgrade Project Schedules

How to Eliminate Lagging Plant Construction and Upgrade Project Schedules

This post was written by Tim Green, U.S. operations manager at MAVERICK Technologies.

When a company launches a major project—something on the order of a new plant, process unit, or automation system upgrade—a schedule is produced. For far too many companies, the schedule is merely symbolic, as they do not believe it will be strictly adhered to as the project progresses.

Such an attitude is simply wrong, and can end up being a self-fulfilling and expensive prophecy. Schedule delays add costs, because labor resources spend more time than planned. But maybe worse, the company loses income every day its new plant or equipment is not running.

projects-fall-behind-schedule-can-cause-significant-cost-increases

Projects that fall behind schedule can cause significant cost increases beginning with capital expenses by virtue of the additional time needed.

Many articles have been written on project planning, so for this discussion we will assume that planning has been well executed, and concentrate on the second and especially the third factors.

Active management

Once a project moves from planning to actual construction, it has to be actively managed to stay on course and on schedule. The key word is actively. Without proper supervision, projects can quickly fall behind. Why do schedules get off track?

Too many companies look at construction as a black box. They understand it begins and ends, but there is little sense of what goes on in the middle, so management is passive. Those charged with project management often simply expect everything to happen on its own. Without people who can dive in and get a grip on each project task, companies feel helpless to drive or even manage the process.

The project management team must work with contractors and monitor the schedule constantly, but must avoid becoming micromanagers. There is a fine balance between being effective and being a nuisance, and each situation is different. It can take some time for novice managers to develop such a sense, but it is critical for effective project management.

A key first step in active management is opening the black box and breaking the construction timeline into subsections, so each can be monitored and tested along the way. Some things do have to happen in sequence. A flowmeter cannot be installed before there is piping, but it should be possible to look ahead while the piping is being fabricated. Are the flowmeter and all its necessary fittings on hand so it can be installed without delay when the piping is done? Does it require any additional support structure to be fabricated along with the piping?

Steps that can run simultaneously should do so, because this makes the schedule collapse on itself, and it reduces overall project time. Those steps are not likely to happen simultaneously without getting into the construction black box.

Testing has to happen at each step as soon as possible to ensure every element is functioning properly before a particular contractor moves on. Few companies realize the benefits of this approach until they see it firsthand. Integrating testing with construction drives the schedule and keeps everyone, employees and contractors, focused. Individual contractors are held accountable for correcting any problems they caused and for fixing items left undone, all while they are still on site.

Proactive testing

Maintaining an aggressive project schedule requires testing at each step of the process to verify performance and fulfillment of specifications. Given the criticality of testing, it needs to be divided into three phases or stages, corresponding to the relevant stage of construction (table 1). In a large-scale project, all three phases can often take place simultaneously in different areas.

Prestatic inspection is designed to identify installation issues early in the process. It also helps quantify the percentage of completion of a specific portion of the project. This is critical to keeping parallel activity moving to fulfill the schedule. Without it, the timeline often stretches out, because the desired overlap is eliminated.

Static checks deal with wiring issues during electrical construction. These checks are performed before any of the equipment is energized, primarily checking wiring continuity and correct wire tagging. For motors, thermal protection and correct rotational direction may be verified. Like prestatic inspection, this step is critical to verifying a contractor’s performance and percentage of completion.

Predynamic testing is the first stage when equipment is energized. It confirms functionality of instruments, valves, and motors—and verifies operability from the appropriate controller. Process variables are simulated to verify scaling; valves are given full-stroke tests; motors are bumped; variable frequency drive operation is confirmed, and so forth. This is the last phase of functional testing before full dynamic testing.

Each test is performed as early as possible, corresponding to the stage of construction, and all testing is documented and incorporated into the schedules.

Commissioning and startup

As the final stages of construction are taking place, the project is almost ready for full dynamic testing. Using the heel-to-toe process of testing at each phase of the project means there is no lengthy period at the end when all elements have to be tested at once.

Full dynamic testing is the final stage when equipment is energized and controller logic is exercised. This testing ensures adequate interlock protection is in place for a safe process, and operational logic functions are in accordance with the needs of the specific process. This live logic testing confirms operation as defined by the control narratives.

Now the instrumentation and controls (I&C) team gets to see the culmination of all the efforts, as the last elements are put in place and late-stage design modifications are made. But there is still much to be done as commissioning and startup begin. At this point, ownership transfers from the I&C provider to plant personnel. The I&C team members of the larger startup team support the process experts as they execute their full dynamic test plans.

  • The plant and the automation solutions provider typically verify functionality and interlocks by performing water runs or some other full functionality simulation.
  • Complete control system functionality is verified.
  • Final loop tuning is performed.

Resource flexibility

Considering all the project activities that must take place in a short period, any company trying to carry out such a comprehensive range of tasks has to have huge resource flexibility. Over the weeks and months of a project, there will be times of relative inactivity, and other periods when many things have to happen together, such as when a major phase is nearing completion.

Keeping things moving as quickly as possible while maintaining peak personnel efficiency demands constant adjustment of not only headcount on a site, but also of personnel skill sets. The ability to have the right number and right type of people on the clock, no more and no less, requires a pool of highly qualified engineers and technicians ready to move as needed. Major automation solutions providers have this kind of flexibility, and this capability is critical to realizing all the gains possible from more aggressive scheduling, while still controlling costs.

A major upgrade project can require more than 100 people to be on site simultaneously for the startup and commissioning of just the electrical and automation system portions of the project, typically requiring the plant owner to engage outside assistance to meet the schedule.

Hot cutover techniques

For many production facilities, the costs of interrupting production to perform an automation system upgrade are prohibitive. However, in some cases the need to implement such an upgrade is also highly compelling. The solution to this dilemma is performing a hot cutover while the plant remains in production. This stage of a project requires very careful planning and coordination between the plant operations team and the automation solutions provider.

A hot cutover moves one control loop at a time from the old to the new automation system. Each loop needs to be verified and tuned in the process. All parts of the new system hardware must be thoroughly tested and need to perform flawlessly so there are no unfortunate surprises at any step. Both automation systems have to run simultaneously, so all digital communication networks must be capable of supporting increased traffic.

Also, engineers and technicians must test control code in the new system to make sure it operates correctly during and after the transition. They also need to compare output image tables of the new controller with image tables in the live existing system for parity. Once they have confirmed 100 percent parity, the cutover can begin.

This kind of transition can be smoothly carried out if done carefully by experienced engineers and technicians. Skilled automation solutions providers have performed many such transitions during large distributed control system upgrades and migrations without interrupting production. Such a track record depends on having talented people and the know-how to perform a wide range of critical functions.

Closeout and documentation

For new systems or those upgraded during an outage, a full dynamic test is the final check before putting the new unit or automation system into production. At this point, every hour counts, because everyone wants to close out the project and realize its income.

Using traditional project management methods, full dynamic tests can be slow and painful as numerous bugs and problems crop up. Inadequate testing at earlier stages leaves undiscovered problems, so the test moves by fits and starts as those bugs are identified and fixed. Many expensive people may be left standing around while a technician corrects the rotation of a pump or figures out why a valve will not respond to a control system command. In many cases, test results prove inconclusive, and engineers must burn up time with troubleshooting and fixes.

By contrast, the right testing and other procedures ensure this final stage moves quickly and deterministically toward a positive conclusion, because:

  • All systems and subsystems have already been verified at each critical stage
  • Commissioning and startup personnel do not have to wait for components to be fixed
  • The process gets to run continuously without stops and starts
  • Safety incidents and damage to equipment are avoided
  • Plant personnel see a reliable representation of the live process for a conclusive test

Meanwhile, all the documentation compiled throughout the planning and construction process can be easily assembled, so it is accurate, complete, and ready for turnover.

As a final step, the automation solutions provider sits down with the plant project team to discuss collective reactions to the project. This debriefing is something of a celebration, but also a time to gather observations and lessons learned to pass on to operations and maintenance people. Tribal knowledge should be collected and documented to support future projects.

Conclusion

Quality automation solutions providers perform project management, startup, and commissioning work, bringing a high level of experience and technical skill to projects. Some companies consider hiring outside help on such an extravagant scale beyond a project’s budget, believing it cheaper to use only internal resources. However, the reality is few, if any, companies these days have the necessary resources available. If internal people are pulled into a construction project, what other tasks are not going to be done?

Consider the situation as a mathematical proposition: A project of whatever size is going to require some basic number of man-hours to get all the functions done (figure 2). A company can bring in a large number of people to do it in a short time, or it can have a small group of people doing the project on a longer timeline. Either way, the project cost is essentially the same, but if the plant starts up sooner, the company enjoys the income it generates sooner. Moreover, when a project can be managed as described here, the company can realize major efficiency gains.

An effective automation solutions provider improves efficiency by having exactly as many people on a site as the situation demands to push the schedule as far as practical. The number might be 30 this week, but 110 the next. Such flexibility is impossible with internal resources.

When a qualified and experienced automation solutions provider, capable of bringing all the necessary resources to bear on a project, can guide project planning and manage construction and testing, companies receive the highest assurance the project will meet all performance expectations, will conclude on schedule, and will be at or under budget. The costs of such services are typically recovered quickly through improved performance and increased production realized through an earlier startup.

Figure 2. Any project requires a certain level of effort (in man-hours) over a specific period of time. The upper graph illustrates a project operating under a theoretical best-case scenario given the practical constraints of labor efficiency for the work being performed. If all the resources necessary can be brought to bear, the work can be completed in this period of time. The middle graph indicates what happens in most situations where companies try to carry out projects with only internal resources or minimal outside help. The number of actual man-hours does not necessarily increase, but the total amount of time required to get the job done at the lower effort level pushes out the startup time, delaying the beginning of production and income generation. The lower graph illustrates what can happen when greater resources are available and can be combined with aggressive project management. Under these circumstances, even the theoretical best schedule can often be beaten without increasing overall costs significantly. The biggest benefit is production begins on time, or even early, allowing the company to realize production income earlier.

Figure 2. Any project requires a certain level of effort (in man-hours) over a specific period of time. The upper graph illustrates a project operating under a theoretical best-case scenario given the practical constraints of labor efficiency for the work being performed. If all the resources necessary can be brought to bear, the work can be completed in this period of time.
The middle graph indicates what happens in most situations where companies try to carry out projects with only internal resources or minimal outside help. The number of actual man-hours does not necessarily increase, but the total amount of time required to get the job done at the lower effort level pushes out the startup time, delaying the beginning of production and income generation.
The lower graph illustrates what can happen when greater resources are available and can be combined with aggressive project management. Under these circumstances, even the theoretical best schedule can often be beaten without increasing overall costs significantly. The biggest benefit is production begins on time, or even early, allowing the company to realize production income earlier.

About the Author
tim-greenTim Green is U.S. operations manager for MAVERICK’s Field Services division. Green began working in the electrical field in the U.S. Navy in San Diego, Calif. During his career he has worked as an industrial electrician, instrument technician, PLC programmer, engineering manager, and technical sales professional.

Connect with Tim:
LinkedInEmail

 

A version of this article originally was published at InTech magazine.

Click here to read Tim Green’s article at InTech magazine.

Optimize Your Industrial Automation System with a Resilient Design Strategy

This guest blog post was written by Paul Darnbrough, P.E., CAP, a principal engineer for MAVERICK Technologies.  This is the final part of a three-part series. Click to read Part 1 or read Part 2.

In a previous post regarding good automation system design concepts considered the issue of robust design at the controller and communication levels. This time we shift the focus to software best programming practices, both at the controller and the operator interface levels, which can help result in a solid automation system.Optimize Your Industrial Automation System with a Resilient Design Strategy

Probably the No. 1 programming recommendation is to validate all information before using it. Within the control system platform, validate any process signal data, classic or networked, to confirm it has good quality. This especially applies to ensuring that analog input values are scaled properly within the allowable range. Incorporate debounce logic on discrete inputs so the brief time delay will ensure that a positive field signal has been received.

Validate operator inputs

Many kinds of data are entered by operators via an HMI/OIT or other device, in order to command the control system operation. Always validate operator inputs. This primarily applies to checking that operator-entered setpoint values are within a legal range. However, it can even apply to discrete push-button presses. For instance, incorporating a brief “push and hold” delay on a start button can guard against accidental pushes. Furthermore, multi-mode push-buttons should be made mutually exclusive and configured so that the safest choice (if possible) dominates. For a start/stop pair of buttons, the “stop” function would prevail over the “start” function.

When it comes to validating operator-entered data, many HMI/OIT packages offer this capability. An even more rigorous method is to limit-test the data within the control logic before accepting it. Individual project needs will dictate whether improper data is clamped at the limit, or rejected in a way that warns the operator. For logic that calculates derived values during runtime, if these values are used for subsequent operations they need to be validated just as if they were operator-entered. The most common validation scenarios are ensuring a value is not over-range or under-range, or making sure a value is not negative.

Consequences of bad data

What are the consequences of bad data? Improperly ranged or negative values can cause control loops to wind up. An unexpected zero can trigger a processor-halting divide-by-zero error in a calculation. Or, an improperly set or incremented value used as an indirect array address can actually point to an illegal location outside the valid range, causing a processor to stop running.

Read the complete three-part blog series. Click to read Part 1 or read Part 2.

At a higher level, functional systems should be programmed to enter a safe state (usually “off”) upon system boot up, or on any critical error. Startup routines can be created in order to initialize data and values, and to drive control logic to a safe state. Rarely should sequences automatically restart after an alarm condition is removed. Instead, consider a two-step procedure where operators clear the error, then trigger a restart.

During detailed design, consider adding some enhanced alarming that can help operators identify unusual trouble. For instance, a system with a pump filling a tank may have high level, low level, and pump fail alarms. If it is known that the tank should only take 5 minutes to fill, then a “slow fill” alarm can be incorporated. This alarm will not specifically define why a slow fill is happening, but could prompt the operator to go look for a broken hose or fitting that is spilling water to the floor.

Configure software to self-recover

In addition to disallowing any invalid operational modes, software should be configured to self-recover itself to a safe mode if it is ever inadvertently driven to an illegal mode. Sometimes it makes sense to give the operator a “reset” or “abort” control that can stop and re-initialize a problematic sequence. Keep in mind that some would consider this type of logic to be a Band-Aid intended to make up for other poor programming practices.

Fault and alarm indications on HMI/OIT stations must be clear and understandable. Cryptic messages or codes cannot be reliably acted upon. System reactions to operator inputs must be responsive enough to prevent operators from making multiple selections which could trigger undesired operation. Just as with consumer devices like phones and DVRs, a lagging response will cause the frustrated user to keep pressing buttons fruitlessly.

Develop a test plan

How do you know if your good engineering efforts are sufficient to defend against the unexpected? Test, test, test! Develop a test plan, preferably around the time the system is designed, so that it tests all key features. Attempt to trigger or simulate various failures and potentially illegal operator actions. Execute the test plan, and don’t be afraid to use it as a springboard for developing additional specific test cases that look useful. Make some test actions faster and slower than typical to search out bad interactions.

We started this blog series comparing an automation engineer’s tasks with those of a driver on a challenging road. In both cases, it is clear that training, planning, practice, and experience will lead to the most successful outcome. Not every bad situation can be prevented, but a multi-layered approach for defending and reacting to the unknown is the best bet. There are usually few arguments against building a resilient automation system, which can safely and defensively respond to non-normal conditions. Always be on the lookout for opportunities to improve your designs by challenging them with various conditions that flush out potential weaknesses.

About the Author
Paul-Darnbrough
Paul Darnbrough, P.E., CAP, is a principal engineer for MAVERICK Technologies. He has more than 20 years of experience in engineering, documentation, and construction of automated industrial and process control systems. Paul has worked with clients ranging in size from small single-owner operations up to Fortune 500 companies and government agencies, involving operations in the plastics, food, dairy, chemicals, material handling, discrete manufacturing, water treatment, and pharmaceutical industries.

Connect with Paul:
LinkedIn

 

Pin It on Pinterest