Taking Denitrification to the Next Level

Taking Denitrification to the Next Level

This excerpt is from the November/December 2014 issue of InTech magazine and was written Jaime A. Alba, Peter Loomis, Robert Litzinger, Bruce P. Stevens, and Paul A. Miller.

In 2006, one of the largest water reclamation facilities in northern Virginia needed to expand the facility from 18 to 24 million gallons per day (mgd) to support future growth in Prince William County, Va. At the same time, new regulations necessitated an upgrade to improve theupgrade

nutrient removal capabilities of the plant. The new waste load allocations for total nitrogen (TN) were based on permitted discharge flows on 31 December 2010 with a 3 mg/L TN concentration. The Prince William County Service Authority (PWCSA) recognized the need to simultaneously increase flow and nutrient removal capabilities. At the same time, they needed to replace their plantwide data acquisition and control system (DACS) with a new modern supervisory control and data acquisition (SCADA) system.

This design-build project enhanced nutrient removal and increased capacity, doubling the existing aeration basin volume and reconfiguring to allow operation in either four-stage Bardenpho or modified Ludzack–Ettinger (MLE) modes. It implemented 14 new deep bed denitrification filters for a total of 24, and methanol feed to the filters was automated to be nitrate load paced and controlled by a proprietary software calculation algorithm. Furthermore, PWCSA installed an additional online analyzer for controlling the methanol feed to the filters for redundancy.

The proprietary software calculation algorithm for control is feedforward/feedback based upon flow and influent and effluent nitrate concentrations. This enhanced the operation and reliability of the process and also reduced the risk of methanol overdose by more closely matching the methanol feed to the actual demand. Consistent methanol dose control is challenging when trying to meet low effluent TN and simultaneously maintain a low effluent carbonaceous biochemical oxygen demand (CBOD).

This plant is currently in full operation and in compliance with the effluent requirements.

Process enhancements

Before the expansion, the plant operated with 10 denitrification filters that had insufficient surface area to process the full plant capacity of 18 mgd in denitrification mode. When the filters were operated in the denitrifying mode, flows beyond 12 mgd bypassed the filters to prevent hydraulic overloading. Even though the filters were capable of hydraulically passing the full plant flow, denitrification could not be achieved at higher flows. Based on the processing limitations and operational cost savings, the filters were often operated seasonally, with methanol added only during the winter for the additional denitrification needed to meet effluent TN requirements. During summer, the plant had sufficient denitrification capability in the secondary treatment (aeration basins) to meet effluent TN requirements, and the filters operated in a polishing mode without methanol addition to remove suspended solids.

During the design phase, the MLE and four-stage Bardenpho processes were selected for implementation based on a wide range of criteria, including capital cost, overall cost, net present value, land requirements, effluent quality, operability, maintainability, and schedule.

The plant was also required to reduce effluent total phosphorus to 0.18 mg/L. Phosphoric acid addition capability is provided in the filter’s area if the filters become phosphorus limited. It was anticipated that any phosphorus allowed to bleed through to the filters or added to the secondary effluent will be removed by the denitrification filters and permit limits will not be exceeded. However, provisions were provided for future implementation of phosphoric acid feed.

The number of denitrification filters was increased from 10 to 24 to meet the new projected demands and to be able to handle when filters are offline for backwashing, bumping, or maintenance/repair.

Plant-wide control system replacement

ChemScan nitrite analysis during startup

ChemScan nitrite analysis during startup

As part of this design-build project, the existing plantwide DACS was replaced. It was obsolete, with key components of the system no longer available from the manufacturer.

The facility transitioned the existing DACS to a modern SCADA system as part of the overall implementation, including designing a system with both new process area control panels and upgraded existing control panels. The final system has about 5,000 I/O points, 25 programmable logic controllers (PLCs) with a self-healing fiber-optic ring, an object-oriented human-machine interface (HMI) system, and a historian interfaced with reporting software that integrates the SCADA and laboratory databases.

The engineer of record managed construction and did quality assurance/quality control (QA/QC) for the new SCADA system and the field instrumentation portion of the project. Some of the activities included QA/QC for the new instruments, startup coordination between PWCSA and the subcontractor, onsite response to design/implementation questions and clarifications, development of maintenance of plant operations (MOPO) plans for transitioning existing and in-service systems to the new SCADA system (with the objective of minimizing the effect on plant operations), and developing and continually updating the SCADA project schedule.

Furthermore, toward the end of the project, the engineer of record also guided, witnessed, and approved the testing procedures and results for the SCADA system as a whole. This activity included network testing, uninterruptible power supply (UPS) testing, software testing, and PLC programming testing. The subcontractor performed loop testing (operational readiness test) and the functional demonstration test with coordination from PWCSA, witnessed and approved by the engineer of record.

Click here to continue reading Taking Denitrification to the Next Level at InTech magazine.

About the Authors
Jaime A. Alba, P.E. is a senior process control engineer at DC Water with 11 years of experience in the water and wastewater industry. His experience includes SCADA, HMI, and PLC design, implementation, startup and commissioning, as well as execution of QA/QC procedures and construction management.

Peter Loomis, P.E. is a senior project manager at CDM Smith with 25 years of experience in the water and wastewater industry. His experience includes treatment plant planning, design, construction, and startup/commissioning.

Robert Litzinger is the operations manager at the H. L. Mooney advanced water reclamation facility with a Virginia class I wastewater operator license and 40 years of experience in the wastewater field. His experience includes the initial commissioning of the plant more than 30 years ago as well as the most recent upgrade in 2010.

Bruce P. Stevens is a regional manager with ASA Analytics/ChemScan of Waukesha, Wis. He is based in Atlanta, Ga. and covers the southern U.S. working with municipalities and industrial clients.

Paul A. Miller is a process engineer with 23 years of experience in the municipal and industrial water and wastewater industry. His experience includes operating pilot studies, biological gravity, and pressure filter designs, metals removal technology design, and startup and commissioning of more than 90 treatment systems.

 

What is a Good Security Approach to Water and Wastewater Process Control Systems?

What is a Good Security Approach to Water and Wastewater Process Control Systems?

This post was written by Norman Anderson and Bill Phillips, of ch2m

This article is based on presentations made at the 2013 ISA Water/Wastewater and Automatic Controls Symposium on 7 August 2013. Network security for water sector process control systems (PCS), such as supervisory control and data acquisition (SCADA) systems, is increasingly important and ever evolving due to the need for secure and reliable control systems. Additionally, process control systems continue to grow, and the management of network-connected devices and the expansion of PCS networks can be difficult and cumbersome. To properly secure PCS networks, a multistage process is needed incorporating risk assessment, planning, design, implementation, and maintenance for a comprehensive defense-in-depth strategy. A critical aspect of defense-in-depth is the overall network system architecture and the network segmentation plan. A properly planned and executed network architecture and segmentation strategy lays the foundation for security and simplifies expansion and maintenance of the network.

There are industry-accepted methods for industrial control system (ICS) network architecture and segmentation strategies that can be applied to water sector process control and SCADA systems. Industry-standard techniques, based on recently published standards and network design guides, are used to create a layered network architecture approach to security, including the use of logical subnets and virtual local-area networks for segmentation. The advantage of this approach is simpler configuration of network security appliances and simpler management and expansion of the network, leading to increased network availability and a reduction in threat risk. A case study will be used to provide examples of actual methods implemented for a water sector utility.

Planning-designing for defense-in-depth

Figure 1. Planning and designing for defense-in-depth

As cyberattacks and the threat of compromised network security continue to rise, so does the need for securing industrial control systems. This includes many different types of systems, with water sector process control systems being one of the higher profile targets because their critical infrastructure affects large populations. Past statistics from the Cyber Emergency Response Team show recorded cataloged vulnerabilities and reported incidents continuing to rise through the years. A set of “honeypot” ICS set up by Trend Micro to look like vulnerable power and water plants was attacked by hackers 25 times within 28 days. Security is important for the water sector because attacks can damage critical infrastructure that affects public safety; lead to significant operational downtime; cause financial loss, such as the loss of revenue for the utility and its customers; and attract significant media attention causing loss of confidence and fear from the public. There are many resources available that provide guidance on where to start and how to secure networks. In general, there are four key steps in the process of planning and designing to secure networks for defense-in-depth, as shown in Figure 1.

Organizations should use a layered network architecture separating components within a water sector PCS by levels, using access control lists for communications between levels and keeping the most critical parts of the network in the deepest and most secured level of the network. To effectively organize this architecture, it is further necessary to logically segment the system by following industry-standard subnet organization. Further divide network-connected equipment into VLANs to allow robust communication between critical components that need to communicate and segregating components that do not require communications with each other but communicate over the same media. Subnet organization refers to organizing similar components within a network by IP addresses and more precisely by IP address blocks, or subnets, as discussed in subsequent sections. By designing a layered network architecture that uses logical network segmentation and organization, the network implementation and maintenance can be simplified, further enhancing overall network security.

Differences between corporate IT and water sector PCS

Water sector PCS Corporate IT network
Real time Not real time
Mainly used for equipment and processes to function Mainly used by personnel to create and store data
Response time is critical Consistent response time desired
Generally low bandwidth High bandwidth requirements
Rebooting must be scheduled or avoided Frequent rebooting is acceptable
Human safety and process uptime are paramount Data confidentiality and integrity is highest importance
System uptime is most critical System and data protection is most critical

Table 1. Comparison of water sector PCS vs. IT network operational requirements

In modern water sector PCSs, the use of commercial-off-the-shelf (COTS) network components has increased as these systems continue to adopt more Ethernet-connected control system components and budgets for upgrades continue to decrease. COTS solutions provide benefits to utilities, such as ease in getting replacement components, reduced cost, and simpler system integration, but have also been part of the rise in water sector PCS security concerns.

While the COTS components may be the same between corporate and industrial networks, there are critical differences between the requirements of water sector PCS networks and corporate IT networks, as illustrated in table 1.

These differences are centered on the fact that water sector PCSs are critical systems that must be kept online and running, while a corporate IT environment can tolerate downtime much more easily and are focused more on the availability and security of data. The differences in these two environments lead to different methodologies in the configuration and use of the similar network components within these systems and in the design of these networks. For example, corporate IT is fairly flat to allow many users within a facility access to the same data. Additionally, VLANs on an IT network are generally used to segment services and not necessarily to separate portions of the network where data is still required to communicate between devices. A typical corporate network may consist of the following VLANs:

Typical corporate network VLANs
Voice
Data
Server
Management
Wireless
Public Internet

Table 2. Typical corporate network VLANs

This structure allows a large number of users on the corporate network to have access to the same resources and provides a system where data is highly accessible. For a water sector PCS, it is not desirable for many users to have full access to all data on the network. Conversely, for an ICS and specifically a water sector PCS, data is needed between control processes and by a few operators. Allowing many users access to this data could be detrimental to process operations and the overall operation of water and wastewater treatment plants. For these systems then, a different network structure is needed. Here, a hierarchical layered network architecture is recommended, where VLANs are used not only to segment specific types of network traffic but also user and equipment groups to limit the accessibility of the data to specific users and processes.

A layered approach to network architecture for increased security

The starting point for a hierarchical layered network architecture is to divide the network into functional zones and to provide a hierarchy from most trusted to least trusted networks to control information flow between zones and access across zones. This description is similar to the nomenclature of zone and conduits set forth in the ISA-62443 (ISA-99) series of standards and the cell and area zones described in the Cisco and Rockwell Automation CPwE design and implementation guide. Typical zones might include the process control zone (most trusted), the PCS data zone, the PCS demilitarized zone (DMZ), the enterprise zone, the enterprise DMZ, and the external zone (untrusted). The zones are usually arranged bottom-to-top with the most trusted process control zone at the bottom and directly connected to the process, and the most accessible untrusted external zone at the top. Information flow between zones is restricted at the boundaries between zones, and access to each zone, except the external zone, is limited. A general method for devising a layered network architecture can be seen in figure 2.

This figure shows a simplified version of the organization of a water sector PCS and a recommended method for separating equipment into zones based on trust level. In general, the Internet and utility or municipality business networks should be seen as untrusted networks and ideally would be air gapped (i.e., not connected). However, there are many advantages to leveraging these untrusted networks to support remote access, reduce communications cost, and provide remote system vendors with access for maintenance and troubleshooting the package control systems. Other valuable uses of Internet connectivity include remote alarm notification and software and firmware update management. When these services are needed or desired, the remote networks used should be given the lowest trust level (0) possible and direct access should not be allowed to the SCADA or PCS programmable logic controller (PLC) networks. Only the DMZ should have access to the business networks and Internet in order for the layered architecture to be effective.

Figure 2. Layered network architecture approach

The PCS network DMZ is the location that relays communications between networks external to the PCS network and systems internal to the PCS network. Computers and applications located in the DMZ should be limited to only those necessary for remote access and notification, such as web servers or terminal servers for SCADA human-machine interface (HMI) monitoring, remote alarm notification servers, and software and firmware update servers. In some instances, it may also be necessary to locate a PLC or separate SCADA server in the DMZ to manage data communications with remote facilities communicating through the business network or Internet, and these devices should be separate from those located at higher trust levels within the PCS network. Other systems are likely to also be present in the DMZ, or preferably in separate DMZs, such as physical security related systems, reporting systems, and voice over IP (VoIP) equipment. These systems should be fully separated from process control systems, and using separate hardware is recommended where feasible. A PCS domain controller is necessary at this level to authenticate and authorize users and apply group policies to further limit access to equipment and applications. It is recommended that the domain controller at this level be a read-only domain controller reading from the domain controller in the SCADA network layer. Within this layered architecture, the DMZ is the first layer of protection for the PCS network and the most vulnerable area of the PCS network. Equipment located in this layer and the functions and applications available in this layer should be carefully selected to ensure critical system components are not compromised.

The next higher trusted level in the layered architecture is the SCADA network. This is the main operator access level into the overall water sector PCS. Within this layer are components, such as SCADA servers, operator workstations, terminal servers (supporting local SCADA users only), and printers to support the main HMI structure of the PCS. It may also include other subsystem devices, segregated on separate VLANs and filtered by the firewall, such as physical security servers and VoIP equipment. The SCADA servers are the central devices in this layer, requiring communications to operator interfaces and communications to the PLC network and DMZ to support process control and remote system monitoring and alarming. A domain controller is required at this layer to support authentication and authorization of SCADA system users. This domain controller is the primary device controlling where user accounts and groups are added and maintained for the SCADA system. It should not support other systems, such as the security system. This layer does not have direct communications with outside business networks or the Internet, as it is buffered by the DMZ. The SCADA network level is a critical network for operator access and process control but is also often required to communicate to devices in less trusted layers in order to implement remote functions. Communications between layers should be limited to the SCADA servers and domain controllers to minimize available routes across layers. Domain controller communications should be limited to an IPSec tunnel to the read-only domain controller in the DMZ.

The most trusted level in the layered approach is the PCS PLC network, sometimes referred to as the plant floor. This level is the most critical layer in the network and contains the components required for process control and system safety. Ethernet protocols are increasingly being used at this layer, but this layer may include other digital communications protocols, such as Modbus RTU, DeviceNet, ControlNet, Profibus, and Foundation Fieldbus, which can be treated as separate sublayers within the overall system. Within this layer are PLCs to control processes and often additional Ethernet-connected devices used within the process control system, such as variable frequency drives, motor control centers, local operator interface units, digital power meters, and instruments. A separate physical layer should be provided to support other end devices located in the field for the security and VoIP systems, if necessary, and these systems need to remain segregated from process systems using separate VLANs and firewall rules. Communications between PLCs and process control components within this layer are critical for proper operation and safety of water and wastewater treatment plants. These systems must be online and operational. By locating critical components within the most trusted layer, they are more difficult to access since multiple other layers need to be compromised first. Process control communications should be given the highest quality of service priority within this level.

Using a layered approach to network architecture provides multiple levels of protection for critical process components, and the organization and security of this architecture can be further refined through additional network segmentation for a complete defense-in-depth strategy.

Logical segmentation enhances system performance

A multilevel network architecture has multiple layers of protection for critical process systems and is a critical component of the defense-in-depth protection strategy. Network organization and segmentation provide additional network security, improve network performance and reliability, and help maintain the network. Networks can be organized using appropriate subnets and VLANs, where all devices on each subnet are members of the same VLAN. Both VLANs and subnets define a broadcast domain, which significantly reduces network traffic by reducing the number of devices that receive each address resolution protocol broadcast. VLANs allow segmentation at Layer 2 where subnets are a Layer 3 construct, so pairing VLANs with subnets allows subnet segmentation to be extended across a Layer 2 LAN environment. This improves network security and further reduces broadcast domains. In this way, overall network costs can be reduced by allowing different VLANs, i.e., separate networks, to reside on the same Layer 2 devices and share communication media (Layer 1).

To begin organizing the network addressing, a logical IP addressing strategy tailored to the applications is required to ease network management and support, as well as to allow easy network expansion, upgrade, and adaptation to changing needs. The Internet Assigned Numbers Authority (IANA) has developed guidelines for private, public, and reserved IP address ranges through a series of request for comments (RFCs), which are managed by the Internet Engineering Task Force. For organizing a PCS network, the private IP address ranges, listed in table 3, are of the most interest and are published in RFC 1918. These addresses are based on IPv4, which uses 32-bit IP addresses. The bit block refers to the number of bits, or IP addresses, available for use within the network, and the class slash ( / ) notation refers to the number of static bits used to define the subnet.

RFC 1918 Name Address range Network class
24-bit block 10.0.0.0 – 10.255.255.255 Class A (10/8 prefix)
20-bit block 172.16.0.0 – 172.31.255.255 Class B (172.16/12 prefix)
16-bit block 192.168.0.0 – 192.168.255.255 Class C (192.168/16 prefix)

Table 3. Address allocation for private Internets

These addresses can then be used without any coordination with the IANA or an Internet registry. Organization within the water sector PCS network should be selected based on the size and topology of the PCS network as well as existing enterprise-wide policies and procedures. A small PCS may not require large-class networks. The following is an example recommendation for a large utility or municipal network:

  • 24-bit block: Devices and equipment within a treatment plant or facility
  • 20-bit block: Used for connections on private networks between facilities, such as metro-Ethernet, or wireless links such as microwave or frequency-hopping spread spectrum radios.
  • 16-bit block: Wide-area-network- (WAN) or metropolitan-area-network-connected devices, such as connections on city- or county-wide networks where the network may be shared with other users
  • For large networks, particularly when the business network and the PCS network share resources or network infrastructure, the PCS IP addresses should be carefully coordinated with network administrators to ensure addresses are not duplicated within a private network. By following a standard methodology for IP addressing within a private network, it becomes simpler to prevent address duplication and to manage the different PCS networks. The steps involved in developing the IP addressing scheme include:
    • Making a list of primary network segments
    • Laying out a rough network topology with Layer 3 interfaces separating primary network segments
    • Developing an initial IP numbering strategy
    • Creating a strategy and continuing to add detail and make adjustments as required
    • Ensuring that the scheme has the capacity and flexibility to accommodate expansion and to adapt to the ever-changing network environment

For each subnet assigned on the network, a VLAN number and description needs to be assigned. The default or native VLAN is VLAN 1. IP addresses not assigned to any other VLAN will be assigned to this VLAN, making it subject to intrusion. For this reason, it is generally recommended to avoid use of VLAN 1 to enhance network management and security. The following is a recommended approach for the use of VLANs within a PCS network:

  • Use VLANs in the range of 2-1001; various restrictions apply to other VLANs
  • Do not use VLAN 1 (default or native VLAN)
  • Use devices that support IEEE 802.1q VLAN Encapsulation Protocol
  • Use a logical approach in VLAN number selection to support network management
  • Incorporate VLAN numbers into IP addresses

By using the recommended approach, VLANs can provide network segmentation and aid in network management and organization. Combining a well-thought-out VLAN approach into an IP addressing plan can improve network management, device IP address assignments and identification, and security. An example approach to combining VLANs into an IP addressing scheme is shown in figure 3.

Figure 3. IP addressing example

The major components of the IP addressing example in figure 3 are the facility and VLAN numbers. Facility numbers can generally be selected by actual facility numbers or unit process numbers associated with a facility to make the IP address a usable and recognizable number. VLAN numbers can be anything in the allowable VLAN range; however, the following approach outlines a method of selecting VLAN numbers to aid in equipment identification and to aid in trust level identification and firewall configuration:

  • Organize VLAN numbers in a similar order as trust levels (e.g., VLAN 10 is most trusted, and VLAN 900 is least trusted).
  • Separate VLAN numbers selected to allow for future growth and make addresses more distinguishable. Use VLAN numbers such as 10, 20, and 40.

Figure 4 shows an example VLAN selection scheme based on the recommended approach for selecting VLAN numbers and a network topology similar to that shown in figure 2. In this example, VLANs start with 10, with the highest trust level being the network management VLAN. This VLAN would be solely used by the network administrator to use network monitoring and management software and would have access to all equipment on the network. The VLAN order then closely follows the layered architecture shown in figure 2, where the PLC network was the most trusted network and the utility WAN was the least trusted layer in the example architecture. Additional VLANs can be assigned as required for other systems and networks to meet the requirements of the PCS. By completing a detailed network assessment and devising a network security strategy, a successful network segmentation scheme can be planned and designed to allow for logical segmentation of water sector PCS networks. This logical segmentation can aid in network organization and identification of networks, locations, and components with a PCS network and can be used in conjunction with a layered network architecture and security rules for an organized approach to overall system cybersecurity.

Figure 4: VLAN selection example

Coordinating network architecture and segmentation for a complete solution

Adding VLANs as part of the network segmentation plan for a multilayered PCS network architecture provides additional security, lowers network utilization, and makes network management simpler. Combining these solutions helps to eliminate unnecessary routes through the layers and reduces network traffic, which improves both network security and optimization. Figure 5 shows an example water sector PCS control center LAN architecture utilizing the network segmentation approach previously presented.

Figure 5. Multilayered PCS network architecture with segmentation

Figure 5 is an example of a typical PCS network with remote access and communications to other remote PCS control rooms that are physically separated to allow coordination across the water sector utility. Figure 5 is an example for a PCS network for remote pump stations. In this case, the pump stations have PLCs located remotely throughout a municipality that must be polled and data reported to a central SCADA system for monitoring and control. In this example, the local network is broken into multiple VLANs as shown in figure 6.

Figure 6. VLAN assignments

As shown, multiple VLANs reside at Layer 2 and can communicate with other components on the same VLAN using only Layer 2 devices, thereby allowing separation of different network components but allowing communications using less expensive Layer 2 devices for the PCS LAN. In this example, the SCADA network is the most trusted network, because it is located in the innermost layer of the network. The PCS PLCs are the next most trusted component in the network since they poll data from remote pump stations via point-to-point VPNs with data encryption. This is reversed from the network architecture in the example of figure 2, showing that network assessment and planning are necessary to identify the critical components of a network and that water sector PCS networks are unique. They require customized solutions that follow a standardized practice. Continuing with this example, VLANs 10, 11, 20, 30, and 40 reside on the local network and are trunked at the local firewall for the network. This firewall provides both routing and access control lists that govern communication between the VLANs on the network. Additional communications routes are also provided to a remote control room. These communications consist of a primary route through the utility/municipality network, as well as disaster recovery communications through a private metro-Ethernet network. The configuration of the VLANs for each of these networks simplifies the allowed communications between these networks.

As noted previously, communications between PLCs are done through remote VPN connections, but PLCs are still all on the same VLAN to allow direct communication between PLCs without the need for a specific route to be established. Figure 7 shows the remote VPN connections needed for remote device communications. Note that only VPN connections are necessary and that additional routes do not need to be established since devices that need to communicate with each other are on the same VLAN.

Figure 7. Remote connections using VPNs

By combining a layered architecture with logical network segmentation, network organization and remote communications are simplified. The effort needed to establish communication routes and access control lists is simplified making network configuration and management easier. Figure 8 summarizes the VLAN and subnet organization for the example architecture. (click on image to see a larger version)

Figure 8. Example IP addressing scheme (click on image to see a larger version)

Configuration and management

One of the main features of a logical segmentation plan within a multilayered network architecture is a simplified approach to firewall configuration, thereby making network security and routing management simpler. Using VLANs to coordinate similar equipment with similar access rights and trust levels makes routing and access control list configuration for communications between subnets on the network easier. Communications routes between networks for devices added to a given VLAN are then already in place. Network expansion is then simplified because new routes and rules within network security and routing appliances do not need to be added or revised each time a piece of equipment is added to the network. Figure 9 shows an example firewall trust level configuration for the network presented in figure 5. This example exhibits how trust levels can be defined simply for large groups of network-connected equipment based on VLAN assignments.

Figure 9. Firewall trust levels

Figure 9 depicts the trust levels for the various VLANs and notes global rules for each VLAN, such as what other networks or VLANs each trust level is allowed to access. To provide a complete security configuration, additional access control rules are necessary to further define allowed communications between networks. Using VLANs with associated trust levels simplifies global rules, which make implementation and management of the network simpler. Figure 10 is an example of firewall rules and allowed VPN tunnels for the example shown in figure 5 and trust levels summarized in figure 9. Figure 10 is a summarized form of the rules to be implemented in an actual firewall. It shows how organizing a network into subnetworks using VLANs can greatly simplify the rules implemented in an actual firewall.

Figure 10. Firewall access control rules

As shown in figure 10, each set of rules is defined by VLAN, or could be defined by subnet, but not by each device or each specific IP address. Having an organized network approach then allows for global definitions of access control lists and global management of devices within groups in lieu of having to manage each device separately. By combining similar devices into groups by planning and organizing the network, network configuration and management is simplified.

Conclusion

The approach for a multilayered water sector PCS using VLAN segmentation for subnetworks provides a foundation for which secure PCSs can be developed. By using the strategies and tools such as the four-phase process of assessment, design, implementation, and procedures/maintenance, a water sector PCS network can be customized to provide a secure and manageable PCS network. As with any system, planning and design must be carefully coordinated to ensure components are located within the correct layer of the architecture and that the proper firewall rules and access control lists are implemented. By using VLANs, this segmentation can be extended across a Layer 2 LAN. The advantage of the approach presented is that it allows for simpler configuration of network security appliances and for simpler management and expansion of the network, leading to increased network availability and a reduction in threat risk as part of a comprehensive defense-in-depth strategy.

List of acronyms

ACL Access control list
CERT Cyber Emergency Response Team
COTS Commercial-off-the-shelf
DMZ. Demilitarized zone
HMI Human-machine interface
ICS Industrial control systems
IP Internet Protocol
IT Information technology
MCC. Motor control center
NIST National Institute of Standards and Technology
PCS. Process control system
PLC Programmable logic controller
SCADA Supervisory control and data acquisition
VFD Variable frequency drive
VLAN Virtual local area network
VPN Virtual private network
VoIP Voice over IP

References

Water Sector Coordinating Council Cybersecurity Working Group. Roadmap to Secure Control Systems in the Water Sector. AWWA and DHS, March 2008.

United States Government Accountability Office. Critical Infrastructure Protection – Cybersecurity Guidance is Available, but More Can Be Done to Promote Its Use. GAO-12-92, December 2011.

Repository for Industrial Control System (RISI) Incident Report, March 2010.

CERT, February 2009. Retrieved March 2013.

Government Security News, March 2013.

Security for Industrial Automation and Control Systems Part 1: Terminology, Concepts, and Models.www.isa.org/link/ANSI/ISA-62443-1-1

Security for Industrial Automation and Control Systems: Establishing an Industrial Automation and Control Systems Security Program.www.isa.org/link/ANSI/ISA-62443-2-1

Eric Byres. Revealing Network Threats, Fears – How to use ANSI/ISA-99 standards to improve control system security.www.isa.org/link/networkthreats

Standards NIST 2013, Guide to Industrial Control Systems (ICS) Security. NIST Special Publication 800-82 (Revision 1), May 2013.

Paul Didier, et al. Converged Plantwide Ethernet (CPwE) Design and Implementation Guide. Cisco and Rockwell Automation, September 2011.

Recommended Practice: Improving Industrial Control Systems Cybersecurity with Defense-In-Depth Strategies. Department of Homeland Security, October 2009.

Musaria Mahmood and Fawzi Al-Naima. Developing a Multi-Layer Strategy for Securing Control Systems of Oil Refineries. Wireless Sensor Network, Volume 2, pp. 520-527, July 2010.

Scott Hillman. Physical Security 101: Evolving “defense in depth” – What good is cybersecurity if the bad guys can walk through the front door?www.isa.org/link/physicalsecurity101

Bela Genge and Christos Siaterlis. An Experimental Study on the Impact of Network Segmentation to the Resilience of Physical Processes. Networking 2012, Lecture Notes in Computer Science, Volume 7289, pp. 121-134, 2012.

(1)A honeypot is a trap designed to look like a real functioning computer network, but is actually isolated and monitored, and used in this case to look like a SCADA system to detect and research attacks on water sector ICS.

About the Authors
Norman Anderson, P.E., has more than six years of experience in the design and commissioning of process control systems and security systems for the water sector. Norman has provided secure and reliable PLC, SCADA and network hardware and software architecture designs and provided control system automation solutions for a range of facilities. He has an M.S. in electrical engineering from Iowa State University and an M.S. in physics from the University of Florida. Contact Norman at norman.anderson@ch2m.com.

Bill Phillips, P.E., specializes in the delivery of secure and reliable process control and SCADA network and communications systems, cybersecurity vulnerability assessment and facility automation and information system planning and implementation. Bill has more than 30 years of process control and SCADA system experience and has focused on control system network and communications cybersecurity for the last decade. He has a B.S. in electrical engineering from Clemson University. Contact Bill at bill.phillips@ch2m.com.

A version of this article originally was published at InTech magazine.

Gaining control of effluent residual (Part 2)

Gaining control of effluent residual (Part 2)

This guest post is authored by Narciso Santiago, a project engineer at EMA, Inc. in Orlando, Fla.  Narciso was a presenter at the 2013 ISA Water/Wastewater Symposium.

In Part 1 of this post,  I explained the difference between being “in control” versus being “out of control” of effluent residual, and indicated that unless the process has long and variable dead time advance process control technology is not a requirement for gaining control of effluent residual. The consequences of being “in” or “out” of control were explained and several important factors for effective residual control were mentioned.Waste Water Treatment

In this post, a list of important factors is given and specific trim controller configuration is discussed. Before gaining control of effluent residual, these key factors must be addressed. The following checklist will help you determine which factors will enable the ability to gain automatic control of effluent residual.

1. Chlorine feed system working correctly. Metering pumps or chlorinators working correctly.

2. Chlorine feed flowmeters (if any) are working correctly for the entire operating range and are calibrated properly. If the flowmeter is unable to give accurate measurement during low flow conditions it should not be utilized for flow control, and the metering pump (or chlorinator) must be controlled by volumetric command (%).

3. Chlorine analyzer is installed according to manufacturer specifications. Sample transport time should be less than 30 seconds whenever possible. Sample lines should be cleaned and flushed on a periodic basis based on experience.

4. All chlorine analyzers require periodic maintenance. Implement a maintenance program (if not already in place) to follow the manufacturer’s maintenance recommendations. Perform conformance sampling by taking a grab sample, running a standard laboratory analysis, and comparing the answers on a periodic basis based on experience.

5. If redundant analyzers are utilized, perform redundancy validation on a periodic basis based on experience. Although not required to gain control of effluent residual, analyzer redundancy is recommended to increase reliability.

6. Mixing done at or near the point of chemical injection. Poor mixing may result in a non-representative of the stream as a whole. Achieving good control without a representative sample is not possible.

7. The control measuring point is near point of injection. The immediate chlorine demand must be satisfied to gain control of effluent residual. The residual measurement used for control must be taken shortly after chlorine addition and mixing.

8. Process dead time. Best residual control is achieved with small process dead time. To measure process dead time while the process is in a steady state and control set to flow pace mode only, increase the dosage setpoint (large increase) and record the time it takes the measured residual start to increase.  When performing a test to determine the process dead time, a historical trend is recommended to be able to also determine the system time constant, the CV delta change (%) and the PV delta change (%). These data are required to calculate initial tuning parameters. Note that advance process control (APC) technology may be required to gain control of a process with long and variable process dead time.

9. Current control algorithm. Knowing the control algorithm enables operators to determine if the control system is working correctly. The fundamental control algorithm is simply a flow pace formula multiplied by a correction factor. [Feed Rate = Flow Pace Rate * (Trim Controller Output)].  The “flow pace rate” portion of the formula calculates a desired feed rate based on dosage setpoint. The “trim controller output” portion of the formula calculates a correction factor based on the error between the setpoint and the residual measured.

10. Control strategy. Is the influent residual measured used for control? In automatic control, is the operator able to set the automatic control for flow pace only and also able to set the system to flow pace plus trim control? Is the operator able to change the dosage (mg/l)? If using bleach, is the operator able to change concentration (%) and specific weight (lbs./gal)?

Once all items in the checklist above have been checked off, the trim controller PID must be configured and initial tuning parameters calculated.

The initial cycle time (time between executions, also known as task period) must be set to equal or greater than twice the influent residual analyzer sampling time. This means that the trim controller output, which is also the correction factor, updates once every time the cycle time elapses. The scale factor shown provides a bump-less transfer to trim mode functionality (Figure 1).

Figure 1, Trim Controller

Figure 1, Trim Controller

A process response graph is shown in Figure 2 as an example. The actual process response trend and initial tuning parameters are determined utilizing the data collected during the process dead time test, checklist item No. 8.

Figure 2, Process Response Graph

Figure 2, Process Response Graph

The process gain:
G = (delta PV %) / (delta CV %)
System time constant:
T = (t1 – t2) * 1.5
Process dead time = t0
Initial PID parameters:
K = 1.5T/Gt0
Ti = 2.5t0
Td = 0.4t0

Set the initial dead band to 0.5PPM, and turn off the PID zero crossing functionality. Since chlorine addition based on effluent residual measured will be out of phase with incoming flow due to detention time, the immediate chlorine demand (shortly after adding it) must be satisfied to gain control of effluent residual.

For effluent processes without excessive dead time, implementing the fundamental considerations discussed in part 1 of this post, addressing the important factors outlined in the checklist above, and proper configuration of the trim controller result in gaining control of effluent residual.

What a historical trend of effluent residual at your plant would depict? The residual measured consistently tracking the setpoint? Or a residual measured not tracking the setpoint? A 24 hour historical trend of the effluent residual will answer the question.

Narciso Santiago

About the Author
Narciso Santiago, CAP, is a project engineer at EMA, Inc. in Orlando, Fla., a company dedicated to the design, development, and implementation of control systems and automation for water and wastewater facilities. Prior to joining EMA, Narciso spent over 12 years at Gencor Industries, a manufacturer of asphalt plants, soil remediation plants, combustion systems and control systems. Narciso is an adjunct instructor for the School of Electronics Technology at ITT Technical Institute in Lake Mary, Fla. He earned a bachelor’s degree in electronics engineering technology from the University of Puerto Rico, and an associate of instrumentation engineering technology from the Technological Institute of Puerto Rico. Narciso has been a certified automation professional since April 2009. Contact Narciso at: nsantiago@ema-inc.com.
Gaining control of effluent residual (Part 1)

Gaining control of effluent residual (Part 1)

This guest post is authored by Narciso Santiago, a project engineer at EMA, Inc. in Orlando, Fla.  Narciso was a presenter at the 2013 ISA Water/Wastewater Symposium.

Many times I have walked into different wastewater facilities control rooms and looked at the effluent residual trend, which to me looks like it is out of control. When I ask for the first time how the effluent residual control is performing, most likely I will get the same answer, it’s doing ok.Effluent Residual

But in reality, there is a big difference between a trend depicting “in control” of effluent residual (Figure 1, below) and a trend depicting “no control” (Figure 2, below). Good control can simply be determined by the fact that the process variable will track the setpoint (in this case the residual measured). A lack of understanding of effluent residual control fundamental considerations results in higher operational and chemical costs.

A trend depicting a small deviation from setpoint demonstrates excellent process tuning. But in wastewater effluent facilities not having good control of residual results in chemical waste, and therefore, gaining effluent residual control is more important than having an excellent process tuning. The immediate result of gaining control of effluent residual is chemical savings.

Figure 1

Figure 1, in control

Figure 2

Figure 2, no control

Advance process control (APC) technologies can provide ultimate optimization for effluent residual control and should be utilized when process conditions demand process modeling and predicting capabilities such as processes with long and variable dead times. But since not all programmable logic controllers are APC capable, and gaining control of the effluent residual is more important than optimizing process tuning, APC technology is not a requirement to gain control of effluent residual.

Important factors for effective effluent residual control include correctly sized control elements; installation and maintenance practices; types of chlorine analyzers; and analyzer redundancy. But more important factors are fundamental considerations that include mixing at the point of injection and how it affects the flow stream; the measuring point location and why it matters; the process dead time and how it relates to choosing a more advanced control method or not; the processing algorithm which is an industry standard; and how a correction factor based on error between setpoint and process variable is implemented.

When there is no effective control of effluent residual, operational activities include time spent daily verifying the effluent residual will not go out of compliance, and taking multiple samples daily to determine the actual value of the effluent residual. Also, the operator is more likely to set the system to manual control, and set the chemical feed rate for enough chemical feed to guarantee the effluent residual will not go out of compliance. The result of these activities will show up in the effluent residual trend depicting a process variable not tracking the setpoint. Worse, a residual trend indicating the residual measured at full scale does not represent the actual value of the residual measured; it’s higher.

In contrast, when the effluent residual is under control, the time spent verifying if the residual is in compliance is now a matter of checking the influent and effluent residual trends for consistency; and a dosage adjustment is now based on the knowledge of actual process disturbances.

In wastewater facilities with no effective control of effluent residual, addressing the important factors and implementing fundamental considerations mentioned above results in gaining control of effluent residual and reduction of operational time and chemical consumption. Upgrading the programmable logic controller and software is most likely not required. Implementation and initial tuning can be completed in a few days. Additional tuning time may be required, but the overall implementation cost will most likely prove to be affordable with immediate cost savings in chemicals.

In Part 2 of this post, I will offer a checklist to help you assess factors that must be considered to achieve automatic control of effluent residual.

Narciso Santiago

About the Author
Narciso Santiago, CAP, is a project engineer at EMA, Inc. in Orlando, Fla., a company dedicated to the design, development, and implementation of control systems and automation for water and wastewater facilities. Prior to joining EMA, Narciso spent over 12 years at Gencor Industries, a manufacturer of asphalt plants, soil remediation plants, combustion systems and control systems. Narciso is an adjunct instructor for the School of Electronics Technology at ITT Technical Institute in Lake Mary, Fla. He earned a bachelor’s degree in electronics engineering technology from the University of Puerto Rico, and an associate of instrumentation engineering technology from the Technological Institute of Puerto Rico. Narciso has been a certified automation professional since April 2009. Contact Narciso at: nsantiago@ema-inc.com.
Using smarter technology to solve the municipal water conundrum

Using smarter technology to solve the municipal water conundrum

This guest post is authored by Carey Hidaka, an IBM worldwide client solutions professional.  Carey was the keynote speaker at the 2013 ISA Water/Wastewater Symposium.

Do you think about where your water comes from and if it will be there tomorrow?  Do you think about where water goes after you’ve used it and how it’s reused as a part of the earth’s water cycle?  Did you know that the amount of fresh water on earth is about the same now as when the dinosaurs walked the planet, and that more and more people are using it now at an ever faster rate?Water in glass

Clean, fresh water is a basic human need that is rapidly becoming a scarce resource.  South Africa even states that access to “sufficient water” is a constitutional right.

Water isn’t just for personal consumption.  Seventy percent of the world’s fresh water is used for irrigation, producing about 40 percent of the world’s food.  Water also generates the world’s electricity by producing steam, powering turbines, cooling equipment and conveying power generation by-products.  Our way of life is inextricably tied to water in many ways.

Population growth stresses our fresh water supplies.  According to the United Nations, water use has been growing at more than twice the rate of population growth in the last century.

Unchecked urbanization and industrialization are contaminating the fresh water we have, making the problem even worse.  In rapidly industrialized China in 2012, up to 40 percent of the rivers were seriously polluted and 20 percent were so bad that their water quality was rated too toxic for human contact.

And the infrastructure that delivers clean water and collects and treats dirty water is falling apart.  The American Society of Civil Engineers in 2009 assigned a report card GPA of D to the U.S. infrastructure, with drinking water and wastewater both achieving grades of D-.  This “improved” in a 2013 update to grades of D.  The World Health Organization estimates that by 2050 70 percent of the world’s population will live in cities; this will further stress an already fragile infrastructure.

Water is one of the most abundant resources on earth, but usable fresh water only makes up less than 1 percent of the total.  Unless we do something differently about how we source clean water and treat wastewater, there won’t be enough fresh water for current and future generations.

Cities don’t have the funding to repair disintegrating infrastructure, install new infrastructure and desalinate to produce enough “new” fresh water to avoid the impending crisis.  Traditional engineering solutions alone can’t solve these problems.

The Way Forward Starts With Data

Lots of data is generated when clean water is produced and wastewater is collected and treated.  How can we analyze that data and combine it with new data sources like crowd sourcing and social media, to create new insights and make better and timelier decisions about water?

For example, the City of South Bend, Indiana is using data from its wastewater collection system to help eliminate combined sewer overflows that contaminate the St. Joseph River with untreated wastewater.  This saves the city from spending millions of dollars in traditional engineering solutions (sewer separation, storage facilities, etc.).

Miami-Dade Parks, Recreation and Opens Spaces is using water meter data to determine where water is leaking and being wasted, reducing water consumption by 20 percent and saving $860,000 per year.

Data doesn’t create more fresh water, but it feeds analysis and decision making about where to source water, when to desalinate, how to stop pollution and what pipes to fix before they break.

What Can We Do?

It’s been said that the world’s next wars will be fought over water, not oil.  How do we take a scarce resource like water and use it in the wisest way possible to support life?  With today’s new technologies, the economic uncertainties in our cities, the challenges of drought, climate change, and failing infrastructure, what else can we be doing?  In what ways can we use data to make better decisions?  What kinds of problems can we solve?  Can we use data and technology to unleash innovation and creativity to leap from reacting to problems to proactively solving them?

OLYMPUS DIGITAL CAMERA

About the Author
Carey Hidaka is an IBM worldwide client solutions professional in the IBM Software Group, with experience as a consultant in the smarter water management business development team and mobile/wireless services. He has 29 years of information technology experience and practiced for nine years as a consulting environmental engineer and registered professional engineer where he focused on water resource planning and water and wastewater treatment plant designs and implementations for public and industrial sector clients. Carey has a master’s degree in business administration with finance specialization from the University of Chicago, a master’s degree in environmental engineering from the University of Illinois at Urbana-Champaign, and a bachelor’s degree, with honors, in civil engineering from the University of Colorado at Boulder. He is also a member of the American Water Works Association.

Pin It on Pinterest