ACM CCS Conference Tutorial Nov.

2009

Cyber Security for the Power Grid: Cyber Security Issues & Securing Control Systems
Andrew Wright
Cyber Security Solutions For <Client Name>

CTO, N-Dimension

andrew.wright@n-dimension.com

Power Grid Communications & Control Systems

Internet

Control Systems

borrowed from NIST Smart Grid Twiki

Agenda
High-Level
Industrial Control Systems and Cyber Security Issues Securing Control Systems

Detailed
Security Issues in Industrial Control Systems Todays Threats Securing Control Systems

A Control System
Sensor(s) + Actuator(s) + Controller(s)

Types of Industrial Control Systems (ICS)

Supervisory Control And Data Acquisition (SCADA) Process Control Systems (PCS)

Distributed Control Systems (DCS)

Automation

Historical ICS
Proprietary Complete vertical solutions Customized Specialized communications
Wired, fiber, microwave, dialup, serial, etc. 100s of different protocols Slow; e.g. 1200 baud

Long service lifetimes: 1520 years Not designed with security in mind

Modern ICS Trends

Enterprise Network
Workplaces
Firewall

Internet

IP

Enterprise Optimization Suite

Enterprise Network
Third Party Application Server

Mobile Operator

Services Network
Connectivity Server Historian Server Application Server Engineering Workplace

Control Network
Serial, OPC or Fieldbus

Redundant

Device Network
Third Party Controllers, Servers, etc.

Serial

RS485

Technology Trends in ICS

COTS (Commercial-Off-The-Shelf) technologies
Operating systemsWindows, WinCE, embedded RTOSes ApplicationsDatabases, web servers, web browsers, etc. IT protocolsHTTP, SMTP, FTP, DCOM, XML, SNMP, etc. Networking equipmentswitches, routers, firewalls, etc.

Connectivity of ICS to enterprise LAN

Improved business visibility, business process efficiency Remote access to control center and field devices

IP Networking
Common in higher level networks, gaining in lower levels Many legacy protocols wrapped in TCP or UDP Most new industrial devices have Ethernet ports Most new ICS architectures are IP-based

New IP-Based Industrial Control Systems

ODVA (Rockwell) Profinet Foundation Fieldbus HSE Telvent ABB 800xA Honeywell Experion Emerson DeltaV Yokogawa VNET/IP Invensys Infusion Survalent

IP to the Control Network or even Device Network Not all are fully compatible with ordinary IP

Security Risks to Modern ICS

COTS + IP + connectivity = many security risks All of those of Enterprise networks and more
Worms and Viruses DOS and DDOS impairing availability Unauthorized access Unknown access Unpatched systems Little or no use of anti-virus Limited use of host-based firewalls Improper use of ICS workstations Unauthorized applications Unnecessary applications Open FTP, Telnet, SNMP, HTML ports Fragile control devices Network scans by IT staff Legacy OSes and applications Inability to limit access Inability to revoke access Unexamined system logs Accidental misconfiguration Improperly secured devices Improperly secured wireless Unencrypted links to remote sites Passwords sent in clear text Default passwords Password management problems Default OS security configurations Unpatched routers / switches

When ICS Security Fails

Loss of production Penalties Lawsuits Loss of public trust Loss of market value Physical damage Environmental damage Injury Loss of life

USSR pipeline explosion, 1982 Bellingham pipeline rupture, 1999 Queensland sewage release, 2000 Davis Besse nuclear plant infection, 2003 Northeast USA blackout, 2003 Browns Ferry nuclear plant scram, 2006

$$$.$$

So How Do We Secure Industrial Control Systems?

ACM CCS Tutorial Nov. 2009

There is No Silver Bullet!

No Silver Bullet!

Defense in Depth
Perimeter Protection
Firewall, IPS, VPN, AV Host IDS, Host AV DMZ

Interior Security
Firewall, IDS, VPN, AV Host IDS, Host AV IEEE P1711 (AGA 12) NAC IDS IPS Scanning
DMZ VPN AV NAC

Monitoring Management

Intrusion Detection System Intrusion Prevention System DeMilitarized Zone Virtual Private Network (cryptographic) Anti-Virus (anti-malware) Network Admission Control

50000 Foot View

Internet
IT Stuff

Enterprise Network

IT Stuff

Scan IDS NAC FW IDS FW

VPN FW Proxy AV IPS Log Mgmt Host IPS Host AV

Control Network
VPN AV

Event Mgmt Reporting

Host IDS Host AV

Partner 62351 Site

VPN

FW

IPS

P1711

Scan Field Site NAC

Field Site

Field Site

Security Issues in Industrial Control Systems

ACM CCS Tutorial Nov. 2009

Availability, Integrity and Confidentiality

Enterprise networks require C-I-A
Confidentiality of intellectual property matters most

ICS requires A-I-C

Availability and integrity of control matters most control data has low entropylittle need for confidentiality Many ICS vendors provide six 9s of availability

Ensuring availability is hard

Cryptography does not help (directly) DOS protection, rate limiting, resource management, QoS, redundancy, robust hardware with high MTBF

Security must not reduce availability!

DoS and DDoS Attacks

Denial of Service (DoS) attack overwhelms a system with too many packets/requests
Exhausts TCP stack or application resources Defenses include connection limits in firewall

Distributed Denial of Service (DDoS) attack coordinates a botnet to overwhelm a target system
No single point of attack Requires sophisticated, coordinated defenses Weapon of choice for hackers, hacktivists, cyber-extortionists

DoS, DDoS particularly effective when Availability is critical, i.e. against ICS

Fragile ICS Devices

Many IP stack implementations are fragile
Some devices lockup on ping sweep or NMAP scan Numerous incidents of ICS shut down by uninformed IT staff running a well-intentioned vulnerability scan

Modern ICS devices are much more complex

Some IEDs include web server for configuration and status More lines of code leads to more bugs Modern IEDs require patching just like servers

Unpatched Systems
Many ICS systems are not patched current
Particularly Windows servers No patches available for older versions of windows

OS and application patches can break ICS

OS patches are tested for enterprise apps

Uncertified patches can invalidate warranty Patching often requires system reboot Before installation of a patch:
Vendor certificationtypically one week Lab testing by operator Staged deployment on less critical systems first Avoid interrupting any critical process phases

Limited use of Host Anti-Virus

AV operations can cause significant system disruption at inopportune times
3am is no better than any other time for a full disk scan on a system that operates 24x7x365

ICS vendors only beginning to support anti-virus

Anti-virus is only as good as the signature set Signatures may require testing just like patches

AV may be losing ground in enterprise deployments

impact on hosts, endpoint security not getting better virus writers have learned to test against dominant AV

application whitelisting can be a good alternative

enumerate goodness rather than badness

Poor Authentication and Authorization

Machine-to-machine comms involve no user Many ICS have poor authentication mechanisms and very limited authorization mechanisms Many protocols use cleartext passwords Many ICS devices lack crypto support Sometimes passwords left at vendor default Device passwords are hard to manage appropriately
Often one password is shared amongst all devices and all users and seldom if ever changed This is happening AGAIN in Smart Meter deployments!

Poor Audit and Logging

Many ICS have poor or non-existent support for logging security-related actions
Attempted or successful intrusions may go unnoticed

Where IDS logs are kept, they are often not reviewed Various regulatory requirements are driving some change in this area
NERCNorth American Electric Reliability Corporation FERCFederal Energy Regulatory Commission Sarbanes Oxley and PCAOB (Public Company Accounting Oversight Board) FISMAFederal Information Security Management Act

Unmanned Field Sites

Many unmanned field sites Many with dialup access Some with high-speed connectivity to control center Most with poor authentication and authorization backdoor to the control center!

Legacy Equipment
Much legacy equipment Usually impossible to update to add security features Difficult to protect legacy communications
but see IEEE P1711 for serial encryption

Password protection is weak Little or no audit and logging

Unauthorized Applications
Unauthorized apps installed on ICS systems can interfere with ICS operation Many types of unauthorized apps have been found during security audits
Instant messaging P2P file sharing DVD and MPEG video players Games, including Internet-based Web browsers

Inappropriate Use of ICS Desktops

Web browsing from HMI can infect ICS
Browser vulnerabilities Downloads Cross-site scripting Spyware

Email to/from control servers can infect ICS

Sendmail and outlook vulnerabilities

Disk storage exhaustion can crash OS

Storage of music, videos

Little or No Cyber Security Monitoring

internal monitoring is essential to detect low profile compromises
IDS port scanning vulnerability scanning system audit

without internal monitoring dont know whether systems have been compromised

Requirement for 3rd Party Access

Firmware updates and PLC, IED programming are sometimes done by vendor
Many ICS have open maintenance ports Infected vendor laptops can bring down ICS

Partners may require continuous status information

Partner access is often poorly secured Partner channels can serve as backdoors

3rd parties may include:

ISO, transmission provider or grid neighbor, equipment vendor, emissions monitoring service or agency, water level monitoring agency, vibration monitoring service, etc.

People Issues
ICS network often managed by Control Systems Department, distinct from IT Department running enterprise network
ICS personnel are not IT or networking experts IT personnel are not ICS experts

Majority of control systems workforce is older and nearing retirement

Few young people entering this field Few academic programs

Harsh Environments

Temperature Vibration Dust Humidity Electrical Transients

Attack Vectors into Control Systems

Includes Infected Laptops and Is Growing

Source: 20032006 data from Eric Byres, BCIT

Security Assessments on ICS

Various groups perform security assessments and penetration tests on ICS (generally under NDA)
Idaho National Labs Sandia National Labs N-Dimension Solutions Other private organizations

Vulnerability assessments always uncover problems For penetration tests, we always get in
Not a question of if, but how long

Other Issues
Unusual physical topologies Many special purpose, limited function devices Static network configurations Multicast Long service lifetimes

For More Information ...

See Smart Grid Cyber Security Strategy and Requirements, NISTIR 7628, www.nist.gov/smartgrid
particularly Appendices C and D

Todays Threats

ACM CCS Tutorial Nov. 2009

Intense Media Visibility on the Cyber Security Issue

Smart Grid Security Frenzy: Cyber War Games, Worms and Spies in Smart Grid (June 09) President Obama: securing the electric infrastructure is a national security priority (June 09) Hiroshima, 2.0 Cyberspying of the US Electric Grid (April 09) Cyberspies penetrate electrical grid (April 09)

earth2tech.com

'Smart Grid' vulnerable to hackers (March 09) CIA: Hackers Have Attacked Foreign Utilities (Jan 2008)

Limited Information About Incidents

Little information sharing about actual attacks
BCIT incident database has about 30 incidents per year vs. 100s of thousands of incidents per year in CERT database Few cyber attacks on ICS for which details are public

Little information sharing about actual vulnerabilities

some are not easily or rapidly fixed assessments are done under NDA

Difficult to estimate risk

Difficult to demonstrate ROI for security spending

But lots of data about significant financial losses in enterprise and e-commerce
Why would control systems be immune?

Accidents Happen ...

Attacks Can Cause Similar Results

INL National Lab Aurora Demonstration, March 2007

Cyber Security Regulatory Requirements

FERC releases Smart Grid Policy - cyber security mandatory for Utility rate recovery (July 09) Regulators provide Smart Grid Stimulus Funding criteria - cyber security is mandatory (June 09) Strengthened Cyber Security Standards Approved for North American Utilities (May 09) Ontario Green Energy Act Drives Smart Grid With Security (May 09) NIST developing interoperability and security standards for Smart Grid AMI-SEC working group developed security requirements for AMI

AMI-SEC Task Force

Securing Control Systems

ACM CCS Tutorial Nov. 2009

Adversaries
Script kiddies Hackers Organized crime Disgruntled insiders Competitors Terrorists Hactivists Eco-terrorists Nation states

Threat Model
Targeted and untargeted threats
Targeted: terrorist, specifically crafted worm/virus, botnet Untargeted: generic worm/virus, script kiddy

Assume adversary has:

Complete knowledge of network Beachhead in enterprise network Limited access to control network But no valid credentials

How an Attack ProceedsStep #1

IED Engineering Workstation Management Console HMI Modem Pool Email Server Web Server

IED Data Historian RTU FEP

Control System Network

enterprise Firewall ICS Firewall Web Server

Enterprise Network

Internet

Business Workstation

Domain Name Server (DNS) Database Server Attacker

How an Attack ProceedsStep #2

IED Engineering Workstation Management Console HMI Modem Pool Email Server Web Server

IED Data Historian RTU FEP

Control System Network

enterprise Firewall ICS Firewall Web Server

Enterprise Network

Internet

Business Workstation

Domain Name Server (DNS) Database Server Attacker

How an Attack ProceedsStep #3

IED Engineering Workstation Management Console HMI Modem Pool Email Server Web Server

IED Data Historian RTU FEP

Control System Network

enterprise Firewall ICS Firewall Web Server

Enterprise Network

Internet

Business Workstation

Domain Name Server (DNS) Database Server Attacker

How an Attack ProceedsStep #4

IED Engineering Workstation Management Console HMI Modem Pool Email Server Web Server Vendor Web Server

IED Data Historian RTU FEP

Control System Network

enterprise Firewall ICS Firewall Web Server

Enterprise Network

Internet

Business Workstation

Domain Name Server (DNS) Database Server Attacker

How an Attack ProceedsStep #5

IED Engineering Workstation Management Console HMI Modem Pool Email Server Web Server Vendor Web Server

IED Data Historian RTU FEP

Control System Network

enterprise Firewall ICS Firewall Web Server

Enterprise Network

Internet

Business Workstation

Domain Name Server (DNS) Database Server Attacker

How an Attack ProceedsStep #6

IED Engineering Workstation Management Console HMI Modem Pool Email Server Web Server

IED Data Historian RTU FEP

Control System Network

enterprise Firewall ICS Firewall Web Server

Enterprise Network

Internet

Business Workstation

Domain Name Server (DNS) Database Server Attacker

How an Attack ProceedsStep #7

IED Engineering Workstation Management Console HMI Modem Pool Email Server Web Server

IED Data Historian RTU FEP

Control System Network

enterprise Firewall ICS Firewall Web Server

Enterprise Network

Internet

Business Workstation

Domain Name Server (DNS) Database Server Attacker

Defending ICS
Separate control network from enterprise network
Harden connection to enterprise network Protect all points of entry with strong authentication Make reconnaissance difficult from outside

Harden interior of control network

Make reconnaissance difficult from inside Avoid single points of vulnerability Frustrate opportunities to expand a compromise

Harden field sites and partner connections

mutual distrust

Monitor both perimeter and inside events Periodically scan for changes in security posture

50000 Foot View

Internet
IT Stuff

Enterprise Network

IT Stuff

Scan IDS NAC FW IDS FW

VPN FW Proxy AV IPS Log Mgmt Host IPS Host AV

Control Network
VPN AV

Event Mgmt Reporting

Host IDS Host AV

Partner 62351 Site

VPN

FW

IPS

P1711

Scan Field Site NAC

Field Site

Field Site

Logical Overlay on SP99 / Purdue Model of Control

Level 5 Level 4
Email, Intranet, etc. Enterprise Network

Site Business Planning and Logistics Network

Enterprise Zone

Terminal Services

Patch Mgmt

AV Server

DMZ
Historian (Mirror) Web Services Operations Application Server

Level 3

Production Control

Optimizing Control

Historian

Engineering Station

Site Operations and Control

Level 2

Supervisory Control

HMI

Supervisory Control

HMI

Area Supervisory Control

Control Zone

Level 1 Level 0

Batch Control

Discrete Control

Continuous Control

Hybrid Control

Basic Control

Process

Logical Architecture
Enterprise Zone contains typical business systems
Email, web, office apps, etc.

DMZ provides business connectivity

Contains only non-critical systems that need access to both Control and Enterprise Zones Enforces separation between Enterprise and Control Zones Consists of multiple functional sub-zones Separated by Firewall, IPS, Anti-Virus, etc.

Control Zone demarcates critical control systems

Consists of multiple functional sub-zones Internally protected by Firewall, IDS, Anti-Virus, etc.

How NOT to connect Control / Enterprise

Dual-homed server Dual-homed server with Host IPS / AV Router with packet filter ACLs Two-port Firewall Router + Firewall combination

See NISCC Good Practice Guide on Firewall Deployment for SCADA and Process Control Networks, NISCC and BCIT, Feb 2005

DMZLogical View
AV
Terminal Services Patch Mgmt

Emergency Disconnect
Proxy
AV Proxy

VPN

IDS FW IPS

Multiple Functional Sub-Zones

Scan IDS

DMZ

Host AV Host IPS

Historian Mirror

Web Services Operations

Application Server

No Direct Traffic Emergency Disconnect

DMZ Design Principles

DMZ contains non-critical systems Multiple functional security sub-zones Traffic between sub-zones undergoes firewall (& IPS or IDS) DMZ is only path in/out of Control Zone Default deny for all firewall interfaces No direct traffic across DMZ No control traffic to outside Limited outbound traffic from Control Zone Very limited inbound traffic to Control Zone No common ports between outside & inside Emergency disconnect at inside or outside No network management from outside Cryptographic VPN and Firewall to all 3rd party connections

DMZ Implementation (1)

Enterprise LAN

NAT DMZ LAN 2 DMZ LAN 3 DMZ LAN 4 Anti-Virus Proxy Host IPS / Anti-virus

Security Appliance With Multiple Ports Routing FW IPS

DMZ/Control Interconnect WAN/LAN

DMZ Implementation (2)

Enterprise LAN

VLAN-capable L2 switch DMZ VLAN 2 DMZ VLAN 3 DMZ VLAN 4 Anti-Virus Proxy Host IPS / Anti-virus dot1q trunk

NAT Security Appliance Routing FW IPS VLAN

NOT L3!

DMZ/Control Interconnect WAN/LAN

DMZ Implementation
Sub-zones implemented by physical LANs or VLANs
Physical LANs require multi-port Security Appliance VLANs require: VLAN-capable Security Appliance and Switch anti-VLAN hopping protections on switch and FW NO L3 (routing) on switch

FW implements policy between

DMZ LANs, Enterprise Zone, Control Zone

Anti-virus proxy controls outbound HTTP and/or FTP access to enterprise or Internet resources Host IPS and/or Host Anti-virus protects DMZ servers

Remote Access
Enterprise LAN

Terminal Services

AAA Server

Certificate Authority

Remote Access VPN DMZ

Remote Access Pool

DMZ/Control Interconnect WAN/LAN

Remote Access
Security Appliance terminates Host-to-site VPN into remote access pool
IPSEC VPN, SSL VPN, PPTP VPN

Authenticates user via:

AAA server, LDAP, Active Directory, etc. Can enforce use of multi-factor hardware token Time-varying password tokens for vendor access

Clients use VNC, Citrix, or Remote Desktop (RDP) to connect to Terminal Server Then VNC, Citrix, RDP, or Control System Apps to Control System Servers

Direct Remote Access

Enterprise LAN

AAA Server

Certificate Authority

Host-toLAN VPN
Endpoint Posture Assessment

Remote Access Pool

DMZ

RoleBased ACLs constrain network access within Control Network

DMZ/Control Interconnect WAN/LAN

Direct Remote Access

Security Appliance terminates Host-to-site VPN into remote access pool Security Appliance assesses endpoint security posture Authenticates user via AAA, LDAP, AD, etc. Clients use VNC, Citrix, or RDP to connect directly to Control System

BUT
Security Appliance enforces Authorization via User and/or Group ACLs
Role-Based Access Control

Control ZoneLogical View

DMZ

Level 3

Production Control

Optimizing Control

Historian

Engineering Station

Site Operations and Control

Level 2

Supervisory Control

HMI

Supervisory Control

HMI

Area Supervisory Control

Control Zone

Level 1 Level 0

Batch Control

Discrete Control

Continuous Control

Hybrid Control

Basic Control

Process

Control Zone Design Principles

Multiple functional security sub-zones Firewall and IDS between sub-zones Minimal number of connections to DMZ Control Zone independent of DMZ, Enterprise
Separate Security Appliance from DMZ Separate Time Server Separate AAA Allows emergency disconnect from DMZ

Cryptographic VPN and Firewall to all offsite IP connections (Field Site or Partner) IEEE P1711 for all offsite serial ICS connections Host IDS, Host AV, or app whitelisting where feasible Management only from management zone

Control Zone ImplementationHierarchical

Fast routing between VLANs via L3 switch ACLs between VLANs but no Stateful Firewall

DMZ/Control Interconnect WAN/LAN

Level 3 IDS Scan Level 2

SPAN L2

FW
L3

FW
L3 Gigabit dot1q Trunks L2 QoS, Shaping, Policing Port Security

Control Zone

10/100

Level 1 Host IDS Host AV

Control Zone ImplementationRing

Ring reduces wiring for linear sites like power dams but spanning tree can have problems with large rings

DMZ/Control Interconnect WAN/LAN

Level 3 IDS Scan Level 2

SPAN L2

FW
L3

FW
L3 Gigabit dot1q Trunks L2 QoS, Shaping, Policing Port Security

Control Zone

10/100

Level 1 Host IDS Host AV

Perimeter Protection in Utilities

Firewall IDS/IPS Client VPN Site-to-site VPN

DMZ

Proxy Network AV Host IDS/IPS NAC

Interior Protection in Utilities

IDS Port Scan Vuln Scan Firewall NAC Firewall SCADA VPN Port Scan IDS

SCADA VPN

Monitor, Log, Analyze, Report

Log Analyze Report Compliance

Managed Security

Beyond Network Security

Planning, processes, procedures, physical security, etc. are also important NERC CIP Regulatory Requirements provide reasonably good guidance in this area: CIP-001: Sabotage Reporting CIP-002: Critical Cyber Asset Identification CIP-003: Security Management Controls CIP-004: Personnel & Training CIP-005: Electronic Security Perimeters CIP-006: Physical Security CIP-007: Systems Security Management CIP-008: Incident Reporting & Response Planning CIP-009: Recovery Plans for Critical Cyber Assets

See www.nerc.com -> Standards -> Reliability Standards -> CIP

Summary
Todays ICS are mix of modern and legacy
vulnerabilities due to both lack of security design in legacy and security issues in newer equipment

Defense in depth is essential

both perimeter (DMZ) and interior security are crucial

Regulation and government action is driving change Smart Grid must be designed with strong security

Thanks!
andrew.wright@n-dimension.com

ACM CCS Tutorial Nov. 2009

Standards Efforts
NERC CIPs NIST Smart Grid Interoperability Standards Project NIST SP800-82 NIST SP800-53 NIST PCSRF Protection Profiles AMI-SEC ISA SP99 ODVA

IEEE P1711 (AGA 12) -- serial SCADA encryption

A Few References
www.nist.gov/smartgrid Securing Your SCADA and Industrial Control Systems, Version 1.0, DHS, ISBN 0-16-075115-8 Guide to SCADA and Industrial Control System Security, NIST SP800-82 ISA99 Industrial Automation and Control Systems Security, www.isa.org/MSTemplate.cfm? MicrositeID=988&CommitteeID=6821 AGA 12/IEEE P1689 SCADA Encryption Standard, scadasafe.sf.net