IP and Ethernet Communication Technologies for Substations Substation of the Future: Improve Network Reliability and Protocol Interaction

Grid InterOp Dec 2010

Navindra Yadav (Principal Engineer), nyadav@cisco.com Eruch Kapadia (Solution Architect), ekapadia@cisco.com December 2, 2010
2010 Cisco Systems, Inc. All rights reserved. -Ver 0.3

Grid-Interop 2010
1

Outline
Reliability and Availability Basics Inter Substation Traffic
Use Cases like Tele Protection, Primary Protection, etc Reliability and Design improvement

Intra Substation Traffic

Use Case local Protection Reliability and Design improvement

Intra Substation Designs to achieve 6+ Nines of System Availability

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
2

Reliability and Availability Basics

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
3

Measurement of Availability
Availability = (MTBF)/(MTBF + MTTR)
MTBF = Mean Time Between Failure MTTR = Mean Time To Repair

Unavailability = ( 1 Availability ) Availability = ( 1 Unavailability )

AvailabilityOfSerialParts = (AvailabilityOfPart) AvailabilityOfParallelParts = 1 - [ (1-AvailabilityOfPart)]

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
4

Calculation of Network Availability

Identify the Serial and Parallel components
1 2 4 6 2 3 4 8 5

9 11

10

12

13

14

15

Failure of any node, link disrupts Network Availability = A = A1 * A2 * A3 * A4 * A5 * A6 * A7 If all A* are 90%. A = 0.4782969 = 47%

Assume a simple Active-Active Design Network Availability = 1 (UnAvail_P1 * UnAvail_P2 * UnAvail_P3) = = 1 ((1-Avail_P1) * (1-Avail_P1) * (1Avail_P1)) If all A* are 90% A= 1 ((1-0.59049)* (1-0.729) * (10.4782969)) = 1 (0.40951* .271 * . 0.5217031) = 0.942102845513649 = 94%
5

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010

IEC 61850 GOOSE and SV over the WAN Challenges and Solutions

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
6

IEC 61850-90-1 Solution to Carry GOOSE/SV over the WAN

Tunnel. Example
Layer 2 Tunneling Protocol (L2TP / L2TPv3) RFC 3931 Generic Routing Encapsulation (GRE) Tunneling - RFC 2784

Gateway
Example Proxy Gateways GWs Must Terminate Protocols GWs must Understand Applications and configuration changes in the application Latency and Jitter addition, especially when GWs are implemented in software

Tunneling or Encapsulation is the more realistic option

MPLS, VPLS, PWs are examples of Encapsulation technologies
2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
7

Problem: IEC 61850 GOOSE/SV over WAN Layer2 Tunneling Fault Domain Extension Lower Availability
This approach creates Large Fault Domains
Substation network Faults spread Dramatically lower availability Faults in unrelated parts of the network propagate Calculation of Availability means factoring Availability of the entire L2 domain means UNRELATED networks too!!

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
8

Solution: IEC 61850 GOOSE/SV over IPv4/v6 Higher Availability

Small and contained Fault Domains with IPv4/v6
Layer 2 domains are small Substation network Faults do NOT spread Higher availability Calculation of Availability involves only relevant networks

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
9

Problem: Layer 2 GOOSE / SV over the WAN Implications on Scaling, Security, Replication, Flooding, etc

Issues:
Intra Substation Replication Inter Substation Replication Information Leakage Security Implications Wasted Bandwidth Limited Scale
2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
10

Solution: IEC 61850 with IPv4/v6 profile provides Scalability, Security, etc
GOOSE/SV on IPv4/v6 routable protocol
Scalable Low (in usecs) Latency All HW forwarding Path Low (in usecs) Jitter Cyber Security benefits Easy to trouble shoot and manage over WAN proven model

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
11

 IEC 61850-90-5 for PMUs is working on a 61850 profile to carry GOOSE / SV over TCP/IP[v4v6] stack IP profiles being developed above must Not only be restricted to PMUs, but also to other all relays and applications.

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
12

Comparison IEC 61850 SV/GOOSE over Ethernet Vs UDP/TCP/IP[v4/v6] Stack

Topic
Maturity

IEC 61850 over Ethernet

Deployed in Production Environments Wide Spread Low (10s of sites). Overlay on top of the WAN topology. Mgmt/Troubleshooting burden Lower Availability. WAN tunnel failure is detected in order of seconds. Overlay Tunnels, etc create additional state in the Network. Which reduces the overall availability of the solution. LARGE. Layer 2 faults spread across substation domains. Eg loops in one substation may disrupt other substations Inefficient. Replication at WAN edge boundary (overlay). Inefficient. Flooding inside the substation LAN (Vlan)

IEC 61850 over UDP/TCP over IPv4/v6

Standard Drafting Stage (IEC 61850-90-5 draft has a IPv4/v6 profile for GOOSE and SV) --Extremely High (Internet Scale) Native Routing and Forwarding Fast Convergence techniques can repair WAN faults in order of sub 50 msecs Less state in the network

IED/Relay Vendor Support Scalability Tunnels / Encapsulation WAN Availability / Reliability Inherent Network State

Fault Domain Size

SMALL. Restricted fault domains

Inter Substation Traffic Replication (Multicast) Intra Substation Traffic Replication (Multicast)
2010 Cisco Systems, Inc. All rights reserved.

Efficient. WAN Network replicates at most optimal points Efficient. No flooding inside the substation multicast delivered to only interested hosts (IGMP/MLD snooping)

Grid-Interop 2010
13

Comparison Contd
Topic
Latency

IEC 61850 over Ethernet

High. Tunneling (typically in software/ucode fast path - msecs) Protocol Translation (typically in software 10 to 100 msecs)

IEC 61850 over UDP/TCP over IPv4/v6

Low. Extremely low latency ( < 20 micro secs). ASIC based forwarding Low (few usecs) Jitter. ASIC based forwarding. Superior. No flooding. Traffic delivered to authorized parties only.

Jitter Cyber Security

Higher Jitter (order of multiple msecs) Weaker. Large Flooding domain across substations create security challenges (like spoofing, replay, DOS, info leak domains)

State Scaling Domain (Eg. Mac Table size) Layer 2 Media dependent solution

Inferior. Limited all substations learn of everyone else thanks to flooding Yes

Superior. Only stations that communicate with each other need to know about each other No

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
14

Intra Substation Network Topology Choices

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
15

Background
A lot of discussions going on Ethernet topologies for Substation Two Leading topologies
Rings Redundant Trees

The Utility industry is familiar with Rings Cisco has massive deployment expertise with
Ring Topologies Redundant Tree Topologies and many others

There are applications where both topologies make sense Cisco wants to share its experience with these different Topology choices Next few slides compare Pros/Cons of Rings Vs Redundant Tree topologies
Majority of Ethernet deployed world wide in IT space (Enterprise/Datacenter) is based on Redundant Tree Topologies

Evaluate these Topologies (and their Pros/Cons) when you are designing
New Substation Designs Substations going through major retrofits

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
16

Some Characteristics of Ring Topology

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
17

Some Characteristics of Redundant Tree Topology

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
18

Comparison notes between the two leading Topologies

Topic
Physical Redundancy Connectivity/Topology

Redundant Trees
Yes More conductor. If conductors are laid inside trenches and conduits with limited capacity for extra cables its an issue for trees. Fixed and deterministic latency. Tree depth determines the number of hops. Smaller Fault Domain (fault limited to the triangle of switches in the tree) All inter switch traffic contends at limited and few fixed points in the tree topology. Superior (only the the root switches need to have policies and mac addresses for every device. The leaf switches just need to have capacities to support their downstream end devices) Superior (no downtime to the network to add a new leaf switch) Fewer and equal number of contention points, through which all traffic passes when going between two access switches, yields a fairer system Fast convergence can be achieved by using FlexLinks. Zero down time with PRP (from IEC 62439). Fewer and fixed number of switches in the switching path results in a higher MTBF. As there are fewer switches to switch through. Also aggregation switches can be designed for higher availability. Superior (not all switches have to have all vlans, also the flooding domains are smaller) Authenticator Function (for protocols like 802.1x) only on Switches 2010 Cisco Systems, Inc. All rights reserved.

Rings
Yes Simpler, Less conductor/fiber. In some cases trench and conduit size may make ring the only viable option. Latency varies. The number of hops between the source and the destination depends on where the loop in the ring is broken. When the blocking point changes the latency also changes. The whole ring is the fault domain. All inter switch traffic contends for the ring bandwidth

Predictable Latency

Smaller Fault Domain Bandwidth / QoS Predictability Scalability

Inferior (all switches have to learn about all end points. Least capable switch determines the capacity of the ring)

Maintenance and serviceability Fairness

Inferior (downtime seen by the ring when a switch is added or deleted from the ring) Traffic sent by the edge switches has to compete with similar class of traffic at every hop on the ring, the contention points and their number can change over time. Also the contention points vary between two access switches. Faster convergence (sub 50 msecs) can be achieved by using some like REP. Zero down time with PRP (IEC 62439) protocols. Variable MTBF as the number of switches in the number of switches vary depending on the topology.

Fast Convergence Availability

Cyber Security / Traffic containment

Inferior. (all switches have to have all vlans). If the Ring nodes are end points then over all security suffers. End hosts do not authenticate other end hosts
Authenticator Function (for protocols like 802.1x) has to be pushed to all 19 devices on the ring like end devices

Grid-Interop 2010

Substation Network Designs for 6+ Nines of Availability

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
20

Designs to meet or exceed 6+ Nines of Availability (Proactive and Reactive Redundancy) with Redundant 61850 Actors

WAN

WAN

SUBSTATION

Blue Network is Active Primary Network. All devices are at least attached to the Blue network

Green Network is Active Redundant Network. Smaller in size as not all devices will be dual attached

Redundant Actors and sensors

Other

MU

BPU

BIED

MU

BPU

BIED
Simplified view of the Power Network

Power Line

Superior Availability Characteristics Above can be built using a Tree or Ring topologies Active-Active Design No down time due to network convergence events Zero Down time for any failure N-1 Redundancy for any failure Can be improved even further if Blue and Green end points can process each others updates besides their own too. Get to N-2 redundancy
2010 Cisco Systems, Inc. All rights reserved.

Legend: MU = Merging Unit BPU=Bay Protection Unit BIED= Breaker IED

Grid-Interop 2010
21

Designs to meet or exceed 6+ Nines of Availability (Proactive and Reactive Redundancy) without Redundant Actors

WAN

WAN

SUBSTATION

Blue Network is Active Primary Network. All devices are at least attached to the Blue network

Green Network is Active Redundant Network. Smaller in size as not all devices will be dual attached Using say IEC 62439 - PRP

Other

MU

BPU

BIED

NO Redundancy of Actors and sensors

Power Line

Simplified view of the Power Network

Superior Availability Characteristics Above can be built using a Ring topology too Active-Active Design No down time due to network convergence events Zero Down time for any network device failure
2010 Cisco Systems, Inc. All rights reserved.

Legend: MU = Merging Unit BPU=Bay Protection Unit BIED= Breaker IED

Grid-Interop 2010
22

Navindra Yadav
nyadav@cisco.com

Eruch Kapadia
ekapadia@cisco.com

2010 Cisco Systems, Inc. All rights reserved.

Grid-Interop 2010
23