1
Performance Evaluation of Counter-Based
Dynamic Load Balancing Schemes for Massive
Contingency Analysis with Different Computing
Environments
Yousu Chen, Member, IEEE, Zhenyu Huang, Senior Member, IEEE, and Daniel Chavarría-Miranda
Abstract—Contingency analysis is a key function in the
Energy Management System (EMS) to assess the impact of
various combinations of power system component failures based
on state estimation. Contingency analysis is also extensively used
in power market operation for feasibility test of market solutions.
High performance computing holds the promise of faster analysis
of more contingency cases for the purpose of safe and reliable
operation of today’s power grids with less operating margin and
more intermittent renewable energy sources. This paper
evaluates the performance of counter-based dynamic load
balancing schemes for massive contingency analysis under
different computing environments. Insights from the
performance evaluation can be used as guidance for users to
select suitable schemes in the application of massive contingency
analysis. Case studies, as well as MATLAB simulations, of
massive contingency cases using the Western Electricity
Coordinating Council power grid model are presented to
illustrate the application of high performance computing with
counter-based dynamic load balancing schemes.
Index Terms—Contingency Analysis, Energy Management
System, Parallel Computing, and Dynamic Load Balancing.
I. INTRODUCTION
CONTINGENCY analysis is a key function in the Energy
Management System (EMS), which assesses the ability of
the power grid to sustain various combinations of power grid
component failures based on state estimates. Because of the
heavy computation involved, today’s contingency analysis can
be updated only every few minutes for only a selected set of
“N-1” contingency cases (i.e., failures of one component). For
example, the EMS system at Bonneville Power Administration
(BPA), one of the well-maintained systems, runs 500
contingency cases in a time interval of five minutes. Though it
has been a common industry practice, analysis based on
limited “N-1” cases may not be adequate to assess the
vulnerability of today’s power grids due to new development
in power grid and market operations.
The trend of operating power grids closer to their capacity
and integrating more and more intermittent renewable energy
demands faster analysis of massive contingency cases to
978-1-4244-6551-4/10/$26.00 ©2010 IEEE
safely and reliably operate today’s power grids. As electricity
demand continues to grow, the personnel operating and
managing the power grid are facing some new fundamental
challenges.
One of the new challenges in power grid operation is the
separation of administrative boundaries – called Balancing
Areas (or BAs) – which own, operate, and/or manage their
own areas of the grid. When performing contingency analysis,
each BA looks no further than its own boundaries. The case of
contingencies occurring simultaneously is not considered,
which will likely have very large system-wide impact. This
requires “N-x” contingency analysis, i.e. analysis of
simultaneous occurrence of multiple contingencies in multiple
BAs.
Another challenge is that as the current power grid
operation is closer to the edge of stability, the consequence
could be massive blackouts resulting in significant disruption
of electricity supplies and economic losses [1][2]. Power grid
blackouts often involve failure of multiple elements as
revealed in recent examples. Preventing and mitigating
blackouts requires “N-x” contingency analysis. The North
American Electricity Reliability Corporation (NERC) moves
to mandate contingency analysis from “N-1” to “N-x” in its
grid operation standards [3]. All this calls for a massive
number of contingency cases to be analyzed. As an example,
the Western Electricity Coordinating Council (WECC) system
has about 20,000 elements. Full “N-1” contingency analysis
constitutes 20,000 cases, “N-2” is roughly 108 cases, and the
number increase exponentially with “N-x”.
Earlier high-performance computing (HPC) applications to
power system problems such as state estimation and
contingency analysis have achieved promising results
[5][6][7]. The authors’ previous work [4] established the
framework of “N-x” parallel massive contingency analysis
with a dynamic load balancing scheme using a single counter,
illustrated by case studies of massive 300,000-contingency-
case analysis using the Western Electricity Coordinating
Council power grid model. Since the scalability of dynamic
load balancing schemes with a large number of processors is
likely to be limited by counter congestion, in this paper we
 2

continue to investigate counter congestion management with
multiple-counter dynamic load balancing schemes under
different computing environments and provide guidance for
users to select suitable schemes to meet their own needs.
This paper starts with an introduction of counter-based
dynamic load balancing schemes in Section II and an
overview of two main network environments, InfiniBand and
Ethernet, in Section III, followed by Section IV on the
MATLAB simulation results for multi-counter based dynamic
load balancing schemes. Section V presents the actual case
studies and performance analysis using a HP cluster computer,
NWICEB. Section VI provides the guidance for users to select
suitable schemes and discusses relevant issues on contingency
selection and decision support capabilities in the context of
massive contingency analysis. Section VII concludes the
paper with future work suggested.
computational efficiency is determined by the longest
execution time of individual processors. Hence, the
computational power is not fully utilized as many processors
are idle while waiting for the last task to finish; with the
dynamic load balancing scheme, the computation time on each
processor is optimally equalized.
Figure 2 shows the performance comparison of static and
dynamic computation load balancing schemes with WECC
full N-1 contingency cases (17,346 cases). Clearly, dynamic
load balancing has better linear scalability. Figure 3 shows the
evenness of execution time with different load balancing
schemes with 32 processors, where the variation of execution
time with dynamic load balancing scheme is much smaller
than its counterpart.
14,000-bus WECC Full N-1 Analysis
35

II. COUNTER-BASED DYNAMIC LOAD BALANCING SCHEME 30

Static load balancing
Dynamic load balancing
25
Contingency analysis is inherently a parallel process

Speedup
20
because multiple contingency cases can be easily divided onto
multiple processors and communication between different 15

processors is minimal. The data access is homogeneous. 10

Therefore, cluster-based parallel machines are well suited for 5

contingency analysis. For the same reason, the challenge in 0

parallel contingency analysis is not on the low-level algorithm 0 5 10 15 20

Number of processors
25 30 35

parallelization but on the computational load balancing (task Figure 2 Performance comparison of static and dynamic computation load
partitioning) to achieve the evenness of execution time for balancing schemes with WECC full N-1 contingency cases
multiple processors.
The framework of parallel contingency analysis is shown 1000 Static Load Balancing
in Figure 1 [4]. Each contingency case is essentially a power Dynamic Load Balancing

flow solution. In our investigation, the full Newton-Raphson 950

Time (seconds)

power flow solution is implemented. Given a solved base 900

case, each contingency updates its admittance matrix with an Average: 834 sec

incremental change from the base case. One processor is 850

designated as the master processor (Proc 0 in Figure 1) to
800
manage case allocation and load balancing, in addition to 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
running contingency cases. Processor #

Figure 3 Evenness of execution time for WECC full N-1 contingency analysis
Proc 0:
(1) Distribute base case Y0 matrix with different computational load balancing schemes
(2) Perform load balancing (static/dynamic)
Proc
Proc00
(3) Distribute case information to other processors The speedup performance of dynamic load balancing
(4) Perform contingency analysis scheme can be estimated using the following equation:

Speedup = N P
(tc + tio ) (1)
Proc
Proc11 Proc
Proc22 … Proc
ProcNN
t c + tio +t + P
(N − 1)t w
cnt
Other Proc’s: 2
(1) Update Y matrix based on case information: Y = Y0 + ΔY
(2) Perform contingency analysis
where N P is the number of processors, t cnt is the counter
Figure 1 Framework of parallel massive contingency analysis
updating time, t w is the waiting time due to counter
They are two main categories of load balancing schemes: congestion, t c and tio are the average computation time and
static load balancing and dynamic load balancing. The static I/O time, respectively. In order to improve speedup
load balancing scheme pre-allocates an equal number of cases performance, it needs to reduce the counter updating time
to each processor, while the dynamic load balancing scheme t cnt , as well as the waiting time t w .
allocates tasks to processors based on the availability of a The counter update time t cnt is mainly determined by the
processor. With the static load balancing scheme, the overall

network bandwidth and speed. Minimizing t cnt usually means
selecting a high-performance network connection between
processors. The waiting time t w is due to counter congestion.
Though more processors would improve the speedup
performance, they also increase the possibility of counter
congestion as shown in (1). Counter congestion will occur
when multiple requests arrive at the same time. Therefore, the
scalability of dynamic load balancing schemes is likely to be
limited by counter congestion.
In order to better manage counter congestion, we propose a
multi-counter based dynamic load balancing scheme with task
stealing. The framework of this new scheme is illustrated in
Figure 4 with two counters. The equal number of cases is pre-
allocated to the two counter groups. Each group has its own
counter (Proc 0 in Figure 4). Inside each group, the dynamic
load balancing scheme is applied based on the availability of
processors. When the pre-allocated tasks are finished in one
group, the counter in this group “steals” tasks from the other
group to continue the computation until all tasks are done. By
implementing the multi-counter dynamic load balancing
scheme, counter congestions can be reduced, and further
speedup is expected.
Figure 4 Framework of multi-counter based dynamic load balancing scheme
with task stealing.
The cost to minimize counter congestion using multi-
counter schemes is the overhead with managing multiple
counters. Even though additional counters can reduce counter
congestion, it is possible that the overhead would compromise
the benefit gained by reducing counter congestion. Therefore,
it is important that we evaluation the performance of these
load balancing schemes and determine under what conditions
the multi-counter scheme had superior performance over the
single-counter scheme.
III. NETWORK ENVIRONMENT COMPARISONS
As stated earlier, minimizing the counter updating time t cnt
usually means choosing a high-performance network
connection among processors. However, due to the cost
issues, PC-based Ethernet network dominates the current
utility control center environment. In order to better
understand the effect of high-performance networks, it is
important to study the performance of multi-counter dynamic
load balancing schemes under different network
environments.
The main network properties are latency and bandwidth. In
this paper, we are interested in the latency and bandwidth in
two common networks: 1GB/Sec Ethernet, and InfiniBand
network. The typical values of latency and bandwidth in these
two networks are listed in Table 1. The latency of InfiniBand
3
network is approximately 1/20 of that of Ethernet.
TABLE 1 THE TYPICAL VALUES OF LATENCY IN 1GB/SEC ETHERNET, AND
INFINIBAND NETWORKS
1GB/Sec Ethernet InfiniBand
Latency (μSec) ~30 ~1-2
Bandwidth 100~200 MB/Sec 10GB/Sec*
* InfiniBand is a type of communications links between processors and I/O
devices that offers throughput of up to 2.5 GB/Sec, and it can achieve
10GB/Sec or higher bandwidth through double-rate and quad-rate techniques.
IV. MATLAB SIMULATION RESULTS
Before the actual case studies are conducted, Matlab
simulations are performed to predict the performance of multi-
counter schemes under different simulated computing
environments. The advantage of using simulated environments
is the flexibility of studying different configurations and
reducing the time implementing the schemes on actual parallel
computers.
The main factors could affect speedup include: (a) the
number of processors, N P ; (b) the number of cases, N; (c) the
latency of the network communication; (d) the counter
updating time; and (e) the bandwidth of the network. Since the
effects of factor (c), (d), and (e) are equivalent in terms of the
contribution of total time, these factors can be treated as one
term, tcnt, for the purpose of MATLAB simulation. In order to
study the sensitivities of the factors of N P , N, and tcnt, two
sets of simulations are studied with the number of processors
ranged from 20 up to 210: (a) different number of cases, N =
500, 5000, 20000 with respect to different number of
processors; and (b) different tcnt, tcnt = 0.0001, 0.002, and
0.005, with respect to different number of processors.
In order to make the simulation data closer to the actual
computational time, the actual computational time for WECC
full N-1 contingency analysis is studied, based on the
histogram of the time, and then is used in MATLAB
simulation studies.
Figure 5 shows the speedup performance with different
numbers of cases. The Tcnt is fixed to 0.0001 for this set of
simulations. The horizontal axis is the number of processors in
base-2 exponentials, and the vertical axis is the speedup. It is
clear that the better speedup can be achieved when the number
of cases increases. This statement has been confirmed by the
case studies in [4]. With 512 processors, the speedup is 462
for 20,000 contingency cases, 503 for 150K cases and 507 for
300K cases.
The sensitivity of tcnt is shown in Figure 6. In this
simulation, the number of contingency cases is fixed to be
20,000. Same as in Figure 5, the horizontal axis shows the
number of processors in base-2 exponentials, and the vertical
axis shows the speedup.
There are three observations from Figure 6. The first
observation is that for certain number of processors, the larger
tcnt, which represents the low speed communication network
(e.g. Ethernet network), the less speedup archived for both
 4

single-counter and two-counter schemes; the second
observation is with larger time of tcnt, the two-counter dynamic
load balancing scheme shows better performance than the
single-counter scheme when the number of processors
increases. The larger the number of processors, the better
performance for two-counter scheme appears; and the third
observation is that when the number of processors is relatively
small, e.g. less than 128, and with tcnt=0.0001s, which
represents the high speed communication network, the
performance of single-counter scheme is better than that of the
two-counter scheme. However the two-counter scheme has
better performance than single-counter scheme when tcnt is
larger for the number of processors less than 128.
balancing schemes is implemented on a NWICEB cluster
machine, which is a total of 128 processors with the high-
speed InfiniBand communication link. The 14,000-bus WECC
power grid model is used as the study model. In order to
simulate the Ethernet environment on the NWICEB machine,
the operation of counter update is executed 20 times more to
mimic the low speed communication. The reason for using the
number of 20 is because that the latency of an Ethernet
network is approximately 20 times of that of InfiniBand.
Three different scenarios with different number of
contingency cases, N = 500, 5000, and 17346, are tested on
NWICEB using both the single-counter and two-counter
schemes. Two different environments (InfiniBand and
simulated Ethernet) are compared. The execution time of all
scenarios, excluding disk I/O time for the purpose of
eliminating side-effects, with InfiniBand network is listed in
Table 2, while Table 3 shows the execution time with the
simulated Ethernet network.

TABLE 2 EXECUTION TIME* IN SECONDS OF MASSIVE CONTINGENCY ANALYSIS

ON THE NWICEB MACHINE UNDER INFINIBAND NETWORK
# of cases 500 5000 full N-1 (17346)
# of procs 1_cnt 2_cnt 1_cnt 2_cnt 1_cnt 2_cnt
1 260.7 260.7 2285 2285 9038 9038
2 147.39 154.40 1327.9 1348.5 5095.6 5177.2
4 85.487 91.243 748.69 751.41 2880.3 2887.5
Figure 5: Single-counter vs. two-counter comparison for different number of
cases with respect to different number of processors 8 50.1 53.044 406.3 417.26 1488 1546.6
16 28.494 27.609 205.95 210.78 765 767.6
32 14.8325 14.949 108.46 110.35 397.81 386.2
64 8.1825 9.4521 62.339 57.92 187 201.9
128 5.7331 6.2403 30.01 33.251 96.63 97.87
* Disk I/O time excluded

TABLE 3 EXECUTION TIME* IN SECONDS OF MASSIVE CONTINGENCY ANALYSIS

ON THE NWICEB MACHINE UNDER SIMULATED ETHERNET NETWORK
# of cases 500 5000 full N-1 (17346)
# of procs 1_cnt 2_cnt 1_cnt 2_cnt 1_cnt 2_cnt
1 261.12 261.12 2359.9 2359.9 8994.1 8994.1

Figure 6: Single-counter vs. two-counter comparison for different counter time 2 147.51 144.16 1341.5 1316.4 5098.4 5015.2
with respect to different number of processors 4 83.546 85.655 754.4 764.7 2869.9 2804.6
8 49.191 48.145 402.45 419.18 1569.5 1532.7
The third observation is important for most utility/control
center users, who want to implement massive contingency 16 26.236 26.007 207.39 210.18 804.6 778.95
analysis in their current environments. Currently, most 32 14.12 16.903 109.03 112.7 408.36 399.34
utilities/control centers do not own a large number of 64 8.1423 8.7142 58.26 60.537 211.03 202.66
processors, and they are mostly using Ethernet network
128 5.8304 6.869 29.959 32.098 112.17 108.17
environment. Therefore, the two-counter dynamic load
balancing scheme would be useful for their contingency * Disk I/O time excluded
analysis applications.
In Table 2, the execution times with the two-counter
V. CASE STUDIES OF MASSIVE CONTINGENCY ANALYSIS WITH scheme are larger than those with the single-counter scheme,
DIFFERENT COUNTER-BASED SCHEME which indicates that the two-counter dynamic load balancing
scheme does not show better speedup performance than the
The massive contingency analysis framework with the
single-counter scheme under InfiniBand environment.
single-counter and two-counter dynamic computational load

Corresponding to Table 2, the speedup results with both
single-counter vs. two-counter schemes under InfiniBand
environment are shown in Figure 7. This result matches the
MATLAB simulation in Section IV. The main reason for this
phenomenon is that the latency in InfiniBand is low and the
communication speed is fast. As such, the counter congestion
is less likely to happen with a relatively small number of
processors. Therefore, the overhead introduced by an
additional counter impairs the performance under the testing
circumstances. As shown by the MATLAB simulation, the
two-counter scheme is more suitable for the application of a
larger number of processors, i.e. with a large number of
processors, the two-counter scheme will improve the
computational performance of massive contingency analysis.
100
90
80
70
60
Speedup
50
40
30
20
10
0 20
1 2 4 8 16 32 64 128
The number of processors
Figure 7: Single-counter vs. two-counter comparison with respect to different
number of under InfiniBand environment
When the communication speed is relatively low, counter
congestion is more likely to happen. As shown in Table 3,
when the number of contingency cases is large, (N=17,346),
the two-counter scheme can improve the overall performance
under the simulated Ethernet environment. Very importantly,
this is true when the number of processors is relatively small
and the number of contingency cases is large enough. For
example, for full N-1 contingency analysis cases and with 16
processors, the execution time with single counter is 804.6
seconds, while the time with two counters is 778.95 seconds,
which is 26 seconds less. The speedup with single-counter vs.
two-counter schemes under the simulated Ethernet
environment is shown in Figure 8. Figure 8 shows that when
N is large (=17,364), the performance of the two-counter
scheme is better than that of the single-counter scheme; while
the performance of the two-counter scheme is worse when the
number of cases is small. These results match the MATLAB
simulation in Section IV, Figure 6.
VI. DISCUSSION
The case studies, as well as MATLAB simulation results,
reveal insights regarding the performance of load balancing
schemes, which can serve as guidance for utilities to select
suitable counter schemes to implement massive contingency
analysis under their computing environments. Considerations
for the implementation include:
5
(a) In the case of a computing environment with an Ethernet
network or equivalent and no high-speed networking
capabilities with a large number of processors, the two-
counter dynamic scheme is expected to have better
performance for the application of massive contingency
analysis;
(b) In the case of a computing environment with a high-
speed communication network but only a small number of
processors, the single-counter dynamic scheme is suggested;
(c) In the case of a computing environment with a high-
speed communication network and a dedicated cluster
computer with a large number of processors, the performance
of the two-counter dynamic scheme will be better than that of
the single-counter scheme.
These considerations can be used as guidance for actual
implementation of massive contingency analysis.
90
Single N=500 80
Two N=500 70
Single N=5000 60 Single N=500
Speedup
Two N=5000 50 Two N=500
Single N=17346 40 Single N=5000
30
Two N=17346 Two N=5000
Single N=17364
10
Two N=17364
0
1 2 4 8 16 32 64 128
The number of processors
Figure 8: Single-counter vs. two-counter comparison with respect to different
number of under Simulated Ethernet environment
As mentioned in the Introduction section, the number of
cases increases exponentially as the “x” in “N-x” increases.
When N-x contingency are considered, there are two major
issues: massive number of cases and massive amount of data.
Since the sheer number of contingency cases leads to the
impracticality of even the simplified computation for all cases,
to solve the first issue, we need smart contingency selection
methods, as well as high performance computing (HPC)
techniques and hardware. The technical challenge of solving
the second issue is how to navigate through the vast volume of
data and help grid operators to manage the complexity of
operations and decide among multiple choices of actions. The
state-of-the-art industrial tools use tabular forms to present
contingency analysis results. When massive “N-x”
contingency cases are analyzed and the system is heavily
stressed, the tabular method of display is rapidly overloaded
and it is then impossible for an operator to sift through the
large amounts of violation data and understand the system
situation within several seconds or minutes. Thus the
usefulness of massive contingency analysis is undermined and
the HPC benefit is diminished. In order to solve the second
issue, advanced visualization techniques, as well as human
factors, are needed to provide real-time situational awareness
 6

and help operators to anticipate, recognize, and respond to

emergencies in time.
VII. CONCLUSIONS
The performance of single-counter and two-counter based
dynamic computational load balancing schemes is simulated
in MATLAB, and the schemes are implemented on a cluster
computer with an InfiniBand network environment and a
simulated Ethernet network environment. The observations of
actual case studies match the findings of MATLAB
simulation. The findings from MATLAB simulation results
and the case studies can serve as guidance in using a high-
Yousu Chen (M’07) received his B.E. in Electrical Engineering from Sichuan
University, China, his M.S. in Electrical Engineering from Nankai University,
China, and M.S. in Environmental Engineering from Washington State
University. Currently he is a Research Engineer at the Pacific Northwest
National Laboratory in Richland Washington. His main research areas are
high-performance computing applications, power system stability and control,
and power system operations. Mr. Chen is an IEEE member and the Chair of
the Richland Chapter of the Power Engineering Society.
Daniel Chavarría-Miranda is a Senior Research Scientist at the Pacific
Northwest National Laboratory in Richland Washington. He can be reached at
902 Battelle Blvd. MSIN K7-90, Richland WA, 99352, E-Mail:
daniel.chavarria@pnl.gov.

performance computer for the application of massive

contingency analysis with different network environments.
The single-counter dynamic load balancing scheme is
suggested for the users who own high-speed communication
network, but only have a small number of processors. The
two-counter dynamic load balancing scheme is suitable for the
users who either have a resource of large number of
processors, or have a small number of processors with a
relative low-speed network.
Future work shall include the study of smart contingency
screening and advanced decision support through techniques
such as visualization and parallel post-processing techniques.

VIII. REFERENCES

[1] D. N. Kosterev, C. W. Taylor, and W. A. Mittelstadt, “Model Validation

for the August 10, 1996 WSCC System Outage,” IEEE Trans. Power
Syst., vol. 14, no. 3, pp. 967-979, August 1999.
[2] U.S.-Canada Power System Outage Task Force, “Final Report on the
August 14, 2003 Blackout in the United State and Canada: Causes and
Recommendations”, Apr. 2004. Available at https://reports.energy.gov/.
[3] NERC standards, Transmission System Standards – Normal and
Emergency Conditions, available at www.nerc.com.
[4] Z. Huang, Y. Chen and J. Nieplocha, “Massive Massive Contingency
Analysis with High Performance Computing”, in: Proceedings of the
IEEE Power Engineering Society General Meeting 2009, Calgary,
Canada, July 26-30, 2009.
[5] Zhenyu Huang, and Jarek Nieplocha, “Transforming Power Grid
Operations via High-Performance Computing,” in: Proceedings of the
IEEE Power and Energy Society General Meeting 2008, Pittsburgh, PA,
USA, July 20-24, 2008.
[6] J. Nieplocha, A. Marquez, V. Tipparaju, D. Chavarría-Miranda, R.
Guttromson, Zhenyu Huang, “Towards Efficient Power System State
Estimators on Shared Memory Computers,” in: Proceedings of the IEEE
Power Engineering Society General Meeting 2006, Montreal, Canada,
June 18-22, 2006.
[7] Zhenyu Huang, Ross Guttromson, Jarek Nieplocha and Rob Pratt,
“Transforming Power Grid Operations via High-Performance
Computing,” Scientific Computing, April 2007.

IX. BIOGRAPHIES
Zhenyu Huang (M'01, SM’05) received his B. Eng. from Huazhong
University of Science and Technology, Wuhan, China, and Ph.D. from
Tsinghua University, Beijing, China, in 1994 and 1999, respectively. From
1998 to 2002, he conducted research at the University of Hong Kong, McGill
University, and the University of Alberta. He is currently a staff research
engineer at the Pacific Northwest National Laboratory, Richland, WA, and a
licensed professional engineer in the state of Washington. His research
interests include power system stability and control, high-performance
computing applications, and power system signal processing.