This article has been accepted for inclusion in a future issue of this journal.
Content is final as presented, with the exception of pagination.
IEEE SYSTEMS JOURNAL
Development and Application of a Real-Time
Test Bed for Cyber–Physical System
Ceeman B. Vellaithurai, Member, IEEE, Saugata S. Biswas, Student Member, IEEE, and
Anurag K. Srivastava, Senior Member, IEEE
Abstract—Enhanced integration of information and communi-
cation technologies in the smart grid has led to an increase in the
number of cyber assets and has also opened up the possibility of a
cyberattack. It is necessary to understand the complex relation-
ship between the cyber and physical domains, and its potential
impact on the power grid because of a successful cyber–physical
attack. A cyber–physical test bed that can model and simulate the
smart grid is necessary to test and validate algorithms and devices.
This paper presents the development of an end-to-end, real-time
cyber–physical test bed using Real-Time Digital Simulator and
Network Simulator 3 (ns-3). A methodology for integrating the
hardware phasor measurement unit and the phasor data con-
centrator in the test bed is presented along with the detailed
modeling of the communication network for the power system.
The developed test bed is validated and used to demonstrate the
impact of different cyberattacks on the power system and tested
algorithms.
Index Terms—Cyber security, cyber–physical test bed, Network
Simulator 3 (ns-3), real time, Real-Time Digital Simulator (RTDS),
smart grid.
I. I NTRODUCTION
T HE smart-grid investments resulted in a major upgrade of
the electric power grid (EPG) infrastructure by improv-
ing efficiency, reliability, and sustainability [1]. Technological
advancements that support smart grid include phasor measure-
ment units (PMUs), digital fault recorders, smart meters for
measurements, wired and wireless communications technology
for data transfer, and distributed and parallel computing for fast
analysis of data using various applications. The requirements
that must be satisfied by smart-grid communication technolo-
gies and comparative analysis on the available technologies
are provided in [2]. The National Institute for Standards and
Technology has documented the cyber vulnerabilities associ-
ated with the use of existing technologies [3]. Additionally, the
U.S. Department of Energy has listed attack resilience as one of
Manuscript received December 12, 2014; revised June 24, 2015; accepted
July 29, 2015. This work was supported in part by the Department of Energy
under Award DE-OE0000097 (Trustworthy Cyber Infrastructure for the Power
Grid).
C. B. Vellaithurai is with Schweitzer Engineering Laboratories, Pullman, WA
99163 USA (e-mail: ceeman_vellaithurai@selinc.com).
S. S. Biswas is with Alstom, Bellevue, WA 98004 USA (e-mail:
saugatasbiswas@gmail.com).
A. K. Srivastava is with The School of Electrical Engineering and Computer
Science, Washington State University, Pullman, WA 99164-2752 USA (e-mail:
asrivast@eecs.wsu.edu).
Digital Object Identifier 10.1109/JSYST.2015.2476367
1932-8184 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1
the seven properties required by the smart grid to meet future
demands [4].
The EPG has been recognized as a critical infrastructure with
a high risk of becoming the target of a cyberattack [5]. Because
of the idiosyncrasies of the power grid, the application of
existing technologies to prevent or mitigate cyberattacks is not
a direct possibility. The disruptions because of a cyberattack on
the smart grid transcend the cyber realm to affect the physical
realm as well. Hence, the approach to security of the smart
grid must combine cyber security and power system security
into cyber–physical security (CPS). Cyberattacks include false-
data injection attacks on state estimation and electricity markets
[6], [7], denial-of-service (DoS) attacks on a critical asset [8],
malicious intrusion, and the recent Distributed Network Pro-
tocol, Version 3.0 (DNP3) implementation vulnerability doc-
umented by the Industrial Control Systems Cyber Emergency
Response Team [9]. The impact of these kinds of attacks on the
physical devices in the EPG can be catastrophic. Power equip-
ment usually has high costs associated with it, and replacements
for some of these devices may have a long lead time to obtain.
Typically, attackers do not have complete information of the
system. A weak attack may be based on limited information
such as topology of the system and limited local measurement
data [10]. Undetectable attacks and countermeasures to defend
such attacks based on topological and limited measurement
information of the system have been presented in [11]. Tech-
niques and challenges of intrusion detection in cyber–physical
systems are discussed in [12]. Algorithms, devices, and ap-
plications to be used in the smart grid must be tested before
deployment. To enable testing of such devices and applications,
a prototype that can simulate the actual operating conditions
with sufficient detail and accuracy is required. Cyber–physical
test beds serve as one of the ideal tools to be used in order
to understand the relationship between cyber and physical
domains associated with the smart grid. It can be also used for
testing and validating the research on CPS of the EPG. The
development of test beds can be a difficult process because
of the complexity of integrating the cyber system and power
system simulators to run in real time. Majority of the studies
related to the development of test beds deal with simulated
and emulated tools; however, these tools are not run in real
time. The primary difference between simulation and emulation
is that, in emulation, the packets generated in the external
hardware can be sent to the destination through the emulator
incorporating communication delays.
The major value of this paper compared with other similar
test beds include: 1) the integration of hardware PMUs and
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2 IEEE SYSTEMS JOURNAL
associated infrastructure into the real-time cyber–physical sim-
ulator using Real Time Digital Simulator (RTDS) and Network
Simulator 3 (ns-3); 2) the methodology for validation and
testing the real-time test bed; 3) flexibility for integrating real
and simulated components in the test bed; 4) a detailed model
of the power-system communication network in real time with
support for different protocols; 5) the use of simulated entities,
such as a virtual host with the capability to interact with real
hosts; and 6) the use of ns-3, which is an open-source software
tool. Therefore, the source code is available to users to modify
and implement features as needed. The developed test bed
is limited by resources to model the large system, which can
be easily solved by securing additional financial resources.
The major advantage of the test bed is that the operations
and actions do not need to be scheduled beforehand. Events
can be triggered or performed at any desired time with the
consequences reflected as in a real system. This makes the
test bed ideal for studying cyber–power system attacks and
developing methodologies to mitigate the effects of these
attacks.
In the following sections, the development of a real-time
cyber–physical test bed using both RTDS and ns-3 is described.
Section II provides a short summary of related work for mod-
eling the power system and the cyber system. In addition, the
uniqueness of using RTDS and ns-3 in the test bed is also
examined. Section III provides an introduction to cyber–power
system modeling. Section IV examines the architecture of the
developed real-time test bed. Section V presents the validation
of the test bed. Section VI describes the potential applications
and presents actual case studies of the test bed.
II. R ELATED W ORK
Cybersecurity research needs have driven the development
of smart-grid cyber–physical test beds. There have been sev-
eral efforts aimed at the development of cost-effective and
accurate test beds. Cosimulation of heterogeneous systems is
common in other areas of research and has gained popularity
in cyber–physical simulation. This section provides a detailed
look into cosimulation environments that have been developed
for studying the smart grid. In [13] and [14] a hybrid simu-
lation architecture, which is based on IEEE 1516 High-Level
Architecture, was presented, and it enabled combined simu-
lation of power and communication systems in an integrated
environment.
The Electric Power and Communication Synchronizing Sim-
ulator (EPOCHS) is a platform for agent-based electric power
and communication network simulation. It is the integration
of three different simulators: PSCAD/EMTDC for transient
timescales, PSLF for power system modeling, and ns-2 for
communication network modeling [15]. A carefully designed
software mediator called runtime infrastructure is responsi-
ble for interfacing and synchronizing between the individual
simulators by allowing them to exchange data periodically. A
cyber–physical test bed using Internet-Scale Event and Attack
Generation Environment for emulating wide-area network com-
munications and RTDS/DigSilent for simulating power systems
has been developed at Iowa State University [16]. The test bed
is aimed at simulating the smart-grid environment and studying
the effects of cyberattacks on the EPG.
SCADASim, which is a framework for building supervisory
control and data acquisition (SCADA) simulations, is built on
top of OMNeT++ network simulator [17]. It is similar in
operation to EPOCHS, except that the simulator also allows for
the possibility of including real devices to a limited extent. By
using schedulers, it is possible to test the effect of attacks on real
devices, although it uses a simulated environment. GridSpice,
which is a distributed simulation platform for the smart grid,
uses Gridlab-D and MATPOWER as the network and power
simulation tools, respectively [18]. This is a cloud-based simu-
lation platform aimed at simulating large networks with hun-
dreds of connected generators and distribution networks. In
addition to power and communication network simulation, it
also allows market operations. Potential applications include
renewable energy integration, home area control and smart
algorithms, electric vehicle infrastructure, distributed energy
resources, microgrids, demand response and distribution oper-
ation, and utility-scale storage. GridSim is built using Gridstat
and TSAT [19]. GridStat is a wide-area data delivery framework
based on a publish–subscribe architecture [20]. It is used to
deliver data simulating the communication network. A transient
stability simulator, i.e., TSAT, is used for power system simu-
lation. GridSim simulates the power grid, the information and
communication technology (ICT) infrastructure that overlays
the grid, and the control systems. The primary focus is the
design and testing of wide-area control applications using PMU
and other high-rate timestamped data for large systems. The
ORNL Power System Simulator setup uses ns-2 and A Discrete
EVent system Simulator (ADEVS) simulation tools [21]. An-
other variation that uses OPNeT++ instead of ns-2 has also
been developed. The IBCN Smart Grid Simulator simulation
environment is implemented using OMNeT++ and MATLAB
[22]. The environment is designed as layered architecture, in
which three layers are defined: application, middleware, and
support layers. It is used to evaluate demand-side management
algorithms for electric vehicles.
Some test beds are more specific in their purpose, such
as the Test Bed for Analyzing Security of SCADA Control
Systems (TASSCS) developed by the University of Arizona,
which is meant for research on SCADA systems only [23]. It
uses OPNeT++ system-in-the-loop simulation for the com-
munication system, and the PowerWorld simulator to provide
EPG simulation. It is primarily used for research on intrusion
detection. SCADA CST [24] is another platform that is similar
to TASSCS, except that RINSE is used to simulate the cyber
system. The National SCADA Test Bed is a foundational test-
bed initiative, which represents a national laboratory collabo-
rative project [25]. The Virtual Control Systems Environment
developed by Sandia National Laboratory uses OPNeT++ and
PowerWorld simulator such as the TASSCS test bed. It uses
simulated, emulated, and physical devices to provide a ver-
satile reconfigurable platform [26]. The Global Event-Driven
Cosimulation Framework (GECO) combines the power-system
load flow and ns-2 to provide a cosimulation framework. The
main goal here is the modeling and simulation of wide-area
monitoring, protection, and control schemes [27].
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

VELLAITHURAI et al.: DEVELOPMENT AND APPLICATION OF REAL-TIME TEST BED FOR CPS 3

III. C YBER AND P OWER S YSTEM M ODELING

The developed cyber–physical test bed is the outcome of the
integration of RTDS, which simulates the power system in real
time, and Network Simulator 3 (ns-3), which emulates the com-
munication network with controllers and energy management
tools. This section provides a brief overview of the modeling
of these two systems. The hardware and software resources for
monitoring and control are part of the test bed.

A. Power System Modeling

RTDS is a dynamic virtual power system simulator designed
for continuous real-time operation. A model of the simulated
power system is defined using a graphical modeling language
called RSCAD. The operation is in real time because the typical
time step of operation is around 50 ms. RTDS can be connected
to external devices through dedicated analog and digital signal
exchange devices, making it possible to perform hardware- Fig. 1. Conceptual cyber–physical test-bed representation.
in-the-loop simulations. In addition, the RTDS is modular,
allowing for the use of DNP3 protocol; software PMUs, which
comply with the IEEE C37.118.1 standard; GOOSE mes-
saging; and IEC 61850-9-2 standard [28]. These expansion
modules/cards provide software devices capable of exchanging
information with other equipment. These definitive features
make the RTDS one of the better power-system simulators.

B. Cyber System Modeling

ns-3 is used for emulating the communication network,
which overlays the simulated power system. ns-3 has a modular
implementation and contains a core library that provides the ba-
sic framework of the communication network. The simulator li-
brary specifies simulation time objects, schedulers, and events.
Protocol entities implemented in ns-3 are written to mimic a
real-world implementation facilitating data exchange between
a simulated host in ns-3 and hardware in the test system. This
makes ns-3 suitable for emulation purposes. ns-3 is run on a Fig. 2. Reduction of IEEE 14 bus system into substations.
dedicated server to emulate the communication network in real
time. Note here that the real-time implementation of ns-3 uses
may be leased if required. Multiprotocol Label Switching is
the system time to schedule events. This time is synchronized
used for managing the IP network traffic. The service segrega-
to a GPS clock input that is used to time-synchronize the
tion used to differentiate traffic includes telemetry protection,
devices in the test bed. ns-3 has been successfully used in the
AMI, SCADA, and engineering access [29]. The conceptual
Open-Access Research Test Bed for Next-Generation Wireless
cyber–physical test-bed representation integrating the power
Networks (ORBIT) to emulate communication.
and cyber simulators is shown in Fig. 1.
At the transmission level in the power grid, fiber-optic cables
are generally used between all the major substations and the
control center forming redundant paths for data transfer. Cables IV. C YBER –P HYSICAL T EST B ED
are typically run along the transmission lines on the same tower.
A. Cyber Network Architecture
All-dielectric self-supporting fiber-optic cables are installed
along the transmission lines and use the same tower support in- It is beneficial to derive the power-system communica-
frastructure. Redundancy is provided to cover for the failure of tion network topology based on the power network topology
one or more links in the system. If it is not feasible to lay a fiber- because it will resemble a real system. The communication
optic line, private Worldwide Interoperability for Microwave network topology derived for the IEEE 14 bus system shown
Access networks are used. For distribution network communi- in Fig. 2 will be used to explain the working of the test bed.
cations, such as advanced metering infrastructure (AMI) and Two or more buses that are connected via a transformer are
distributed automation, low-speed networks with bandwidth in assumed located in the same substation and are modeled as
the range of 200 kb/s are used. Public communications lines a single communication node. Ten substations are identified
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
4 IEEE SYSTEMS JOURNAL
Fig. 3. Network IP for each node in the IEEE 14 bus system.
in the IEEE 14 bus system. Each node in Fig. 3 represents
a private local area network (LAN). Interconnection of the
LANs is represented by the communications links between
them. Because the IEEE 14 bus system is a transmission-level
system, fiber-optic links are considered for communications
links. The length of fiber-optic cables is calculated based on the
impedance of the transmission line by converting it to miles,
as done in [30]. The distance is then used to calculate the
propagation delay associated with that link. Fig. 3 shows the
communication network topology representation along with
the propagation delay for the IEEE 14 bus system. The control
center is assumed located near the slack bus because, typically,
the spinning reserve with the largest generation is located at the
slack bus.
1) Wide-Area Communication Network: The delays to be
emulated by the simulator include processing, transmission,
queuing, and propagation delays. These delays are calculated in
runtime depending on the network state. The processing delay
is based on the number of data packets flowing through each
network node, the transmission delay is calculated based on the
bandwidth and size of the packet, and the queuing delays are
emulated as required and depend on the congestion in the sys-
tem at any given point of time. The communication network is
essentially an IP-based routing network. Each private network
(node) communicates with other private networks through their
respective gateway. The gateway routing tables are populated
with network routes based on the shortest path method using
Dijkstras algorithm. Fig. 3 provides the different IP addresses
assigned to each substation.
2) LAN: The measurement devices interface with the RTDS
to receive measurement signals through dedicated wires. In
some cases, the devices maybe simulated within the RTDS.
Therefore, measurement signals are internally assigned. At
each node, there may be several of these devices. Fig. 4 shows
the separation between the ns-3 simulated environment and the
physical devices, such as the RTDS and measurement devices.
The gateway of a particular LAN is located inside the ns-3
simulation environment, as shown in the figure. The hardware
and software devices responsible for measurement and data
transfer are located outside of the ns-3 environment. These
devices communicate with the gateway inside ns-3 as if it were
a real gateway located on its own LAN. For communication
Fig. 4. LAN simulation.
with devices external to the private LAN, packets are routed
through the local gateway.
B. Power System Architecture
The network nodal view can be split into two major views,
i.e., the substation and the control center.
1) Substation View: A substation might have several devices
interconnected and interacting with each other through IEC
61850-compliant protocols or other standard protocols. The
interaction between the devices is flexible and configurable
according to user requirements. In this paper, it is assumed that
the transmission level system is fully PMU enabled and that
each bus has at least one PMU. The substation view of this type
of a system is shown in Fig. 5.
Only one PMU is shown per node because of space con-
straints. The number of PMUs may vary depending on the size
of the substation in question. Interaction between the PMUs is
also a possibility. The PMU output is concentrated in a phasor
data concentrator (PDC) for transmission to super PDCs. Each
substation is considered to be on its own private network, with
access to other private networks through its local gateway.
Engineering access is used to gain access to the private network
to change settings and configuration files in the relays from
the control center. A substation computer may or may not be
present locally to make use of the data archived locally in order
to run any applications.
2) Control Center View: The generalized control center view
is shown in Fig. 6. Depending on the application and data to be
used, the devices and software can be customized. Fig. 7 shows
the control center view for the PMU-enabled transmission
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
VELLAITHURAI et al.: DEVELOPMENT AND APPLICATION OF REAL-TIME TEST BED FOR CPS
Fig. 5. Substation view for the PMU-enabled transmission system.
Fig. 6. Generalized control center view.
system being considered. The super PDC at the control center
is responsible for receiving data from all substation PDCs.
Human–machine interface (HMI) and visualization tools may
be used depending on the application or algorithm to be run in
the test bed. It is assumed that there is one super PDC in the
entire system, which is present at the control center.
C. Monitoring and Control Devices
To complete the test bed, it is necessary to have monitoring
and control devices such as relays, measurement units, data
collection, and data retrieval units. A combination of hardware
and software devices is used to implement the measurement and
control layer of the test bed. A total of 14 PMUs are required if
a PMU is placed at each bus in the IEEE 14 bus system. This
can be achieved through the use of real hardware PMUs and
simulated software PMUs in the RTDS. Similarly, PMU data
collection and aggregation through PDCs can be accomplished
using a combination of hardware and software PDCs.
5
Fig. 7. Control center view for the PMU-enabled transmission system.
Fig. 8. Data transfer from Node 9 PMU to control center PDC.
Six hardware PMUs from different vendors, including SEL,
Alstom, GE, and ERLPhase Power Technologies Ltd., are used
in the test bed. Eight software PMUs are simulated in the RTDS
expansion card. The data from these PMUs are aggregated at
the PDCs to generate an output stream that is sent to the PDC at
the control center. This connection is through the ns-3 network
emulation, thereby incorporating the cyber network dynamics.
SEL-5073 Synchrowave PDC software, SEL-3373 Station PDC
(hardware), and OpenPDC are the PDCs used in the test bed.
The authors would like to clarify that the test bed is not
restricted to the use of IEEE C37.118-format PMU device
data delivery alone. For example, DNP3 traffic may be used
simultaneously with the current setup if devices with DNP3
protocols are available. The network simulator can carry any IP-
based traffic. The ability to emulate a communication network
for any combination of IP protocol-enabled devices makes the
test bed versatile.
D. Integrated Cyber–Physical Test Bed
The test bed consists of three major layers: the power-
system layer, the monitoring-and-control-system layer, and the
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
6 IEEE SYSTEMS JOURNAL
Fig. 9. Overall cyber–physical test bed integrating different resources.
communication-network layer. There is no direct coupling
between RTDS and ns-3. RTDS simulates the power system
and provides measurement signals to the measurement layer.
The measurement devices then communicate to the destination,
through ns-3, emulating the required communication network
characteristics. The substations, each having their own private
network of devices, are interconnected through a multihop
network topology. Data flow is possible between substations
and with the control center. If data flow occurs between two
substations, the data packet is passed through ns-3, which
simulates the appropriate delays associated with the transmis-
sion of that packet between the two substation gateways. For
wide-area communication system emulation, a similar process
is used.
Consider the scenario shown in Fig. 8, where a PMU at
Node 9 needs to send data to the control center. In this case,
the PMU data are first concentrated at the local substation PDC.
This process of intrasubstation communication occurs on a real
LAN network. The local PDC sends data to the super PDC
through the highlighted path in ns-3. This adds the communi-
cation network dynamics based on packet origin, destination,
and network configuration while delivering the packet to the
control center. The complete integrated cyber–physical test bed
is shown in Fig. 9.
V. VALIDATION OF THE T EST B ED
In order to validate the integrated test bed, component-
level performance and system-level performance were com-
pared with benchmark performance. For component-level
performance, the RTDS factory test certification was obtained
from the manufacturer. Additionally, the results from several
test cases obtained using RTDS were compared with other
software tools, such as MATPOWER, and published results
for the standard IEEE system. The development of the open-
source communication network simulator ns-3 started in 2006
and is still in active development. Both of these simulators have
been used extensively and validated by other researchers at
the component level. ns-3 is used in emulation mode, which
means that the virtual applications, which generate packets, are
replaced with real hosts. As far as ns-3 is concerned, the only
change is that the virtual packets are replaced with real packets.
Therefore, the working of ns-3 is essentially the same for real
packets as it is for virtual packets.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
VELLAITHURAI et al.: DEVELOPMENT AND APPLICATION OF REAL-TIME TEST BED FOR CPS
TABLE I
VALIDATION R ESULTS
Control-center components and other devices were also val-
idated individually with offline results using commercialized
software tools for a specific application. Note that most of the
hardware components used are industrial devices with compre-
hensive quality testing. Hence, it is reasonable to assume that
the test bed developed through the integration of all components
should result in a validated integrated test bed. Although vali-
dation of a real-time test bed is difficult to accomplish because
of the unavailability of a benchmark, limited performance tests
can be conducted for validation.
To validate the integrated test bed, the online results are
compared with the offline simulation results, which are per-
formed in ns-3 and presented in Table I as an example. The
values presented are the average round-trip times of 50 packets,
each having a size of 200 B, from each node in the network
to the control-center node. For Node 0, the offline simulation
delay is given as 0.4309 ms, as shown in the second column of
Table I. This value includes the propagation delay, transmission
delay, and queuing delay. It does not take into account the
processing delay at each node. It is assumed that the processing
delay is negligible because the value is measured on the order
of microseconds. However, in emulation mode, the network
simulator is running in real time, and this processing delay
is taken into account. The estimated online simulation delay
presented in the third column of Table I is therefore calculated
after factoring in the processing delay. The processing delay
is assumed very small at 50 ms, and the estimated value is
0.5809 ms. The fourth column in Table I shows the actual delay
observed in the test bed.
The values presented in Table I are slightly higher than the
estimated delay values. This is partly because, in the final
implementation, data delivery is from nodes outside the ns-3
simulation environment. LAN delays were not included as part
of the estimated delay because the delay calculation is from
gateway to gateway. As such, the LAN delay must be accounted
for in each substation network. The actual delay for Node 0 was
observed to be 0.6551 ms. Hence, it is shown that the values
in Table I agree with what is expected, thereby validating the
test bed.
Fig. 10 presents a graph showing the delay incurred by
a packet when sent from substation to control center with
and without ns-3. The values shown were calculated in the
SEL-3373 station PDC, which is synchronized to a GPS clock
7
Fig. 10. Delay difference with and without ns-3.
source. The values shown are for data transfer from Node 4
to the control center. Hence, it is shown that ns-3 is adding the
communication delays as required. The results presented earlier
are for normal operating conditions where congestion is not an
issue. During congestion cases, the amount of data handled by
the network increases, thereby causing potential for errors in the
emulation. However, it was observed that the network simulator
was able to handle and manage the congestion in the network
as per the network characteristics. It was found that ns-3 was
able to emulate successfully congestion during a DoS kind of
attack simulation on the system using the test bed presented in
the succeeding section.
VI. A PPLICATIONS OF THE T EST B ED
A. Aurora Attack Simulation—Impact Analysis
In 2007, Idaho National Laboratories (INL) demonstrated
an attack on a generator that was termed as the Aurora attack.
The attack involves opening and closing of a breaker associated
with a generator over fixed intervals. The assumption is that
the synchronism-check element is disabled in the relay leading
to the possibility of reclosing the generator onto the system
when it is out of sync [31]. The synchronism-check element is
responsible for ensuring that the breaker does not close unless
the angle difference is within tolerable limits. Fig. 11 shows
the phase difference between the A phase voltage of the system
and voltage measured at the generating transformer. The blue
signal represents the generator transformer’s voltage starting to
separate from the system voltage. Initially, when the breaker
is opened, the voltages start to develop an angle difference.
After ten cycles, the breaker is closed when the voltages are
not in phase with each other. The angle separation between
the voltages is caused by acceleration of the generator upon
breaker opening because the load on the generator is lost.
When the breaker is reclosed, the generator is pulled back into
synchronism. This causes the generator to experience very high
mechanical torque, which can lead to permanent damage.
To simulate the attack in the test bed, a new node is placed
in the communication network where the attacker is assumed
to reside. Attackers are typically present on the Internet from
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

8 IEEE SYSTEMS JOURNAL

Fig. 11. Phase separation of system and generator voltage during an Aurora attack.

Fig. 12. Interface between RTDS and relay.

where they gain access to private networks through hacking.

The attacker is assumed to have hacked into the control center
by connecting to the private network through a compromised
username and password. The attacker obtains access to one of
the resources used for engineering access to the relay, which is
used to control the breaker at Bus 2 of the test system. The relay
is an SEL-421 Protection, Automation, and Control System that
is interfaced with the RTDS and controls the breaker at Bus 2.
Output OUT101 is used to open the breaker, and OUT102 is
used to close the breaker.
The interface between the RTDS and relay for digital signal
exchange is shown in Fig. 12. The attacker can gain access
to the relay settings file, which can be modified for malicious
purposes. The settings file is reprogrammed by the attacker Fig. 13. Current at the generating transformer because of reclosing out of sync.
to open the breaker and to reclose within ten cycles, and the
sequence is repeated. The timing is achieved by the use of
timers in the relay. Two timers are programmed to carry out in Fig. 13 reveals that an overcurrent relay is unlikely to trip for
the attack. The first timer is used to pulse the output contact this variation because the pickup for the instantaneous element
OUT101 every 1 s, resulting in the opening of the breaker. is typically higher. The time-overcurrent element is unlikely to
The second timer is used to pulse OUT102 every ten cycles pick up because of the variation stabilizing quickly. Although
after breaker opening, to reclose the breaker. This sequence is not shown in the figure, it was observed that the frequency
repeated until the generator is damaged. The 1-s interval allows deviation is not very high. This proves that it is possible to cause
enough time for the system to recover and not trip because of an Aurora attack without causing the other protective elements
any other relays picking up the induced fault condition in the to trip.
system. In Fig. 14, it is clear that the large and repeated variation
Figs. 13 and 14 show the impact of the attack described in torque experienced by the system will lead to mechanical
earlier on the operation of the generator. The current plot shown damage in the generator. In a practical system, if successfully
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
VELLAITHURAI et al.: DEVELOPMENT AND APPLICATION OF REAL-TIME TEST BED FOR CPS
300
Real Power (MW)
200
100
60
Reactive Power (MVAR)
40
20
-20
1.5
1
Torque (PU)
0.5
0
-0.5
0 0.33333 0.66667 1 1.33333
Time (s)
Fig. 14. Swing in real power, reactive power, and electrical torque at
Generator 2.
coordinated Aurora attacks were to take place, resulting in N-2
or higher contingencies, it could lead to a blackout situation.
Restoration procedures might also be stalled because of the
unavailability of critical generators lost during the attack. Using
the developed test bed, it is possible to simulate attack scenar-
ios, such as an Aurora attack, in real time. This means that, if a
security layer were to be added to the existing test bed, it would
be possible to do attack–defense scenarios to evaluate defense
mechanisms and algorithms from a security perspective. The
test bed also serves as a means of testing defensive techniques
against attacks. For example, to detect an Aurora type of attack,
the rate-of-change-of-frequency protection device can be used
to determine if the frequency shift is occurring too fast and
therefore block the reclosing of the breaker.
B. Implementation and Testing of Algorithms
1) Wide-Area Voltage Stability Monitoring: The wide-area
voltage stability monitoring algorithm is a PMU data-based
application to be executed at the control center. The algorithm
requires the voltage magnitude and angles at each substation to
calculate the voltage stability index, which is a measure of the
stability of the system. The following devices are used for the
implementation:
PMU: Six hardware PMUs and eight software PMUs are
used to make up the full complement of 14 PMUs required
for the simulation. PDC: A total of 14 separate PDC devices
would be needed if individual PDCs would be used for each
9
Fig. 15. Packets per second during the DoS attack at different locations.
substation. However, because of limited availability of re-
sources, a single PDC is used to emulate up to four substations.
A combination of openPDC and SEL-5073 PDCs are used.
1.66667 2
Relays: The relays are placed at each bus in the system to pro-
vide measurements. Additionally, relays are placed at strategic
locations where overloads are likely to occur. Control Center:
The visualization console shown in Fig. 8 is provided as part of
the voltage stability algorithm.
The implementation is similar to that in Fig. 7, except for the
absence of an SEL-3354 substation computer at the substations.
To illustrate the impact on the performance of the algorithm,
a DoS attack is simulated by flooding the selected node with
attack packets. The bandwidth of the attack is not too high
because flooding with too many packets will cause the node
to go offline. Instead, the bandwidth of the attack is controlled
randomly.
Figs. 15 and 16 show the plot for the number of pack-
ets per second and the number of bytes per second, respec-
tively, obtained from a wireshark packet capture. IP addresses
192.168.3.10, 192.168.4.10, and 192.168.5.10 refer to the
PDCs located at Nodes 1, 2, and 3, respectively. The DoS attack
is simulated for Nodes 2 and 3. At Node 3, the attack bandwidth
is very low and infrequent, which is reflected in the minimal
changes in the plot. At Node 2, the attack bandwidth is higher
and frequent, which is reflected in the larger spikes. Because
there are two PDC streams at Node 3, the number of packets is
doubled, as can be seen at Nodes 1 and 2. Note that the transport
layer protocol used in this simulation is Transmission Con-
trol Protocol. Simulations results showed that an acknowledge
message was transmitted back to the source for every two
packets received. Therefore, 30 acknowledge packets are sent
for every 60 data packets received. Hence, for each PMU
sending data at the rate of 60 samples per second, 90 packets
are expected to be seen. In the wireshark capture, it was
observed that the size of each data packet was 140 B, and each
acknowledge packet size was 54 B for data sent from Node 2.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
10
Fig. 16. Bytes per second during the DoS attack at different locations.
A total of 60 data packets plus 30 acknowledge packets amounts
to 10 020 B, as observed in Fig. 16. During the attack, it was ob-
served that packets were retransmitted with sizes up to 1514 B.
This explains the spike in the bytes per second, whereas the
number of packets observed is relatively lower. For a real-time
algorithm, it is important that data is available as quickly as
possible. Due to the DoS, it was observed that the algorithm
was not able to execute due to data unavailability. Therefore,
the actual state of the system is not known during the attack.
Transient fault conditions typically need to be cleared within
ten cycles (167 ms). If data are unavailable or delayed because
of a cyberattack, the control action will be delayed, leading to
an instability condition.
2) Local-Area Voltage Stability Monitoring: The local-area
voltage stability monitoring algorithm is also a PMU-based
application [32]. It is run at each substation using only the
phasor measurements at that particular bus. It requires the slack
bus angle as a reference to the measurement of the voltage
angle. The following devices are used for the implementation:
PMU: At the slack bus, a software PMU (RTDS) is used,
PDC: At the substation, openPDC is used, and at the slack bus,
SEL-5073 software PDC is used. SEL-3354: SEL-3354 is a
substation computer. In this paper, this device is used to retrieve
data from the PDC database and execute the proposed real-time
voltage stability algorithm. The computed index, VSAI, along
with the other measured parameters, is sent to the control center
for visualization. Relays: The inverse-time overcurrent element
of the SEL-421 Relays is used to monitor and trip transmission
lines in case an overcurrent is detected. The relays monitor the
transmission lines in the test case.
The setup for the implementation of this algorithm is similar
to that shown in Fig. 7. The algorithm is implemented at Bus 12,
which corresponds to Node 7 in the communication network
topology. Table II lists the propagation delay to be emulated
by ns-3 for the given implementation. To illustrate the cyber
IEEE SYSTEMS JOURNAL
TABLE II
P ROPAGATION D ELAY B ETWEEN G ATEWAY
Fig. 17. Change in routing of packets between Nodes 0 and 4.
impact for this algorithm, the possibility of a communication
line failure was considered. The communication line between
Nodes 0 and 4 is selected for failure simulation.
Initially, all the lines are considered to be in service. After 60 s,
the communications line between Nodes 0 and 4 goes offline.
This leads to the network being reconfigured. Fig. 17 shows
the information flow path between Nodes 0 and 4. Here, the
counter refers to the seconds at which the data were printed.
The data printed in Fig. 17 are only for transmission of data
packets through that interface and not for reception. At Node 0,
Interface 3 is used for transmission to Node 4, and Interface 2
is used for transmission to Node 1. At Node 4, Interface 2
is used for communication with Node 0, and Interface 3 is
used for communication with Node 1. Fig. 17 shows that,
At the 58th second, acknowledge packets are greater in bytes
through Interface 3 at Node 0. However, at the 61st second,
after loss of communication line, Interface 3 has no data flow.
Additionally, the data received at Node 4 has decreased because
of reconfiguration of the data flow path. During this process, it
was observed that there was a momentary addition in the delay
experienced by the packets. Since the system has redundant
communication lines, the loss of a single communication line
did not pose any major problems.
3) Training and Learning: The test bed operates in real time,
meaning that any change made in the physical or cyber system
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
VELLAITHURAI et al.: DEVELOPMENT AND APPLICATION OF REAL-TIME TEST BED FOR CPS
causes dynamics, as in a real system. This provides an ideal
platform for training and learning purposes. Test scenarios may
be created where a sequence of cyber–physical contingencies
are simulated and the operator’s response is studied. Hence,
the interrelated dynamics of the power and cyber systems can
be studied. In [33], a method for operator training using the
IncSys PowerSimulator was presented. Because of the absence
of a control center, such as HMI screens in this test bed, it is not
ideal for operator training at this point. However, it is useful for
teaching and learning purposes.
VII. C ONCLUSION
To understand the coupling between cyber and physical
systems in a cyber–physical system such as the smart grid,
the development of test beds, which can model the system
accurately, is required. An overview of the existing test bed
and the developed real-time cyber–physical test bed using ns-3
and RTDS are provided in this paper. The test bed uses a com-
bination of simulated, emulated, and physical devices, which
allows for an easily reconfigurable system. Different types of
traffic, such as DNP3 and C37.118.1 PMU data, can be passed
through the cyber network emulated in ns-3. The scalability of
the developed test bed is limited by the available hardware ca-
pabilities. Hardware capabilities depend on financial resources.
The cost of an RTDS upgrade is the highest to model bigger
test systems, but another EMTP-based simulators with capa-
bilities similar to the RTDS can be directly substituted. ns-3
by itself is scalable subject to computing facilities used for
implementing ns-3. Hardware devices that are interfaced are
also subject to financial resources. A control-center software
tool can easily be scaled up to analyze a much bigger system,
subject to computational facilities. The developed interface
and methodology for integrating all the components for the
integrated real-time test bed are scalable.
The practical application of the test bed for impact analysis is
demonstrated through the description of an Aurora type of gen-
erator attack that is simulated in the test bed. For applications
to be deployed in critical cyber–physical systems, such as the
EPG, test beds assist in the testing and validation processes.
Manipulation of the cyber network topology and simulation
of cyberattacks, such as DoS attacks and false-data injection
attacks, aid in understanding the cyber impacts on a smart-grid
application. The implementation and testing of applications is
demonstrated through the practical implementation of the local-
and wide-area voltage stability monitoring algorithms. Future
work will involve the exploration of the impact of more sophis-
ticated coordinated attacks and the various impact mitigation
effort through a cyber–physical approach. The next step in the
test-bed development process is to add a security layer, which
would reflect a real-world scenario. By defining access policies
and firewall rules for each private network, the attack–defense
mechanism testing can be made possible. Recent developments
in ns-3, with the implementation of network address translation
(NAT) and Netfilter, provide a basic framework for the imple-
mentation of a security layer. These improvements are still in
development and need validation and testing before deployment
in the test bed.
11
R EFERENCES
[1] C. Cecati, C. Citro, A. Piccolo, and P. Siano, “Smart operation of wind
turbines and diesel generators according to economic criteria,” IEEE
Trans. Ind. Electron., vol. 58, no. 10, pp. 4514–4525, Oct. 2011.
[2] Y. Mo et al., “Cyber – physical security of a smart grid infrastructure,”
Proc. IEEE, vol. 100, no. 1, pp. 195–209, Jan. 2012.
[3] V. Gungor et al., “Smart grid technologies: Communication technologies
and standards,” IEEE Trans. Ind. Informat., vol. 7, no. 4, pp. 529–539,
Nov. 2011.
[4] “A systems view of the modern grid,” U.S. Dept. Energy (DOE), Nat.
Energy Technol. Lab. (NETL), Washington, DC, USA, 2007.
[5] “Critical infrastructure protection challenges and efforts to secure con-
trol systems,” U.S. Gov. Accountability Office, Washington, DC, USA,
Mar. 2007.
[6] Y. Liu, P. Ning, and M. K. Reiter, “False data injection attacks against
state estimation in electric power grids,” in Proc. 16th ACM Conf. CCS,
New York, NY, USA, 2009, pp. 21–32.
[7] L. Xie, Y. Mo, and B. Sinopoli, “False data injection attacks in elec-
tricity markets,” in Proc. IEEE Int. Conf. SmartGridComm, Oct. 2010,
pp. 226–231.
[8] H. Khurana, M. Hadley, N. Lu, and D. Frincke, “Smart-grid secu-
rity issues,” IEEE Trans. Security Privacy, vol. 8, no. 1, pp. 81–85,
Jan./Feb. 2010.
[9] “Advisory, ‘dnp3 implementation vulnerability’,” ICS-CERT, Washington,
DC, USA, Oct. 2013.
[10] T. Ernster and A. Srivastava, “Power system vulnerability analysis—
Towards validation of centrality measures,” in Proc. IEEE PES Transmiss.
Distrib. Conf. Expo., May 2012, pp. 1–6.
[11] J. Kim and L. Tong, “On topology attack of a smart grid: Undetectable
attacks and countermeasures,” IEEE J. Sel. Areas Commun., vol. 31,
no. 7, pp. 1294–1305, Jul. 2013.
[12] S. Han, M. Xie, H.-H. Chen, and Y. Ling, “Intrusion detection in cyber-
physical systems: Techniques and challenges,” IEEE Syst. J., vol. 8, no. 4,
pp. 1049–1059, Dec. 2014.
[13] S. Muller, H. Georg, C. Rehtanz, and C. Wietfeld, “Hybrid simulation of
power systems and ICT for real-time applications,” in Proc. 3rd IEEE PES
Int. Conf. Exhib. ISGT Europe, Oct. 2012, pp. 1–7.
[14] H. Georg, C. Wietfeld, S. Muller, and C. Rehtanz, “A HLA based sim-
ulator architecture for co-simulating ICT based power system control
and protection systems,” in Proc. IEEE 3rd Int. Conf. SmartGridComm,
Nov. 2012, pp. 264–269.
[15] K. Hopkinson et al., “Epochs: A platform for agent-based electric
power and communication simulation built from commercial off-the-shelf
components,” IEEE Trans. Power Syst., vol. 21, no. 2, pp. 548–558,
May 2006.
[16] A. Hahn, A. Ashok, S. Sridhar, and M. Govindarasu, “Cyber-physical se-
curity testbeds: Architecture, application, and evaluation for smart grid,”
IEEE Trans. Smart Grid, vol. 4, no. 2, pp. 847–855, Jun. 2013.
[17] C. Queiroz, A. Mahmood, and Z. Tari, “Scadasima framework for building
SCADA simulations,” IEEE Trans. Smart Grid, vol. 2, no. 4, pp. 589–597,
Dec. 2011.
[18] K. Anderson and A. Narayan, “Simulating integrated volt/var control
and distributed demand response using gridspice,” in Proc. IEEE 1st Int.
Workshop SGMS, Oct. 2011, pp. 84–89.
[19] D. Anderson et al., “A virtual smart grid—Real-time simulation for smart
grid control and communications design,” IEEE Power Energy Mag.,
vol. 10, no. 1, pp. 49–57, Jan./Feb. 2012.
[20] H. Gjermundrod, D. Bakken, C. Hauser, and A. Bose, “Gridstat: A flexible
QOS-managed data dissemination framework for the power grid,” IEEE
Trans. Power Del., vol. 24, no. 1, pp. 136–143, Jan. 2009.
[21] J. Nutaro, “Designing power system simulators for the smart grid: Com-
bining controls, communications, and electro-mechanical dynamics,” in
Proc. IEEE PES Gen. Meet., Jul. 2011, pp. 1–5.
[22] K. Mets, T. Verschueren, C. Develder, T. Vandoorn, and L. Vandevelde,
“Integrated simulation of power and communication networks for smart
grid applications,” in Proc. IEEE 6th Int. Workshop CAMAD, Jun. 2011,
pp. 61–65.
[23] M. Mallouhi, Y. Al-Nashif, D. Cox, T. Chadaga, and S. Hariri, “A Testbed
for Analyzing Security of SCADA Control Systems (TASSCS),” in Proc.
IEEE PES ISGT, Jan. 2011, pp. 1–7.
[24] C. M. Davis et al., “Scada cyber security testbed development,” in Proc.
38th North Amer. Power Symp., Sep. 2006, pp. 483–488.
[25] “National Scada Test Bed: Fact Sheet,” Idaho Nat. Lab. (INL), Idaho
Falls, ID, USA, 2007.
[26] M. McDonald, “Modeling and for Cyber-physical System Security Re-
search,” Dev. Appl., Sandia Nat. Lab., Livermore, CA, USA, Feb. 2010.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
12
[27] H. Lin, S. Veda, S. Shukla, L. Mili, and J. Thorp, “GECO: Global
event-driven co-simulation framework for interconnected power system
and communication network,” IEEE Trans. Smart Grid, vol. 3, no. 3,
pp. 1444–1456, Sep. 2012.
[28] P. Forsyth, T. Maguire, and R. Kuffel, “Real time digital simulation for
control and protection system testing,” in Proc. IEEE 35th Annu. Power
Electron. Spec. Conf., Jun. 2004, vol. 1, pp. 329–335.
[29] C. Vellaithurai, “Cyber-Power System Analysis Using a Real Time
Test Bed,” M.S. Thesis, Washington State Univ., Pullman, WA, USA,
Jul. 2013.
[30] P. Anderson and A. Fouad, “Power System Control and Stability,” Iowa
State Univ. Press: Ames, IA, USA, 1977.
[31] A. Srivastava et al., “Modeling cyber-physical vulnerability of the smart
grid with incomplete information,” IEEE Trans. Smart Grid, vol. 4, no. 1,
pp. 235–244, Mar. 2013.
[32] S. Biswas, C. Vellaithurai, and A. Srivastava, “Development and real time
implementation of a synchrophasor based fast voltage stability monitoring
algorithm with consideration of load models,” in Proc. IEEE Ind. Appl.
Soc. Annu. Meet., Oct. 2013, pp. 1–9.
[33] C. Vellaithurai, A. Srivastava, and S. Zonouz, “SECPSIM: A training
simulator for cyber-power infrastructure security,” in IEEE Int. Conf.
SmartGridComm, Oct. 2013, pp. 61–66.
Ceeman B. Vellaithurai (S’09–M’12) received
the B.E. degree in electrical and electronics en-
gineering from Anna University Tiruchirappalli,
Tiruchirappalli, India, in 2011 and the M.S. degree
in electrical engineering with specialization in power
systems from Washington State University, Pullman,
WA, USA, in 2013.
He is currently working with Schweitzer Engi-
neering Laboratories Inc., Pullman, as a Protection
Engineer. His research interests include real-time
modeling and simulation of cyber–power systems.
Mr. Vellaithurai received the Best Outgoing Student Award from Anna
University Tiruchirappalli for his academic achievements.
IEEE SYSTEMS JOURNAL
Saugata S. Biswas (S’12) received the B.E. degree
in electrical engineering from Nagpur University,
Maharashtra, India, in 2007 and the Ph.D. degree
from Washington State University, Pullman, WA,
USA, in 2014.
From 2007 to 2009, he was with the Design and
Development Department of a switchgear industry
in India. From 2009 to 2010, he was a Ph.D. Stu-
dent with Mississippi State University, Starkville,
MS, USA, before continuing his doctoral study at
Washington State University. He is currently work-
ing with Alstom, Bellevue, WA, USA.
Dr. Biswas received several Gold Medal awards from Nagpur University
for his academic achievements from 2003 to 2007, and the EECS Outstanding
Ph.D. Student in Electrical Engineering Award from Washington State Univer-
sity in 2013.
Anurag K. Srivastava (S’00–M’05–SM’09) re-
ceived the Ph.D. degree from Illinois Institute of
Technology, Chicago, IL, USA, in 2005.
Since August 2010, he has been with Washington
State University, Pullman, WA, USA, as an Assistant
Professor. From 2005 to 2010, he was an Assistant
Research Professor with Mississippi State Univer-
sity, Starkville, MS, USA. His research interests
include power-system operation and control using
smart-grid data.
Dr. Srivastava served as the Chair of the IEEE
Power and Energy Society (IEEE PES) Career Promotion Subcommittee and
as the Chair of the IEEE PES Student Activities Subcommittee. He currently
serves in other several IEEE PES Technical Committees and as an Associate
Editor for the IEEE T RANSACTIONS ON S MART G RID, and as an IEEE
Distinguished Lecturer.