1

Vijay Kumar
UPS Foundation Professor
University of Pennsylvania
Aerial Robot Swarms
University of Minnesota
April 16, 2014
2
Unmanned Aerial Vehicles
Mass
0 1 10 100 1,000 10,000 100,000
Northrop-Grumman
Global Hawk
(32,200 lbs)
Boeing X-45A UCAV (12,000
lbs)
Bell Eagle Eye (2,250 lbs)
AscTec
Pelican
(3.5 lbs)
Hummingbird (1 lb)
KMel kNanoQuad
(0.12 lb)
Gen. Atomics MQ-9
Reaper (10,000 lbs)
Gen. Atomics Predator (7,000 lbs)
Gen. Atomics
Predator (2,250
lbs)
Images from www.af.mil
Boeing
Scaneagle
(40 lbs)
3
Unmanned Aerial Vehicles



> $10B industry
! Military: Surveillance, force protection, warfare (> 75 countries)
! Civilian commercial: Transportation, agriculture, mining, security
! Civilian private: DIY Drones






Increasingly smaller, lower power and inexpensive
inertial sensing, GPS, and computing
1
10
100
1000
10000
1980 1990 2000 2010
Number of UAVs
worldwide
4
Search and Rescue
Michael, et al, J. Field Robotics, 2012
5
Security and First Response
6
Security and First Response
Flight through building
7
Real Time Agrylitics
8
Apple Orchards,
Pennsylvania State University
Biglerville, PA
9
10
Quadrotor
11
Pico
11 cm
20 g,
6.5 Watts
Max speed 6m/s
12
$10M
$1M
$100
$10
$1
Global Hawk
Predator, Reaper
Scan Eagle, Raven
Hobby Kits
Toys
Chips
$100K
$1K-10K?
Autonomous,
agile micro aerial
vehicles
our focus
GPS, airbag sensors,
processors
DIY Drones, Kick starter projects
10s
100s
1,000,000s
100,000s
10,000s
13
Length Scales
13
!
!#$
!#%
!#&
!#'
!#(
!#)
!#*
!#+
! !#!% !#!' !#!) !#!+ !#$ !#$% !#$'
,-./ 012345
64078/ 92:
;<3<344
=4>?.-0
@122?073?5A
B-0< =>1C
B-0<
==;
=?.<
3-D printing
Folding (w MIT)
Pop-up book MEMS (Wood)
v

L
14
[Kushleyev, Mellinger and Kumar, RSS 2012]
The KMel Nano Quadrotor
15
16
Size does Matter
a
1
R
,
1
R
2
a
1
R
,
1
R
2
linear acceleration angular acceleration
mass M
inertia I
r
R
thrust f
17
1 Dynamics, Control and Planning
2 State Estimation (Autonomy)
3 Swarm technology
Research Challenges
18
Quadrotor
19
x
X
Y
Z
r
!
1

!
2

!
3

!
4

y
#, yaw
$, pitch
%, roll
x, y, z position
Underactuated
z
Control
R
20
x
X
Y
Z
r
!
1

!
2

!
3

!
4

y
#, yaw
$, pitch
%, roll
x, y, z position
Underactuated
z
Control
R
21
Differential Flatness
[x, y, z, #]
A
B
A
B
Minimum snap trajectory
[Mellinger and Kumar, 2011]

x
y
z

x
y
z

min
(t)
Z
T
0
k
....
r (t)k
2
+

(t)
2
dt
SE(3)
22
Control on SE(3)
Specied trajectory
(t) : [0, T] R
3
SO(2)
x

y

z

t
t
R
des
e
3
=
t
t
=
des
R
des
#
des

M = I + I(k
R
e
R
k

)
e
R
(R
des
, R)
k
f = m(r +K
v
e
r
+K
p
e
r
+ g)
. k
[Wen and Kreutz-Delgado, 1991; Lee et al, 2011; Mellinger and Kumar, 2011]
23
Software Architecture
[Kraft 2003; Mellinger, Michael, and Kumar 2010;
Mellinger and Kumar 2011]
Rigid body
dynamics
Motor
controller
Attitude
controller
Position
controller
Trajectory
Planner
R
des

&
des

R, '
r, r
f
M
Basin of
attraction
almost all of
SO(3)
~0.001 s
~0.01 s
~0. 1 s
24
Roll and Pitch
25
Robust, nonlinear controller
tr[I (R
des
)
T
R] < 2 e

(0)
2

min
(I)
k
R

1
1
2
tr

I (R
des
)
T
R

26
Translation
27
[Mellinger and Kumar, ICRA 2011]
Minimum Snap Trajectories
A
C
B
28
Minimum Snap Trajectories
[Mellinger and Kumar, ICRA 2011]
29
[Thomas et al, ICRA 2014]
Aerial Grasping and Manipulation
30
motion capture
cameras
reective markers
Sensing
31
Photograph by Joe McNally
32
Unreliable GPS
33
No GPS
34
1 Dynamics, Control and Planning
2 State Estimation (Autonomy)
3 Swarm technology
35
Microsoft
Kinect
Hokuyo
Laser
Scanner
Operation in Unstructured
Environments
36
x
d
1
d
2
d
3
d
0
3
d
0
2
d
0
1
Pillars
Robot
(x
1
, y
1
)
(x
3
, y
3
)
(x
2
, y
2
)
Concurrently estimate
Locations of pillars (6)
Displacement of the robot (2)
Simultaneous
Localization
And
Mapping
New robot
position
Simultaneous
Localization
And
Mapping
37
1.8 GHz Core i3 processor, 8 GB
RAM
u- blox LEA-6T GPS module
Hokuyo UTM-30LX LiDAR
2 mvBlueFOX-MLC200w grayscale
HDR cameras
(sheye lenses, 752 " 480, 25 Hz)
IMU 100 Hz
[Shen, Mulgaonkar, Michael, and Kumar 2014]
38
Heterogeneous Sensors
Proprioceptive Sensors
! Inertial Measurement Unit

Absolute Sensors
! GPS, Magnetometer
! Altimeter
Relative Sensors
! Laser scanner
! Cameras
z
t
= h(x
t
) +n
t
z
t
= h(x
t
, . . . , x
tk
) +n
t
x
t
= f(x
t1
, u
t1
) +n
t
39
f
2


f
1


f
0

Features in environment
x
0

x
1
f
3


f
4


x
2
x
3
f
6


z
0
0
z
1
0
z
2
0
z
2
1
z
1
1
z
3
2
z
3
3
z
2
3
z
4
3
z
6
3
x
1
= f(x
0
, u
01
, t
01
)
x
2
= f(x
1
, u
12
, t
12
)
x
3
= f(x
2
, u
23
, t
23
)
Robot
Pose
Simultaneous
Localization
And
Mapping
Simultaneous
Localization
And
Mapping
40
Map
Refine (20 Hz)
Pose Graph
SLAM
position
Multi-Sensor
Unscented
Kalman
Filter (100 Hz)
velocity
Estimation and Control Architecture
Trajectory
Generator (20 Hz)
Planner
(20 Hz)
User
Interface
Velocity
estimator
Visual
odometry
Laser
odometry
Altitude
estimator
Downward
Camera (30Hz)
Stereo
Camera (25 Hz)
IMU
(100 Hz)
Pressure
Altimeter (20 Hz)
Laser Scanner
(20 Hz)
GPS (10Hz)
Controller
(100 Hz)
41
Onboard State Estimation
[Shen, Michael, and Kumar 2011]
IMU, Laser scanner, and camera
Auton Robot
Lee, T. (2011). Geometric tracking control of the attitude dynamics of
a rigid body on SO 3. In Proc. of the Amer. control conf., San
Francisco, CA.
Lee, T., Leok, M., & McClamroch, N. H. (2010). Geometric tracking
control of a quadrotor UAV on SE 3. In Proc. of the IEEE conf.
on decision and control, Atlanta, GA.
Mellinger, D., & Kumar, V. (2011). Minimum snap trajectory genera-
tion and control for quadrotors. In Proc. of the IEEE intl. conf. on
robot. and autom., Shanghai, China.
Mellinger, D., Michael, N., & Kumar, V. (2010). Trajectory generation
and control for precise aggressive maneuvers with quadrotors. In
Proc. of the intl. sym. on exp. robot., Delhi, India.
Mesbahi, M. (2005). On state-dependent dynamic graphs and their
controllability properties. IEEE Transactions on Automatic Con-
trol, 50(3), 387392.
Michael, N., Mellinger, D., Lindsey, Q., & Kumar, V. (2010). The
GRASP multiple micro UAV testbed. IEEE Robotics & Automa-
tion Magazine, 17(3), 5665.
Nieuwstadt, M. J. V., & Murray, R. M. (1998). Real-time trajectory
generation for differentially at systems. International Journal of
Robust and Nonlinear Control, 8(11), 9951020.
Ogren, P., Fiorelli, E., & Leonard, N. (2002). Formations with a mis-
sion: stable coordination of vehicle group maneuvers. In Proc.
of intl. sym. on mathematical theory networks and syst., Notre
Dame, IN.
Olfati-Saber, R., & Murray, R. M. (2002). Distributed cooperative con-
trol of multiple vehicle formations using structural potential func-
tions. In Proc. of the IFAC world congress, Barcelona, Spain.
Olfati-Saber, R., & Murray, R. M. (2004). Consensus problems in net-
works of agents with switching topology and time-delays. IEEE
Transactions on Automatic Control, 49(9), 15201533.
Shim, D., Kim, H., & Sastry, S. (2003). Decentralized nonlinear model
predictive control of multiple ying robots. In Decision and
control, 2003. Proceedings. 42nd IEEE conference on (Vol. 4,
pp. 36213626). New York: IEEE.
Tabuada, P., Pappas, G. J., & Lima, P. (2001). Feasible formations of
multi-agent systems. In Proc. of the Amer. control conf., Arling-
ton, VA (pp. 5661).
Tanner, H., Pappas, G. J., & Kumar, V. (2002). Input-to-state stability
on formation graphs. In Proc. of the IEEE intl. conf. on robot. and
autom., Las Vegas, NV (pp. 24392444).
Turpin, M., Michael, N., & Kumar, V. (2011). Trajectory design and
control for aggressive formation ight with quadrotors. In Proc.
of the intl. sym. of robotics research, Flagstaff, AZ.
Vicsek, T., Czirk, A., Ben-Jacob, E., Cohen, I., & Shochet, O. (1995).
Novel type of phase transition in a system of self-driven particles.
Physical Review Letters, 75(6), 12261229.
Matthew Turpin is a Ph.D. candi-
date in the Department of Mechan-
ical Engineering and Applied Me-
chanics at the University of Penn-
sylvania. He works on formation de-
sign and control of micro-aerial ve-
hicles.
Nathan Michael is a Research As-
sistant Professor in the Department
of Mechanical Engineering and Ap-
plied Mechanics at the University of
Pennsylvania. He received his Ph.D.
in Mechanical Engineering from the
University of Pennsylvania in 2008.
He works in the areas of dynamics,
estimation, and control for ground
and aerial robot systems.
Vijay Kumar is the UPS Founda-
tion Professor and the Deputy Dean
for Education in the School of En-
gineering and Applied Science at
the University of Pennsylvania. He
received his Ph.D. in Mechanical
Engineering from The Ohio State
University in 1987. He has been
on the Faculty in the Department
of Mechanical Engineering and Ap-
plied Mechanics with a secondary
appointment in the Department of
Computer and Information Science
at the University of Pennsylvania
since 1987. He is a Fellow of the
American Society of Mechanical Engineers (ASME) and the Institu-
tion of Electrical and Electronic Engineers (IEEE).
42
Autonomous Exploration
IMU, Laser scanner, and RGBD camera
43
44
Lithium Polymer Batteries
0
200
400
600
800
1000
1200
1400
0 50 100 150 200 250 300 350 400 450 500
S
p
e
c
i

c

P
o
w
e
r

(
W
/
k
g
)

Specic Energy (Whr/kg)
Quadrotors with
10-20 min
endurance
[B. Morgan, ARL and Y. Mulgaonkar, Penn]
Bolt (10
secs)
Armstrong
(20 mins)
Loose
weight!
45
Power
0
200
400
600
800
1000
1200
1400
0 50 100 150 200 250 300 350 400 450 500
S
p
e
c
i

c

P
o
w
e
r

(
W
/
k
g
)

Specic Energy (Whr/kg)
10,000
Whr/kg
Lithium Polymer Batteries
Quadrotors with
10-20 min
endurance
[B. Morgan, ARL and Y. Mulgaonkar, Penn]
Bolt (10
secs)
Armstrong
(20 mins)
Loose
weight!
46
Reducing the Payload
CPU: Intel Atom Processor, 1.6 GHz, 1 GB Ram
Sensing: 2 grayscale Matrix Vision cameras,
376x240 + IMU


Weight: 740gram
Power: ~120 W
2012
[Shen, Mulgaonkar, Michael, and Kumar 2013]
47
Vision + IMU State Estimation
! Estimate rotations by tracking features
! Assuming features are known, estimate position
! Estimate scale with second camera
! Assuming pose is known, map features
48
Vision + IMU State Estimation
49
[Shen, Mulgaonkar, Michael, and Kumar,
2013]
Fast (4 m/s), Indoor
Outdoor, foliage
Indoor, 3-D Indoor/outdoor, visual SLAM
50
51
1 Dynamics, Control and Planning
2 State Estimation
3 Swarm technology
52
Collaboration in Small Teams
[Lindsey, Mellinger, and Kumar, 2012]
53
1 Act independently
2 Require only local information
3 Anonymous behavior
54
robot i
robot j
g(t) SO(2) R
3
s
i,j
(t) = x
j
(t) x
i
(t)
Leader-Follower Networks
[Desai, Ostrowski, and Kumar, 1999; Turpin, Michael and Kumar, 2011]
57
Control of Formation Shape and Group Motion
(Turpin, Michael, and Kumar, 2013)
58
[Kushleyev, Mellinger and Kumar 2012]
Control of Formation Shape and Group Motion
59
Search and Rescue
[Michael et al, 2012]
N. Michael, S. Shen, K. Mohta, Y. Mulgaonkar, V. Kumar, K. Nagatani, Y.
Okada, S. Kiribayashi, K. Otake, K. Yoshida, K. Ohno, E. Takeuchi, and S.
Tadokoro, Collaborative mapping of an earthquake-damaged building
via ground and aerial robots, J. Field Robotics, vol. 29, no. 5, pp. 832
841, 2012.
60
7
th
, 8
th
, and 9
th
oors
61
Thank you!
The contributions and collaborations of the
f ol l owi ng col l eagues ar e gr acef ul l y
acknowledged: A. Wolisz, K. Pister, R.
Brodersen, E. Alon, G. Kelson, A. Niknejad, B.
Nikolic, J. Wawrzynek, P. Wright, D. Tse, M.
Maharbiz, J. Carmena, R. Knight, L. Van de
Perre, and B. Gyselinckx, R. Muller, M. Mark,
D. Chen, A. Parsha, S. Gambini, and all my
current and past graduate students.

Cont ri but i ons by t he BWRC member
companies and the MuSyC and GSRC
consortia are truly appreciated.

Many thanks to V. Vinge, N. Stephenson, I.
Asimov, and C. Stross for providing the true
visions!