International Public

Transport Benchmarking:
Learning from others
Ben Condry, Senior Research Associate
RTSC, Imperial College London

University of Sydney, 10th December 2013 Community of Metros

CoMET
 Key elements of the presentation

 Introduction to the RTSC at Imperial College

 History of the benchmarking groups and who
participates
 Why operators participate
 How the process works and examples
 Benefits
 Example Key Performance Indicators
 Conclusions – why the benchmarking
process has become so useful

2
Imperial College London and the RTSC

 Imperial College London

 Founded in 1907, now ranked 3rd in Europe
 Focus: Science, technology, engineering and medicine
 14,000 students and 4,000 staff

 Railway and Transport Strategy Centre (RTSC)

 Founded in 1992 with initial funding from British Rail
 Part of the Centre for Transport Studies in the
Department of Civil & Environmental Engineering
 Benchmarking, operations, economics and policy
 International team of 20 public transport experts

3
RTSC history and experience – 19 years of successful
worldwide benchmarking projects
Community of Metros
1994 Group of Five metros
CoMET
1996 Community of Metros (CoMET) for large metros

1998 Nova group for medium-sized metros

2004 International Bus Benchmarking Group (IBBG) for large urban

bus operators

2010 International Suburban Rail Benchmarking Group (ISBeRG)

2011 American Bus Benchmarking Group (ABBG) for mid-sized bus

operators in North America

Significant benefits led to groups being continuous

projects: e.g. Hong Kong; New York, London: CoMET
members for 19 years, IBBG members for 8 years

4
CoMET and Nova Metro benchmarking groups
Now 31 metros world-wide, including most of the largest

London DLR
Brussels Moscow
Newcastle
Berlin
Toronto London
Underground Paris Istanbul Beijing
Montreal
Naples Nanjing Shanghai
Lisbon Barcelona
New York Guangzhou
Madrid Taipei

Delhi Hong Kong

Mexico City
Bangkok
Kuala Lumpur

Singapore

Rio de Janeiro
Sao Paulo

Sydney
Trains
Santiago
Buenos Aires

CoMET metros Nova of

Community metros
Metros

CoMET
Less than 500 million passengers p.a. More than 500 million passengers p.a
5
 ISBeRG International Suburban Rail Benchmarking
Group 15 Railways, 14 Cities, 6 Continents

Oslo
(NSB)
Copenhagen
(S-Tog)

London
Munich
(Overground) Tokyo
(S-Bahn)
New York (JR East)
San Barcelona
Francisco (Long Island Rail Road, Hong Kong
(FGC)
(BART) Metro-North) (MTR)

Sydney Brisbane
(Sydney Trains) (Queensland Rail)
Sao Paulo
(CPTM)
Melbourne
Cape Town (PRASA)
(Metro Trains)

Group size: Large enough to find diverse practices, small

enough to maintain close contact between individuals

6
 International Bus Benchmarking Group
12 Member Cities, 13 Operators

Dublin
Vancouver Brussels
Montreal Paris
Istanbul
Seattle London

New York (2) Barcelona

Lisbon

Singapore

7
 American Bus Benchmarking Group: 17 Members Across the
US in a Wide Range of Urban and Suburban Environments
Spokane Transit
(Spokane)
C-Tran
(Vancouver)
RGRTA
Lane Transit DART NFTA (Rochester)
(Eugene) (Des Moines) (Buffalo)
Pace GCRTA
UTA (Cleveland)
(Chicago) RIPTA
San Joaquin RTD (Salt Lake City)
(Rhode Island)
(Stockton) RTA
(Dayton)

Omnitrans
(San Bernardino)
The T
(Fort Worth)
Capital Metro
(Austin)
LYNX
(Orlando)
PSTA
(St. Petersburg)

8
Why do major transport operators find benchmarking so
valuable?
 Public transport essential for cities to function effectively
 Our operators spend over £40bn (70bn AUD) per
annum, using scarce public funds
 Growing expectations demand modern, safe, reliable
and efficient networks.
 Limited opportunities for operators to gauge
performance locally
 Benchmarking is a key tool for operators to see if they
are operating optimally, and how to improve

Role of the benchmarking groups:

 analytical work to understand performance
 disseminate and advise organisations on best practice

9
Many common strategic and technical challenges -
Benchmarking has become essential to help address these
Top Strategic Challenges for Metros

Technology can help

address many
challenges:
benchmarking helps
share experience

10
 For members, the benchmarking groups are central to
proactive, effective continuous improvement

Identify high
Why do others do priority issues
things differently? and areas for What has
Can we lean from improvement worked (or not)
them? elsewhere?
“The Search for Best Practices that
Lead to Superior Performance”
Best practice
Informed
transfer and
dialogue with
implementation
Set challenging but stakeholders
achievable
performance targets

11
 Key principles of the benchmarking groups:
Confidentiality, Collaboration, Speed and Independence

 Benchmarking programme owned and

steered by the members
 RTSC, Imperial College carries out
management and analysis
 Continuous, annual cycle
 Group chair rotated on an annual basis
 Confidentiality to allow full data and
information exchange in the groups only
 Complementary to other industry bodies
(e.g. UITP)

12
 The benchmarking process: a continuous annual cycle
80

Key Performance Indicator system 70

60

- to compare performance and show 50

40

where to look for best practices 30

20

10

Drill-Down Studies - In-depth

research on topics of common Rolling Stock Reliability
interest, to identify best practices, Customer Information
often with Expert Workshops Dwell Time Management

Website with Online Forum –

Transport peers consult with each
other, providing quick answers

Two Meetings per Year - typically

attended by senior directors

13
KPI System – measures “whole system” performance
based on the Balanced Scorecard
Growth, Learning & Innovation
% change Network Size & Passenger Journeys
% change Operated Capacity km & Car km
Number of Training Hours / 1000 Staff Hours
Non-fare Commercial Revenue /Fare Revenue &
/Passenger Journey
Financial
Total Commercial Revenue / Operating Cost
Operating Cost / Revenue Car km
Service Operations Cost / Car km
Maintenance Cost / Car km
Administrative cost / Car km
Investment cost / Car km
Operating Cost / Passenger Journey & km
Fare Revenue / Passenger Journey & km
Customer
Capacity Provision & Utilisation
Capacity km / Route km
Passenger km / Capacity km
Service Quality
Passenger Hours’ Delay / Passenger Journey
Passenger Journeys On Time / Passenger Journey
Trains On Time / Total Trains (scheduled + actual)
Train Hours Operated / Hours of Train Delay
Internal Processes
Reliability & Availability
% of Cars Available & Used in Peak Hour
Car km / hours between Incidents (by category)
Efficiency
Passenger Journeys / Staff + Contractor hours
Capacity & Car km / Staff + Contractor hours
Train hours / Driver Hours
% Employee Absenteeism
Traction Energy Consumed / Car km
Total Energy Consumed / Passenger Journey & km
Safety & Security
Total Fatalities / Passenger Journeys
Deaths from Suicide / Passenger Journeys
Deaths from Accidents / Passenger Journeys
Deaths from Illegal Activity / Passenger Journeys
Incidences of Crime / Passenger Journeys
Staff Lost Time through Accidents / Staff Hours
Environment
CO2 per Passenger km
14
 Purpose and use of Key Performance Indicators

 Benchmarking is NOT simply a comparison of data or

a creation of league tables

 The structured KPI comparisons are used for:

 Stimulating productive “why” questions / lines of inquiry

 Identifying high priority problems, strengths and weaknesses
 Identifying trends: who has truly improved performance over
time and why (up to 19 years of data)?
 Internal motivation – setting realistic targets for performance
 Supporting dialogue with Government and other
stakeholders

15
KPIs are just the starting point – the greatest value
comes from applying Best Practice leant from others

KPIs identify variance in

performance
(between members and over time)

Case Studies to understand the

Statistical analysis and
processes and context behind
modelling identifies drivers
the numbers

Identification and Application

of Best Practice

16
A KPI which describes a lot about benchmarking
Is such disparity in performance real?
Car km Between Incidents Causing a Delay > 5 Minutes to Service (2011)
Research Challenge: identify and quantify factors within and outside
management control that affect reliability

Essential to reach comparability: takes several annual cycles

We apply statistical modelling + deep investigation of operational,
technology, engineering and maintenance processes

Are EU metros such poor performers given circumstances?

Modelling: performance often ‘average’ given structural factors

As = Asia Eu = Europe Am = Americas

17
Line-level econometrics benchmarking of reliability shows the
impact of technology and external factors such as asset age
Parameter % Change Delay incidents

+1 year rolling stock age + 0.7-2%

+1 peak tph + 3.5%

+1 tph practical capacity - 5%

Manual > ATO - 26%

+10% passenger journeys

+ 3.0%
Melo, P.C., Harris, N.G., Graham, D.J., Anderson, R.J., and Barron, A., 2011,Determinants of
Delay Incident Occurrence of Urban Metros, Transportation Research Record (2216), pp.10-18.

18
 The numerical benchmarking has to be combined with
“Best Practice Benchmarking” – Drill-Down Studies
Almost 500 topics now covered
 Drill-Down studies conducted annually in studies; examples include
by each group
Engineering
 Members select topics of mutual •Rolling Stock Reliability
interest •Asset Renewal Decision Making
 Reflect key topical issues, and questions
raised by KPIs Operations
•Investing in Punctuality
 Members provide data and •Driver Productivity
 Questionnaires and expert interviews
Customers
 Imperial College analyses and reports •Customer Satisfaction
findings and recommendations •Passenger Information

 Expert “workshops” to share Financial

experiences directly •Fares, Funding and Financing
•Non-Fare Revenue

19
 Benchmarking Benefits:
Improved decision-making by learning from the experience of others
 Asian Metro: driver productivity study: 10% saved through
shift reorganization

 American Metro: Review of station cleaning processes

following study: 10% productivity gain

 South American Metro: $1mp.a. saved on turnstile

maintenance as a result of a Forum question.

 American Metro: justified move from 2 car-pairs to through

gangways: several million £, improving capacity by 10%

 HK MTR: best practice in policy used to successfully argue

for fare adjustment mechanism

 ISBeRG Railway used fleet age information to help make

the case for funding of replacement cars

20
Example: escalator engineering benchmarking:
application with high impact

 Research for London Underground (LU) 2009

 Modelling revealed higher-than-expected costs for

LU prompting more detailed benchmarking analysis

 Deeper research revealed substantial differences in

engineering and maintenance approach

 LU is now undertaking radical changes to escalator

approach, adopting world best practices

 Expected savings: £0.5 bn (e.g. £65 m saving in

the procurement of 50 escalators, June 2012)

21
Some Benchmarking Examples -
Helping with Key Challenges

Community of Metros

CoMET

22
 Railways/metros have returns to density and not scale
(so we can compare organisations of different sizes)
Network Size and Passenger Journeys p.a. (2011)
600 3 .0
Nova Metros CoMET Metros
500 2 .5

Billion Passenger Journeys

Network Length (km)

Nova & CoMET: Part of

the same community
400 2 .0
and contribute to each
others’ studies
300 1 .5
Compared to metros,
Sydney Trains’ network is
200 very large 1 .0

100 0 .5

0 0 .0

Network Length (km) Passenger Journeys

Based on publically available data 23
ISBeRG Suburban Railways: Sydney Trains’ network also large
relative to these, but more comparable
5816 Network Length, Number of Stations and Passenger Journeys
2536 (2012 or Latest Available)
800
Route km, Number of Stations, Million Passengers

Network Length
700
Stations
600 Sydney Trains’
network still very Passenger Journeys
500 large in ISBeRG

400

300

200

100

0
Tk SP HK Sy Mb Mu CT LO SF Ch LI MN Bc Br Os

Based on publically available data 24

A key structural factor and financial constraint for Australian
Railways: low average passenger density
16
Compared to metros, Sydney is
14 very low density
Passenger Journeys per Network Length km

(more comparable with Suburban

12 Rail – see next page)

10

Based on publically available data 25

High frequencies and density in the urban core countered by
long branches – lower returns to density
6.03 Passenger Density (2012 or Latest Available)
3.0
Compared to major
Million Passenger Journeys per Route km

suburban railways,
2.5 Sydney is also relatively
low density

2.0 Even lower

densities in
Very high N. America
1.5 densities in Asia
and S. America

1.0

0.5

0.0
As Am As Eu Af Am Eu As Eu Eu Sy Am Am As Eu

Based on publically available data 26

Empirical Evidence: Average cost per passenger km declines with
density of capacity
Metro Railway Returns to Density
Average Cost per Passenger km
(index: 100 = Average)

Capacity km per Network km

27
 Supply varies both spatially and temporally
(Here: Buses in major cities – over 24 hours)

% of total fleet
Use of Vehicles in Revenue Service (Average Weekday) - 2008
100%

90%

80%

70%

60%

50%

40%

30%

20% What is the right service level?

Marginal costs vary significantly
10%
throughout the day
0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Hour

28
Key challenge that benchmarking addresses: off-setting wage
growth with revenue growth and increased productivity
Cost of Labour per Hour
(Nominal prices: Index 100 = earliest data year)

On lower density systems,

maintaining efficiency is
an extra challenge

29
 When economies were strong, a combination of poor fares policy and
rising costs set metros in the wrong financial direction

Experience of CoMET and Nova metros, 2004-2009

Real Labour
Fares Productivity
Falling Unit Falling Demand
60% Labour 75% up for
Metros & Energy Metros* 89% of
Costs Up metros
65%

Cost Recovery from

Fare Income Falling
70% Metros

*European and North American Metros 30

 A turnaround in fortune since 2009 for many metros, but only temporary,
with fundamentals of energy, labour, fares unchanged

Experience of CoMET and Nova metros, 2009-2011

Unit Energy
Real Fares
Costs
Falling
Labour Falling 60% Unit Labour
64% Metros
Productivity metros Costs
Improved Up
84% Metros 68% metros

Cost Recovery from

Fare Income Demand up for
Improved 81% of metros
76% Metros

31
On average, operating costs +40% has been spent on reinvestment
But fares revenue barely covers operating costs (except in Asia)
Commercial Revenue per Operating Cost, 2011
2.5
Other Commercial Revenue
Fare Revenue
Key factors: density, fares,
2.0 network, efficiency

Observed Metro
1.5 Reinvestment Rate
(~40%)
Revenue = operating cost
1.0

0.5

0.0

32
Case Studies drill down to understand ‘how’ and ‘why’
operators can become more productive (examples)
 50/50 productivity gain shares

 Contracting out of station staff (Hong Kong)

 New contract deals for new drivers (EU metro)

 Part time staff in peak period (S. America)

 Split Shifts (N. American Metro)

 Multi-skilling: Drivers/ station staff (EU metro)

33
CoMET and Nova benchmarking shares experience in use of
technology to improve productivity / effectiveness
 Unattended Train Operation (UTO) (Paris Line 1 & 14)

 Attended driverless train operation (Taipei)

 Automatic train turnaround (Madrid)

 Ticketing: station staff in more customer facing roles / dispatch

 CBTC: Potential for much higher energy efficiency / regeneration

 Automatic Train Operation (ATO) – higher reliability, fewer spare drivers

 Remote signing on for train drivers (European metro)

34
Factor of 9 variation in staff productivity: What is the right level? What is
achievable? Benchmarking helps transport providers make such choices
Car km per Total Staff and Contractor Hours
2 .0 More
Such high level KPIs can mask
1 .8
efficient?
efficiencies and inefficiencies in sub-
divisions: disaggregation,
1 .6 econometric modelling and drill-
down studies are performed
1 .4
Index (Mean = 1)

Less
1 .2
efficient?
1 .0

0 .8
Structural Factors: e.g. Density
0 .6 Quality (customer service etc)
0 .4

0 .2

0 .0

35
Continued world economic growth affects major capital cities such as
London, New York, driving continued rapid passenger growth
Annual Passenger Journeys % Change
3 .0 2006 - 2011
-4%
Growth continues in
2 .5 London, Paris, New Network expansion
York driving passenger growth
Billion Passenger Journeys

in Asian metros
2 .0
11% 10%
6%
70%
1 .5 123%
14%
119%
132%
1 .0 -2% 94% 44%
50%
48% 259%

0 .5
0%

0 .0
As As As Eu Am Eu Eu Am Eu Am As Am As As Eu As
2005 2006 2007 2008 2009 2010 2011

36
Demand for rail also growing across the world

Annual Passenger Journeys

750
5816 (2012)
700 Figures (%): national GDP growth (same period) 5731 (2011)
5741 (2010)
UK evidence: +1% GDP typically leads to +1% demand
650 5767 (2009)
600 2.0%
Annual Passenger Journeys

550
1.5%
500 Major growth in London 3.2%
450 (network expansion) and Sao
400 Paulo (service improvements
350 and economic growth) 2.9%

300
2.4% 0.7%
250
200
0.4%
150 -0.8% -0.8% 0.8%
2.4% 0.6% 0.6%
100
0.6%
50
0
Eu As Eu As Am Eu As Am Eu Eu Am Am As As
2007 2008 2009 2010 2011 2012
Average Annual GDP Growth Over Equivalent Time Period 37
Conclusions: Benchmarking can be highly effective

 Benchmarking has become a key management

tool for operators: at strategic and day-to-day level
 Very different demands between ‘regulatory’ and
‘best practice’ benchmarking
 The long term, continuous approach with
operator ownership, is key
 Benchmarking is not an exact science requiring
qualitative and contextual understanding
 Drill-down necessary: statistical and deep
investigation of practices, processes
 Our process has continued for 19 years due to the
significant benefits achieved by operators

38
 Any questions?

b.condry@imperial.ac.uk
www.imperial.ac.uk/rtsc

39
Metro Codes – used in graphs to represent metros
Community of Metros

CoMET
BA – Buenos Aires Metrovías Bg – BMTROC, Beijing
Bc – Barcelona TMB Bn – BVG, Berlin
Bs – Brussels STIB Gz – Guangzhou Metro Corporation
Bk – Bangkok BMCL HK – MTRC, Hong Kong
Dh – Delhi Metro Rail Corporation Ln – LUL, London
Is – Istanbul Ulasim MC – STC, Mexico City
Lb – Lisbon Metropolitano de Lisboa Md – Metro de Madrid, Madrid
Mt – Montréal STM Mw – MoM, Moscow
Nc – Newcastle Nexus NY – NYCT, New York
Nj – Nanjing Metro Pm – RATP Metro, Paris
Np – Naples Metronapoli Pr – RATP RER, Paris
RJ – Rio Metro SC – Metro de Santiago
Sg – Singapore SMRT Sh – SSMG, Shanghai
Sy – Sydney Trains SP – MSP, São Paulo
To – Toronto TTC Tp – Taipei TRTC

Anonymised Graphs: As = Asia/Australia; Eu = Europe; Am = Americas 40

ISBeRG Member Railways and Codes

Railway City Code

Ferrocarrils de la Generalitat de Catalunya (FGC) Barcelona Bc
Queensland Rail Brisbane Br
S-Tog, Danish State Railways (DSB) Copenhagen Ch
PRASA – Metrorail, Cape Town Cape Town CT
MTR Hong Kong (East Rail and West Rail only) Hong Kong HK
MTA Long Island Rail Road (LIRR) New York LI
London Overground – London Rail London LO
Metro Trains Melbourne Melbourne Mb
MTA Metro-North Railroad New York MN
S-Bahn Munich, Deutsche Bahn (DB) Regio Munich Mu
Commuter Rail, Norwegian State Railways (NSB) Oslo Os
Bay Area Rapid Transit (BART) San Francisco SF
Sao Paulo Metropolitan Train Company (CPTM) Sao Paulo SP
Sydney Trains Sydney Sy
East Japan Railway Company (JR East) Tokyo Tk

Anonymised Graphs: As = Asia/Australia; Eu = Europe; Am = Americas 41