Vous êtes sur la page 1sur 32

Performance measurement in

R&D: exploring the interplay


between measurement objectives,
dimensions of performance and
contextual factors
Vittorio Chiesa1, Federico Frattini2, Valentina
Lazzarotti3 and Raffaella Manzini4
1

Politecnico di Milano - Department of Management, Economics and Industrial Engineering,


Piazza L. da Vinci 32, 20133 Milano, Italy. vittorio.chiesa@polimi.it
2
Politecnico di Milano - Department of Management, Economics and Industrial Engineering,
Piazza L. da Vinci 32, 20133 Milano, Italy. federico.frattini@polimi.it
3
Universita` Carlo Cattaneo LIUC, Corso Matteotti, 22, 21053 Castellanza, Varese, Italy.
vlazzarotti@liuc.it
4
Universita` Carlo Cattaneo LIUC, Corso Matteotti, 22, 21053 Castellanza, Varese, Italy.
rmanzini@liuc.it
Measuring research and development (R&D) performance has become a fundamental concern
for R&D managers and executives in the last decades. As a result, the issue has been
extensively debated in innovation and R&D management literature. The paper contributes to
this growing body of knowledge, adopting a systemic and contextual perspective to look into
the problem of measuring R&D performance. In particular, it explores the interplay between
measurement objectives, performance dimensions and contextual factors in the design of a
performance measurement system (PMS) for R&D activities. The paper relies on a multiple
case study analysis that involved 15 Italian technology-intensive rms. The results indicate
that rms measure R&D performance with different purposes, i.e. motivate researchers and
engineers, monitor the progress of activities, evaluate the protability of R&D projects, favour
coordination and communication and stimulate organisational learning. These objectives are
pursued in clusters, and the importance rms attach to each cluster is inuenced by the context
(type of R&D, industry belonging, size) in which measurement takes place. Furthermore, a
rms choice to measure R&D performance along a particular perspective (i.e. nancial,
customer, business processes or innovation and learning) is inuenced by the classes of
objectives (diagnostic, motivational or interactive) that are given higher priority. The
implications of these results for R&D managers and scholars are discussed in the paper.

1. Introduction

M
488

easuring performance and contribution to


value research and development (R&D)

has become a fundamental concern for R&D


managers and executives in the last decades (Kerssen-van Drongelen and Bilderbeek,
1999). Since the 1990s, several phenomena have

R&D Management 39, 5, 2009. r 2009 The Authors. Journal compilation r 2009 Blackwell Publishing Ltd. 2009,
9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.

Performance measurement in R&D


encouraged the development and adoption of
specic approaches for assessing the performance
of R&D: increasing turbulent dynamics in competitive arenas, shortened life cycles, globalisation, reduced time to market, increased R&D
costs and risks (Nevens et al., 1990; Bayus,
1994; Wind and Mahajan, 1997; Wolf, 2006).
Although it is still acknowledged that the
measurement of R&D performance is a challenging task that might also hinder the creative and
innovative capacity of the rm (Brown and Svenson, 1998), today the issue is being extensively
debated in the innovation and R&D management
literature and it raises the interest of practitioners
as well (Pappas and Remer, 1985; Brown and
Svenson, 1988; Sivathanu and Srinivasa, 1996;
Werner and Souder, 1997; Hauser, 1998; Driva
and Pawar, 1999; Driva et al., 2000; Poh et al.,
2001; Loch and Tapper, 2002; Godener and
Soderquist, 2004; Ojanen and Vuola, 2006).
This paper aims at contributing to this growing
body of knowledge, adopting a systemic and
contextual perspective to look into the problem
of measuring R&D performance. In particular, it
explores the interplay between measurement objectives, performance dimensions and contextual
factors in the design of a performance measurement system (PMS) for R&D activities. More
specically, it aims to understand: (i) which objectives companies pursue when they measure R&D
activities performance and whether they can be
categorized in some ways; (ii) which approaches to
R&D performance measurement are used to pursue different classes of objectives; and (iii) how the
importance attached to different classes of objectives, and the approaches used to pursue them, are
affected by the context in which measurement
takes place. Although the literature on management accounting and control has acknowledged
the importance of these topics (e.g., Simons, 2000;
Azzone, 2006), they have not been properly investigated in R&D settings so far. What is more, a
better understanding of these issues would be
highly benecial for R&D managers. The rich
empirical data presented and discussed in this
paper will provide R&D managers with a number
of insights that represent a valuable starting point
to design a PMS for R&D that is adequate to the
objectives they have in mind, and is appropriate as
well to the context in which their rm operates.
Moreover, some practical suggestions about how
to improve managers satisfaction with the PMS
are discussed in the paper.
In order to pursue its objectives, the paper rst
develops a reference framework, which is used as a
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

guide for the subsequent multiple case study


analysis. The empirical investigation involved a
number of Italian rms operating in technologyintensive industries, for which technological innovation and, therefore, the results of their R&D
efforts, are a major source of competitive advantage. Because of the signicant contribution of
R&D to the companys overall success, these rms
will be far more likely to systematically measure
their innovative activities performance. Therefore, they represent an ideal empirical setting for
investigating the issues we are interested in.
The remainder of the paper is organised as
follows: Section 2 reviews the relevant literature
on R&D performance measurement, whereas
Section 3 describes the reference framework underlying the research. Section 4 illustrates the
methodology used for the empirical analysis,
and Section 5 discusses the result of the multiple
case study. Finally, Section 6 concludes and outlines some avenues for future research.

2. Literature review
Existing research into the measurement of R&D
performance in industrial rms has investigated
the topic from four different perspectives, as
shown in Figure 1.
At a rst level, research has basically focused on
the choice of the indicators or metrics that are best
suited to the characteristics of R&D. Brown and
Svenson (1998) nd that an effective PMS for
R&D is built around a limited number of indicators that measure results rather than behaviour,
and privileges objective and external metrics to
subjective and internal ones. Nixon (1998) advances that performance indicators for R&D
should have a strategic orientation and reect the
rms critical success factors, they should be simple,
able to encourage change and to balance nancial
and non-nancial perspectives. Werner and Souder
(1997) state that the most effective measurement
approaches for R&D are those that balance both
quantitative and qualitative metrics, as also underlined by Pawar and Driva (1999) and Bremser and
Barsky (2004). Hauser (1998) shows that the choice
of the most appropriate metrics should be based on
the type of R&D, whether it is applied research,
core technological development or basic research.
Other scholars in this stream of research investigate
the opportunity to use nancial indicators in R&D
departments (Rockness and Shields, 1988) and to
build a synthetic indicator of R&D productivity or
efciency (Tipping et al., 1995).
R&D Management 39, 5, 2009

489

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini

Performance Measurement
Systems for R&D within the firms internal
and external context
Performance Measurement
Systems for R&D

Dimensions for R&D performance


measurement
Indicators and
metrics for R&D
performance
measurement

Figure 1. Taxonomy of research streams on performance measurement in R&D.

At a second level, research has looked into the


choice of the performance dimensions, or perspectives, along which the measurement of R&D
should be undertaken. Pawar and Driva (1999)
advance that the measurement of R&D performance needs to be articulated into time, costs,
quality and exibility dimensions. Kim and Oh
(2002) identify the following types of R&D performance variables: market-oriented, R&D project-specic and R&D researcher-specic. Davila
(2000) analyses the use of cost, time and customer
(or market) information in the measurement of
new product development performance. Many
scholars in this stream of research have attempted
to apply the Balanced Scorecard (BSC) approach
(Kaplan and Norton, 1992) to R&D. Kerssen-van
Drongelen and Cook (1997), e.g., show how to
develop a measurement approach for R&D performance that, integrating nancial, client, internal business, innovation and learning perspectives,
allows to implement the rms R&D and competitive strategy. Bremser and Barsky (2004) illustrate how the BSC approach should be integrated
with the stage-gate system (Cooper, 1993) for the
organisation of innovation development activities.
At a third level, research has adopted a systemic perspective in the study of R&D performance measurement, assuming the whole PMS
for R&D as the unit of analysis. Kerssen-van
Drongelen et al. (2000), e.g., conceive the PMS
for R&D as comprising the following elements:
metrics organised into a consistent structure,
standards to measure performance against, frequency and timing of measurement and format
for information reporting. Similarly, Ojanen and
Vuola (2006) suggest that an effective PMS for
490

R&D Management 39, 5, 2009

R&D should be an internally consistent set of


measurement perspectives, objectives, control objects and measurement process. In other words,
adopting a systemic perspective means looking at
R&D performance measurement in terms of a
system, which should be made of a set on
integrated and internally consistent elements
(Chiesa and Frattini, 2007), i.e. PMS objectives,
performance dimensions, metrics or indicators,
control objects and measurement process.
Finally, a more strategy-oriented stream of
research has adopted a contextual perspective,
emphasising that a PMS for R&D should be
studied within the context in which it is used,
which is both internal and external to the rm.
This body of literature is consistent with the
largest part of the extant management accounting
and control research (e.g., Gordon and Miller,
1976; Gordon and Narayanan, 1984) and basically reminds that the PMS is used in a specic
R&D setting, being basic and applied research or
NPD (Pappas and Remer, 1985; Chiesa and
Frattini, 2007), with a given amount and quality
of available resources (Godener and Soderquist,
2004), within the scope of a rms specic business strategy, mission, values and management
style (Nixon, 1998; Kim and Oh, 2002; Loch and
Tapper, 2002) and, nally, in a broader competitive, economic, social, cultural and political context (Loch et al., 1996; Pillai et al., 2001).

3. Reference framework
Investigating the interplay between measurement
objectives, performance dimensions and contextual
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

Performance measurement in R&D


factors in the design of a PMS for R&D, the paper
contributes to the fourth level of analysis described
in the last section. Although the literature adopting
a contextual perspective has studied the impact of
several endogenous and exogenous variables on
the design of specic components of the R&D
PMS, an integrated view on how the objectives for
which the PMS is introduced affect the design of its
constitutive elements, and how they are in turn
inuenced by the measurement context, has not
been advanced yet.
A company that seeks to measure the performance of its R&D can do this with many different
purposes. According to Kerssen-van Drongelen
and Cook (1997), there are two main classes of
underlying reasons for R&D performance measurement, i.e. to motivate scientists and researchers and to diagnose activities and processes. Loch
and Tapper (2002) identify the following foremost
objectives for which rms control their R&D
performance: align behaviour and set up priorities, evaluate and reward researchers, establish an
operative control and stimulate learning and improvement. Kerssen-van Drongelen et al. (2000)
add that performance measurement, especially in
complex new product development projects, can
serve the purpose of favouring communication
and coordination among top managers, middle
managers and researchers.
Management accounting and control research
has repeatedly shown that the set of objectives for
which a rm measures its business performance
represents a driving force that heavily affects the
design of the other PMS constitutive elements,
e.g., dimensions of performance, indicators, control objects and measurement process (Neely,
1999; Bititici et al., 2000; Tuomela, 2005). Most
of all, Robert Simons (1994, 1995) explains that
companies pursue different objectives through
different control systems (called levers of control). Specically, belief systems are used to
communicate and reinforce the rms value and
the paths to be followed to identify and exploit
value creation opportunities; boundaries systems
are needed to encourage individual creativity,
within well-dened frontiers; diagnostic control
systems serve the purpose of coordinating and
motivating the implementation of the rms strategy; and interactive control systems are used to
stimulate organisational learning, communication
and the emergence of ideas related to new business
opportunities. Each of these management control
systems has specic characteristics in terms of
performance dimensions and perspectives of analysis, measurement frequencies and standards.
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

In our research, we wanted to understand


whether these general concepts (in particular,
the inuence of measurement objectives over the
characteristics of the PMS) hold true in R&D
settings as well. Moreover, we were interested in
the role of the internal and external contextual
factors. In other words, considering that the PMS
for R&D is embedded in the rms internal and
external context, we wanted to understand
whether and how these factors are able to affect
both the importance the rm attaches to different
classes of objectives and the design of its constitutive elements. This assumption is consistent
with the largest part of extant management control research (Chenhall and Morris, 1986).
Considering that the number of relevant contextual factors is potentially countless and that a
PMS for R&D is comprised of a high number of
interrelated parts, we decided to confer a more
specic and denite scope to our research. As far
as the characteristics of the PMS are concerned,
we focused on the following elements: (i) the
dimensions along which R&D performance is
assessed and (ii) the indicators (or metrics) that
are used to measure performance along the abovementioned dimensions (see, e.g., Chiesa and Frattini, 2007). As far as the context in which measurement takes place is concerned, we focused on
the following factors that have been identied by
extant research as inuential over the design of the
PMS for R&D (Pappas and Remer, 1985; Kerssen-van Drongelen and Bilderbeek, 1999; Davila,
2000; Bremser and Barsky, 2004): (i) the type of
R&D activity that is measured (being it basic and
applied research or new product development); (ii)
the size of the rm and its R&D unit; and (iii) the
industrial sector the rm belongs to. The reference
framework that served as a guide for our empirical
analysis is summarised in Figure 2. It hypothesises

PMS
CHARACTERISTICS:

PMS
OBJECTIVES

-Performance dimensions
-Indicators

MEASUREMENT
CONTEXT:
-Type of R&D activity
-Size of the firm and the R&D unit
-Firms sector of activity

Figure 2. The reference framework.

R&D Management 39, 5, 2009

491

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini


that the objectives for which a rm adopts the
PMS inuence the design of its constitutive elements. Furthermore, it advances that the context
in which measurement takes place has a potential
effect over the choice of the PMS objectives as well
as the design of its constitutive elements. The
empirical analysis will help shed light on the
strength and nature of these hypothesised relationships.

4. Research methodology
We decided to use case study research as an
overall methodological approach for our empirical investigation. As suggested by a number of
scholars, this is in fact a very powerful method for
building a rich understanding of complex phenomena (Eisenhardt and Graebner, 2007) that
requires the capability to answer to how and
why questions (Yin, 2003). In particular, we
used a multiple case study approach, which was
chosen because it allows both an in-depth examination of each case and the identication of
contingency variables that distinguish each case
from the other. Furthermore, multiple case studies are appropriate when attempting to externally validate the ndings from a single case
study, through cross-case comparisons (Eisenhardt, 1989). Therefore, they typically yield
more robust, generalisable and testable interpretations of a phenomenon than single case study
research (Eisenhardt and Graebner, 2007).
The study involved 15 Italian rms from different industries (e.g., aerospace, pharma-biotech,
pharmaceuticals, machining centres, chemicals)
that were studied during the last 2 years (see
Table 1, where real names have been blinded for
condentiality reasons). As a unit of analysis for
our case study, we considered the PMS used in the
rms R&D unit. In a number of cases, the
studied company had more than a single organisational unit devoted to R&D activities; unfortunately, in these instances, we had the opportunity
to study only one of them. This impeded us from
undertaking an embedded analysis that could
have provided richer information. We adopt the
largely applied and broad distinction between
Research and Development, including in the former basic and applied research activities and in
the latter the development of both incremental
and radical new products. We were able to
classify the activities undertaken in the R&D
units that we considered as either Research and
Development, this indicating a widespread orga492

R&D Management 39, 5, 2009

nisational separation between Research and


Development activities (Chiesa, 1996).
We gathered information basically through
direct interviews; in particular, we followed these
steps:
 At the outset of each case, a relationship was
established with a senior manager from the
selected rm. This person was informed about
the research project through a written summary and a telephone meeting. During this
meeting, we asked the respondent to introduce
ourselves to the head of the rms R&D
function or to another R&D manager who
was responsible for the performance of the
R&D unit and the operation of the PMS;
 Then we personally interviewed the selected
R&D managers; we undertook two semistructured interviews for each respondent
(each interview lasted on average one and a
half hour) in order to gather the information
required to pursue the papers research objectives. Direct interviews followed a semi-structured replicable guide (see Appendix A),
which comprised a set of open questions for
each of the relevant constructs in our reference
framework (e.g., objectives of the PMS and
dimensions of performance);
 Secondary information was collected in the
form of company reports and project documentation. In particular, we gathered and
analysed all the reporting documents that
were generated in support to the functioning
of the PMSs. These informed the researchers
with background information about the selected rms, the type of R&D activity they
undertake and the approaches they use for
measuring R&D performance. Above all,
these secondary information sources were integrated, in a triangulation process, with data
drawn from the direct interviews, in order to
avoid post hoc rationalisation and to ensure
construct validity (Yin, 2003);
 All interviews were tape-recorded and transcribed; generally, at this stage a telephone
follow-up with the respondents was conducted
in order to gather some important missing data.
Data and information gathered through the
case studies were manipulated before being analysed. In particular, we applied the following
techniques (Miles and Huberman, 1984): (i) data
categorisation, which requires the decomposition
and aggregation of data in order to highlight
some characteristics (e.g., objectives pursued
with the PMS or context in which measurement
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

Performance measurement in R&D


Table 1. The studied rms
Firm

Sector of activity

No. of
employees

Role of people interviewed

Company A
Company B

Semiconductors
Electronics for industrial
applications
Machining centres
Aerospace
Pharmaceuticals
Pharma-Biotech
Chemicals

50,000
500

R&D projects manager


Director of R&D and quality manager

160
1,800
500
60
19,300
9,000
700
70
60,000

Director of the technical department


Planning and cost control manager
Director of the development department
Chief operating ofcer
Director of innovation & technology in plastic
additives
Program manager
Director of oncology division
General director of research laboratories
R&D platform manager

2,200
1,000
3,000
2,600

Technology and business development manager


Vice President for research and development
Vice President for corporate drug development
R&D director

Company
Company
Company
Company
Company

C
D
E
F
G

Company
Company
Company
Company

H
I
L
M

Company
Company
Company
Company

N
O
P
Q

Aerospace
Pharma-Biotech
Pharmaceuticals
Household electrical appliances
and home automation
Power generation technologies
Medical imaging diagnostic
Pharmaceuticals
Energy conversion

takes place) and to facilitate comparisons; (ii)


data contextualisation, which implies the analysis
of contextual factors, not included in the conceptual model, that may reveal unforeseen relationships between events and circumstances. Then, a
preliminary within-case analysis was performed;
the purpose was to consider each case study as a
separate one and to systematically document the
variables of interest dened in the reference framework. Then, explanation-building procedures
were applied so that the relationships between the
PMS objectives, characteristics of the PMS and
context in which measurement takes place were
identied. Finally, a cross-case analysis was undertaken for comparing the patterns that emerged
in each case study in order to arrive at a general
explanation of the observed phenomenon. These
structured procedures for data collection and
analysis, as well as the use of the semi-structured
interview guide, helped enhance the reliability of
the research (Yin, 2003).
The following section reports and discusses the
empirical evidence we gathered for the 15 cases
included in our sample. It is used to illustrate the
interplay between measurement objectives, performance dimensions and contextual factors in
the design of a PMS for R&D.

5. Results and discussion


The empirical evidence that was gathered for the
cases in our sample is synthesised and mapped
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

along the dimensions of the reference framework


in Appendix B, where some information about
the difculties encountered in the adoption and
use of the PMS, as well as the satisfaction of
the rms executives with this tool, is reported.
Table 2 provides a synoptic view of these data to
allow a more straightforward comparison and
analysis.
An in-depth discussion of this empirical evidence is reported in the following paragraphs.

5.1. PMS objectives and measurement


context
The empirical analysis indicates that rms decide
to measure the performance of their R&D activities with multiple purposes. In particular, based
on our study of the 15 rms, it is possible to
identify the following list of major objectives a
company might aim at when it comes to measuring R&D performance:
(1) Motivate researchers and engineers and improve their performance in R&D activities;
(2) Monitor the progress of R&D activities with
respect to resource consumption targets, temporal milestones and technical requirements;
(3) Evaluate the protability of R&D activities
and their contribution to the rms economic
value;
(4) Support the selection of the projects to be
initiated, continued or discontinued;
R&D Management 39, 5, 2009

493

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini


Table 2. Synoptic representation of the case study evidence
Firm

Type of
R&D
activity

Size of the Sector


R&D unit
(number of
employees)

Company A Basic and


applied
research

700

Company B New product


development

100

Company C New product


development
Company D New product
development

15
300

Company E New product


development

50

Company F Basic and


applied
research
Company G Basic and
applied
research

60
300

Company H New product


development

200

Company I

280

Basic and
applied
research

Company L Basic and


applied
research
Company M New product
development

494

70
200

R&D Management 39, 5, 2009

PMS objectives

High-tech

(1) Motivate scientists


and engineers
(2) Favour coordination
and communication
(3) Stimulate
organisational learning
High-tech
(1) Monitor the progress
of R&D activities
(2) Evaluate the
protability of R&D
activities
High-tech
(1) Motivate scientists
and engineers
High-tech
(1) Monitor the progress
of R&D activities
(2) Evaluate the
protability of R&D
activities
(3) Favour coordination
and communication
Science-based (1) Monitor the progress
of R&D activities
(2) Motivate scientists
and engineers
Science-based (1) Motivate scientists
and engineers
High-tech

(1) Motivate scientists


and engineers
(2) Favour coordination
and communication
3) Stimulate
organisational learning
High-tech
(1) Monitor the progress
of R&D activities
(2) Evaluate the
protability of R&D
activities
(3) Favour coordination
and communication
4) Reduce uncertainty
Science-based (1) Motivate scientists
and engineers
(2) Select R&D projects
(3) Monitor the progress
of R&D activities
Science-based (1) Motivate scientists
and engineers
High-tech

(1) Monitor the progress


of R&D activities
(2) Evaluate the
protability of R&D
activities
(3) Favour coordination
and communication
(4) Stimulate
organisational learning
(5) Reduce uncertainty

Performance perspectives
associated with each
PMS objective
(1) Innovation and learning
(23) Business process

(1a) Business process


(1b) Customer
(2) Financial
(1a) Innovation and learning
(1b) Business process
(1-23) Business process

(1) Business process


(2) Innovation and learning
(1a) Innovation and learning
(1b) Business process
(1a) Innovation and learning
(1b) Business process
(23) Business process

(1a) Business process


(1b) Customer
(2a) Business process
(2 b) Financial
(2c) Customer
(34) Business process
(1a) Innovation and learning
(1b) Business process
(23) Business process
(1a) Innovation and learning
(1b) Business process
(12a) Financial
(12b) Customer
(12c) Business process
(345) Business process

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

Performance measurement in R&D


Table 2. (Contd.)
Firm

Type of
R&D
activity

Size of the Sector


R&D unit
(number of
employees)

Company N New product


development

100

Company O Basic and


applied
research

100

Company P

300

Basic and
applied
research

Company Q New product


development

150

High-tech

PMS objectives

Performance perspectives
associated with each
PMS objective

(1) Monitor the progress


of R&D activities
(2) Evaluate the
protability of R&D
activities
(3) Select R&D projects
High-tech
(1) Monitor the progress
of R&D activities
(2) Evaluate the
protability of R&D
activities
(3) Select R&D projects
Science-based (1) Favour coordination
and communication
(2) Stimulate
organisational learning
(3) Motivate scientists
and engineers
High-tech
(1) Monitor the progress
of R&D activities
(2) Select R&D projects
(3) Evaluate the
protability of R&D
activities

(123a) Customer
(123b) Financial
(123c) Business process

(123a) Financial
(123b) Customer

(1) Business process


(23a) Innovation and
learning
(23b) Financial
(1) Business process
(23a) Business process
(23b) Financial

PMS objectives and performance perspectives are ordered on the basis of the importance attached by each rm.

(5) Favour coordination and communication


among the different people and organisational units taking part in R&D activities;
(6) Reduce the level of uncertainty that surrounds R&D activities; and
(7) Stimulate and support individual and organisational learning.
Moreover, our analysis reveals that these objectives are pursued in clusters by rms. A rst group
of companies can be identied that use performance measurement in R&D with the main purpose of improving the degree of control they
exercise on R&D activities, and to have a support
for taking more effective management decisions.
Examples of these rms are Companies B, D, N,
O and Q, which share the following as main
objectives for R&D performance measurement:
monitor the progress of activities, select R&D
projects and evaluate the protability of R&D
activities. These are labelled in the paper as
diagnostic objectives, as they correspond to the
reasons underlying the use of diagnostic control
systems in Robert Simonss theory (Simons,
1995). A second group of rms can be identied
that use performance measurement mainly as a
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

tool for motivating scientists and engineers, directing their efforts toward the long-term innovation
targets of the rm and overcoming the lack of
commitment that R&Ds largely intangible results
often determine. This is the case of Companies A,
C, F, G, I and L. They believe that motivating
researchers and technicians is a pre-requisite for
improving their performance in R&D activities,
and often establish to reward them on the basis of
the performance that are estimated by the PMS
(as it happens, e.g., in Companies F and I, where
performance measurement provides the data necessary to operate a management by objectives
MBO rewarding system). We consider these rms
as mainly interested in pursuing motivational
objectives through R&D performance measurement, which remind us of the reasons for which
managers use belief and boundaries control
systems (Simons, 1995). Finally, our analysis
unravels the existence of a number of companies
for which performance measurement also serves a
second purpose, besides the diagnostic or the
motivational objectives mentioned above. In
particular, they believe it is a very useful means
to improve and streamline the execution of R&D
activities and processes, overcoming some of the
R&D Management 39, 5, 2009

495

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini


organisational barriers that characterise them. In
particular, performance measurement is also used
to favour coordination and communication, stimulate quick and effective organisational learning
and reduce the level of uncertainty surrounding
critical decisions in R&D. These objectives are
acknowledged as relevant by Companies A, D, G,
H, M and P, and they are called interactive in
this paper, as they closely mirror the use of
interactive control systems described by Simons
(1995). Figure 3 shows the position of the 15 cases
discussed in this paper with respect to the three
clusters of objectives. It is interesting to note that
these results are consistent with the sparse empirical evidence available in the literature (Kerssenvan Drongelen and Cook, 1997; Kerssen-van
Drongelen et al., 2000; Loch and Tapper, 2002).
Interestingly, our analysis suggests that the
importance a rm attaches to each cluster of
objectives is inuenced by some characteristics
of the context in which measurement takes place.
In particular, diagnostic objectives are predominant in those rms that decide to measure the
performance of new product development activities. The cases of Companies B, D, H, M, N and
Q suggest that the need for control is stronger in
NPD than in basic and applied research. The
main reason is that the output of NPD activities is
sold directly on the market; therefore, the respect
of deadlines, quality requirements and target
costs in these activities has a more direct impact
on the rms market competitiveness than in the
case of basic and applied research, whose clients
are basically internal. Moreover, the amount of
nancial and human resources involved in NPD is
DIAGNOSTIC
1) Monitor the progress of
activities
2) Evaluate the profitability
of R&D activities
3) Select R&D projects
Company B
Company N
Company O

Company Q

very large (especially if compared with basic and


applied research), this making a proper evaluation of R&D protability, and an accurate prioritisation of projects, critical challenges for R&D
managers. Our analysis reveals that the need for
diagnostic control in R&D is also particularly
strong in large R&D units. The larger a rms
R&D unit, the higher the number of different
(and often interrelated) projects that are contemporarily undertaken, the higher the number of
researchers and engineers (often belonging to
different departments or functional areas) taking
part in these projects and the larger the amount of
resources devoted to R&D activities. These conditions make the need for a tight diagnostic
control particularly evident, as it is clear for
instance in the case of Companies D and H.
Nevertheless, exerting this type of control over
R&D activities requires that the latter are, at least
to some extent, predictable, that standards to
measure performance against can be easily identied and that the progression of project activities
along a sequence of stages can be a priori identied. It is clear that new product development is
more foreseeable than basic and applied research,
but predictability also depends on the characteristics of the industry in which a rm operates.
Kodama (1995) suggests that it is possible to
classify industrial sectors on the basis of the
probability with which an R&D project is frozen,
which is a measure of the predictability of R&D
activities.1 High-tech industries are characterised
by a freezing rate that decreases throughout the
R&D process, whereas science-based industries
are those in which the freezing rate always

MOTIVATIONAL
1) Motivate scientists and
engineers

Company F
Company E

Company I

Company C Company L

Company D

Company A

Company H

Company G

Company M

Company P

INTERACTIVE
1) Favour coordination and
communication
2) Stimulate organisational
learning
3) Reduce uncertainty

Figure 3. The studied companies and the emerging clusters of objectives. Within each cluster, objectives are ordered on the basis of
their relative importance.

496

R&D Management 39, 5, 2009

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

Performance measurement in R&D


remains high. Our analysis suggests that rms use
performance measurement in NPD with the main
purpose of controlling activities in those industries (high-tech, in the denition of Kodama)
where the failure rate of projects and activities,
and hence their unpredictability, is smaller. This is
evident if we consider, e.g., that Companies B, D
and H operate in the electronics and aerospace
industries, Company M in the household appliance and home automation sectors and Companies N and Q in the power generation and energy
conversion industries. More interestingly, Company O mainly pursues diagnostic objectives
although engaged in basic and applied research,
and this appears to be linked to the lower degree
of uncertainty characterising the industry in
which it operates.
On the other hand, rms tend to pursue motivational objectives through R&D performance
measurement in basic and applied research. This
is clear if we consider the cases of Companies A,
F, G, I and L. The managers we interviewed
acknowledged that the motivational aspect of
performance measurement is stressed here because the activity is very much uncertain, mostly
unforeseeable and with distant time outcomes,
which makes it difcult to align researchers
efforts with the rms strategic goals. In these
instances, improving the performance of researchers is a matter of stimulating their creativity but,
at the same time, directing their efforts to the
aspects that are relevant for the rm as a whole.
The need to motivate researchers through performance measurement seems to be inuenced not
only by the type of R&D activity. Company E
exemplies for instance a situation where the
motivational purpose of performance measurement is felt as particularly critical in new product
development. This is due to the fact that Company E operates in a science-based industry
(Kodama, 1995), where failure rates and degrees
of uncertainties are signicantly higher than zero
also in the downward phases of the R&D process,
i.e. development and testing, that turn out to be
highly unpredictable. Finally, our empirical analysis suggests that motivation of researchers might
become the main objective for R&D performance
measurement in small organisations. In these
cases, in fact, control is very often exerted on a
personal (or clan) basis (Ouchi, 1979), and hence
its very essence lies in the capability to align
employees efforts to the rms priorities. This is
very clear for instance in the case of Company C.
Whereas diagnostic and motivational objectives are mutually exclusive, interactive ones are
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

pursued by the rms in our sample along with


another class of objectives. Moreover, it emerges
that companies that conceive performance measurement in R&D as a means to streamline
communication and coordination, to stimulate
organisational learning and to overcome decision-making inertia (i.e. to reduce uncertainty in
R&D) share as a common feature the fact of
having a very large R&D unit. Our case studies
clearly indicate that, in order to pursue both
diagnostic and motivational objectives, a critical aspect is to prevent researchers and engineers
from perceiving their creativity and autonomy as
being too much constrained by the PMS. Therefore, it is important to adopt an enabling approach in the management of their performance
(Wouters and Wilderom, 2008), where researchers
and engineers are continuously involved in the
measurement process, a double-loop ow of information keeps them informed about the progress of R&D activities and coordination and
collective learning allow researchers and engineers to take more autonomous and empowered
decisions. The rms in our sample acknowledge
that this fundamental objective can be actually
pursued through the PMS, and that it becomes
more critical in large R&D units, characterised by
a higher degree of organisational complexity,
hierarchy, fragmentation and vertical specialisation, which raises the need for a better coordination and more effective organisational learning
processes. Figure 4 shows the conditions under
which each cluster of objectives for R&D performance measurement grows in importance.

5.2. PMS characteristics


Our analysis indicates rst of all that the dimensions along which R&D performance is evaluated
can be brought back to the BSC perspectives, as
suggested by a number of scholars (Kerssen-van
Drongelen and Cook, 1997; Bremser and Barsky,
2004). In fact, the companies that we studied
measure R&D performance taking into account:
 The economic and nancial aspects associated
with R&D (nancial perspective);
 The extent to which R&D identies and
satises the needs of its internal and external
customers (customer perspective);
 The efciency with which specic tasks and
processes are carried out (business process
perspective);
 The extent to which R&D contributes to
generate new knowledge and innovation
R&D Management 39, 5, 2009

497

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini

- New Product Development


- Large firms and R&D units
- High Tech industries

- Basic and Applied Research


- Small firms and R&D units
- Science Based Industries

DIAGNOSTIC
OBJECTIVES

MOTIVATIONAL
OBJECTIVES

- Large firms and R&D units

INTERACTIVE
OBJECTIVES

Figure 4. Clusters of objectives and measurement context.

opportunities (innovation and learning perspective).


There are rms that combine several different
perspectives in the measurement of R&D performance, although they do not knowingly report
using a BSC system. This is the case for instance
of Companies A and C, which take into account
both the innovation and learning and the business
process perspectives, or Company B, which combines the nancial, business process and customer
perspectives. Other companies measure performance along a single most important dimension.
For instance, Company D uses a PMS very much
focused on internal processes efciency.
More interestingly, our analysis suggests that
companies tend to use different performance
dimensions to pursue different classes of objectives. It noticeably emerges that nancial and
customer perspectives are privileged by rms
pursuing diagnostic objectives, as it is clear
from the cases of Companies B, H, N and O.
This is obvious if we consider that selection and
prioritisation of R&D projects is carried out on
the basis of a projects contribution to the rms
competitive advantage, which depends on its
economic/nancial outcome and appealing in
the eyes of the customers. On the other hand,
the innovation and learning perspective is widespread among rms pursuing motivational objectives, like Companies A, C, F, I and L,
suggesting that researchers and engineers need
498

R&D Management 39, 5, 2009

to be motivated mainly on the basis of their


capacity to contribute to the rms innovation
potential. The business process perspective is
instead used by both rms pursuing diagnostic
and motivational objectives. In the latter case,
using this perspective besides the innovation and
learning one serves the purpose of introducing a
dimension of performance that can be more
directly controlled by the researcher, which is
critical for motivational purposes as also indicated by theories of action, design and expectation (e.g., Pritchard, 1990; Moizer, 1991). For
instance, Company A evaluates with this aim the
efciency (subjectively assessed by peers) with
which researchers perform specic tasks or acquire specic competencies. Firms pursuing diagnostic objectives use instead the business
process perspective with the main purpose of
introducing an operative form of control that
nancial and customer-oriented measures do not
allow to perform. As far as interactive objectives
are concerned, it emerges that they are associated
mainly with the business process perspective.
What is interesting to note in this case is that
the efciency in undertaking business processes is
evaluated in an interactive manner, through a
continuous involvement of engineers and researchers in the measurement process, as it
emerges from the cases of Companies D, H and
M. This is in fact a pre-requisite for introducing
the enabling approach in performance management that stimulates the individual autonomy and
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

Performance measurement in R&D


creativity mentioned above. This evidence is consistent with the results reported by Simons (2000)
on the use and role of interactive management
control systems.
Figure 5 suggests therefore that three archetypes for R&D performance measurement
emerge, each characterised by an internally consistent set of objectives, performance dimensions
and characteristics of the measurement context.
As is clear from Figure 5, our analysis suggests
that the choice of the dimensions along which
R&D performance measurement is carried out is
inuenced mainly by the objectives that are pur-

sued, rather than the context in which measurement takes place. In other words, contextual
factors do not appear to affect the design of the
PMS constitutive elements, as instead hypothesised in Section 3. This is clear from the analysis
of Table 2, which shows that companies operating
in different contexts use the same performance
dimensions to pursue identical objectives through
the PMS.
Another interesting aspect unearthed by our
analysis is that rms with large R&D units seem
to be more inclined to use the PMS for diagnostic purposes rather than for motivational ones,

- New Product Development


- Large firms and R&D units
- High Tech industries

- Basic and Applied Research


- Small firms and R&D units
- Science Based Industries

MOTIVATIONAL
OBJECTIVES

DIAGNOSTIC
OBJECTIVES

- Financial perspective
- Customer perspective
- Business process perspective

-Innovation and learning perspective


- Business process perspective

- Large firms and R&D units

INTERACTIVE
OBJECTIVES

- Business process perspective

Figure 5. Emerging archetypes for performance measurement in R&D.

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

R&D Management 39, 5, 2009

499

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini


in comparison with rms with small R&D units.
This is true unless large rms are engaged in basic
and applied research or operate in very uncertain
(science-based) industries, where the motivational use of the PMS also grows in importance
in large R&D units (see, e.g., the cases of Companies A, I and P).
Furthermore, our analysis shows that each
performance dimension requires specic indicators to be properly estimated. Table 3 provides an
overview of the indicators, associated with the
different performance dimensions, that were identied in our empirical analysis. It should be noted
that the choice of the indicators is not affected
by the specic class of objectives pursued by the
rm. The same indicators are used by the rms
in our sample, e.g., to measure the business
processes perspective for diagnostic or motivational purposes.
As is clear, rms try to measure each performance dimension combining input, process and
output indicators (Brown and Svenson, 1988;
Hauser, 1998), without any discernible correlation between the type of indicator and performance dimension. It is also interesting to
underline the predominance of quantitative objective indicators (Werner and Souder, 1997). The
managers we interviewed explained their choice to
privilege objective indicators (also at the costs of
leaving out some important intangible facets of
R&D performance) with the need to ensure the
measurability of these metrics, which is fundamental for both diagnostic and motivational
purposes, as the theory of task motivation and
incentives (Locke, 1968) indicates. The scarcity of
subjective metrics (both quantitative and qualitative) used by the rms in our sample is, however,
partially in contrast with the evidence gathered in
previous research (e.g., Chiesa et al., 2008), and
this dissimilarity deserves special attention in
future research.

5.3. Critical aspects associated with the


use of the PMS
Analysing the problems associated with the introduction and use of the PMS in the rms in our
sample, it clearly emerges that pursuing motivational objectives, especially in basic and applied
research units, is far more challenging than using
a PMS with diagnostic purposes in an NPD
organisation. In the former case, the largest part
of the managers we interviewed have struggled to
make scientists and researchers positively accept
500

R&D Management 39, 5, 2009

the PMS, with sporadic cases in which one or two


researchers left the organisation after the introduction of the PMS. It is interesting to note that,
even in these cases, managers have not abandoned
the idea of bringing in the PMS (apart from
Company I). Rather, they have adopted a more
incremental approach to establish the PMS in the
organisation, have often re-designed its characteristics (e.g., measurement frequency, denition
of targets or performance metrics) to take into
account the complaints or suggestions for improvement coming from researchers and have
more deeply involved them in the measurement
process. Although tangible results are difcult to
observe, and the costs to run the system are
particularly high, these managers are on average
satised with the PMS. On the other hand, no
particular problems have come across in those
rms that have introduced and used a PMS
with diagnostic purposes in their NPD units,
where tangible results (in terms, e.g., of improved
timeliness of development projects and timeto-market) can often be observed and the satisfaction of managers is particularly high. It should be
noted that the design of the PMS is often continuous, with the characteristics of the system
(e.g., the monitored performance dimensions)
that are modied over time to mirror the changes
in the competitive strategy and the environment
in which the rm operates (see, e.g., Companies
F, N and M). Although costly, this approach
seems to be particularly useful to improve the
effectiveness of the PMS and managers satisfaction with it.

6. Conclusions
This paper adopts a systemic and contextual
perspective to look into the problem of measuring
R&D performance. In particular, it explores the
interplay between measurement objectives, performance dimensions and contextual factors in
the design of a PMS for R&D activities. With this
aim, we rst developed a reference framework
that identies: (i) the main contextual factors that
might affect the importance a rm attaches to
different objectives for R&D performance measurement and the approaches it uses to measure
R&D performance (i.e. type of R&D activity,
industry belonging and size); (ii) the main aspects
that should be looked at when designing a PMS
for R&D (i.e. dimensions of performance and
indicators). This framework was used as a reference model for the subsequent empirical analysis,
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

Performance measurement in R&D


Table 3. Performance perspectives and indicators
Performance perspective

Type of
indicator

Indicators

Financial perspective

Input

Total cost of each R&D project (4)


R&D annual spending (3)
R&D annual investment (2)
Cost for acquiring a new technology (2)
Present Value of R&D accomplishments/R&D expenditures (1)
IRR or NPV due to R&D projects (5)
Prots due to R&D (3)
ROI due to R&D projects (3)
Sales (or % of sales) from new products (2)
Cost (or % of cost) reduction from new projects (2)
Market share due to R&D and innovations (2)
No. of interactions with customers during the project (6)
% of budget dedicated to customer analysis or verication (3)
% of customer driven projects (1)
No. of customers included in the project team (1)
Time to market (4)
Engineering hours on projects/engineering hours on projects and
troubleshooting (2)
No. of training sessions signed off by customer and delivered (1)
No. of problem analysis reports requested and delivered (1)
No. of customer complaints (4)
Customer satisfaction (2)
% of support requests fullled (2)
No. of new customers (1)
Response time to customer requests for specials (1)
No. or % of people with management experience (2)
No. of employees in R&D (1)
No. of hours of staff training (5)
% of suggestions implemented (2)
No. of meeting or time dedicated to the analysis of reasons for failure of
previous projects (2)
Capability to acquire new bodies of competencies (2)
No. of new ideas per year (4)
No. of innovations delivered to production and commercialization (4)
No. of citations of the researchers publications (4)
No. of publications (3)
No. of patents registered/pending (3)
% of patent applications that resulted in registered patents (3)
Average product life-cycle length (3)
No. of improvements suggestions per employee (3)
Scientic excellence of the new ideas identied per year (2)
Market attractiveness of the new ideas identied per year (2)
International relevance of the competencies acquired during 1 year (2)
No. of products in development or projects in course (2)
No. of new processes and signicant enhancements per year (1)
Experience of R&D employees (2)
No. or % of employees involved in goal setting (2)
Availability (knowledge) of advanced managerial tools, e.g., project
management techniques (2)
Availability (knowledge) of advanced IT support tools, e.g., rapid
prototyping and design support tools (1)
% of projects respecting costs and budget (6)
Agreed milestones/objectives met (6)
Quality of documentation to development (5)
Average annual improvement in process parameters, e.g., quality cost,
lead time, WIP, reliability, capability, down time (4)
% of collaboration objectives fully satised (3)
% of projects that lead to new or enhanced products or processes (2)
% of R&D expenditures that lead to new or enhanced products or
processes (2)
Time spent on changes to original product/project specication (2)

Process
Output

Customer perspective

Input

Process

Output

Innovation and learning


perspective

Input
Process

Output

Business process perspective

Input

Process

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

R&D Management 39, 5, 2009

501

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini


Table 3. (Contd.)
Performance perspective

Type of
indicator

Output

Indicators
% of projects using a common design platform (2)
Rate of re-use of standard designs/proven technology (2)
Average product/service cost variance (2)
No. of collaborations stipulated/no. of collaboration opportunities
identied (2)
Sum of revised project durations/sum of planned durations (1)
Average project delay (5)
% of projects delayed or cancelled due to lack of funding (5)
% of projects delayed or cancelled due to lack of human resources (4)
Rate of successful projects, i.e. project achieving the assigned time, cost,
quality (4)
% of project milestones completed (4)
Product quality, measured through indicators specic for each industry/
product (3)
No. or % of products/projects completed (2)
Degree of project completion (2)
Total product development time (2)
% of on time deliveries of specication to manufacturing (2)
Average time of re-design (1)
No. of customer detected design faults (1)

In parentheses the number of rms that used the specic metric.

which involved 15 technology-intensive rms actively engaged in R&D activities.


The results of the empirical investigation,
which were discussed at length in the previous
section, indicate the existence of three archetypal
models that companies adopt to measure the
performance of their R&D, which represent internally consistent sets of contextual variables,
objectives for performance measurement and
performance dimensions. They are schematically
represented in Figure 5. The existence of these
models suggests that rms use performance measurement in R&D to pursue different types of
objectives. In particular, two distinct clusters of
rms emerge: one that uses performance measurement with the main purpose of exerting control
over R&D activities and support critical management decisions (diagnostic objectives) and the
other that conceives performance measurement
mainly as a means to improve the motivation of
researchers (motivational objectives). Another
set of objectives is concerned with the capability
of performance measurement to improve coordination and communication, streamline the execution of complex interrelated tasks and favour
organisational learning (interactive objectives).
They seem to be pursued by rms in combination
with diagnostic or motivational objectives. The
analysis also reveals that the importance rms
attach to each class of objectives is signicantly
inuenced by the context in which measurement
502

R&D Management 39, 5, 2009

takes place (see Figures 4 and 5). It also emerges a


specialisation in the performance dimensions used
to measure the different classes of objectives
(see Figure 5). In particular, diagnostic objectives are pursued mainly through the use of
nancial- and customer-related measures,
whereas indicators associated with the innovation
and learning perspective are the most widespread
among companies pursuing motivational objectives. Contextual factors do not seem to directly
affect the choice of the performance dimensions
used in the PMS. Their impact is in fact mediated
by the classes of objectives the rm decides to
pursue. The paper also provides a synoptic view
of the indicators (or metrics) that the rms
investigated in the scope of our research use to
measure each dimension of performance (see
Table 3). No relevant specialisation in the use of
a given type of indicators emerges along the
different perspectives for performance measurement or classes of objectives.

6.1. Implications for managers


Although the results of the paper should be better
conceived in an exploratory fashion, we believe
they hold valuable implications for R&D managers and, especially, for the heads of R&D units
and departments who are interested in designing a
PMS for the organisation they are responsible for.
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

Performance measurement in R&D


First, they are given a number of examples about
how a relevant sample of technology-intensive
rms have designed and used a PMS in their
R&D units, with the aim of pursuing different
classes of objectives under the inuence of dissimilar contextual variables. This rich body of
empirical evidence will provide R&D managers
with a number of insights useful to design a PMS
that is appropriate to the context in which they
operate. The synoptic view of the indicators used
by the rms in the sample (see Table 3) can be a
particularly helpful starting point for the denition of the set of metrics to be used in their PMSs.
Second, our analysis suggests that introducing
and using a PMS in a basic and applied research
unit to pursue motivational objectives can be
particularly challenging and costly. Some approaches that are likely to improve R&D managers satisfaction with this type of PMS appear
to be: use of an incremental approach to establish
the PMS into the organisation (e.g., using it rst
to measure the most repetitive and predictable
tasks); continuous re-design of the PMSs characteristics to incorporate the suggestions for improvement coming from researchers; and a deeper
involvement of researchers in the measurement
process. Finally, the cases in our sample suggest
that a continuous re-design of the PMS to mirror
the evolution of the rms competitive and R&D
strategy is an important ingredient of success,
notwithstanding the main purposes for which
the PMS is designed and used.

6.3. Limitations and future research


The study obviously has some limitations. First,
because of the adopted research methodology,
results cannot be statistically generalised; they
can only be analytically extended to other industrial rms operating in technology-intensive industries. Even if the internal validity of the
empirical results is ensured by the cross-case,
explanation-building and pattern-matching analyses, the study does not explicitly take into
account the effects that other contextual factors
are likely to have on the choice of the objectives
and the characteristics of the PMS elements.
Therefore, further research should be aimed at
exploring the joint effects of other contextual
factors (e.g., the rms R&D strategy) on the
design of the PMS. Second, our analysis is exploratory in intent. Although we provide some
evidence on the satisfaction of R&D managers
and executives with their measurement system, we
do not systematically assess the capability of a
PMS with specic characteristics to accomplish,
in a given context, the objectives for which it has
been designed. This represents an interesting
avenue for future research, which would require
the development of an appropriate measure of
effectiveness for an R&D PMS and a statistical
analysis of a representative sample of rms.
Finally, we believe that adopting a longitudinal
perspective to study the organisation-wide impacts associated with the adoption of the R&D
PMS is another interesting avenue for future
investigation.

6.2. Implications for research


The paper adds to our understanding of performance measurement in R&D because it is one of
the rst contributions, to our best knowledge,
that systematically studies the objectives for
which a rm decides to measure its R&D activities performance. Moreover, it explores whether
established concepts in management accounting
research (e.g., the relationship between objectives
for performance measurement and characteristics
of the PMS) can be applied in R&D settings as
well. Finally, the approach adopted in the paper
can encourage researchers in the eld of R&D
performance measurement to investigate whether
and how the other dimensions of the performance
measurement context (e.g., the rms R&D strategy or the R&D organisational structure) inuence the importance a rm attaches to different
classes of objectives.
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

References
Azzone, G. (2006) Sistemi di controllo di gestione.
Metodi, strumenti e applicazioni. Milano: Etas.
Bayus, B.L. (1994) Are product life cycles really getting
shorter? Journal of Product Innovation Management,
11, 300308.
Bititici, U.S., Turner, T. and Begemann, C. (2000)
Dynamics of performance measurement systems.
International Journal of Operations and Production
Management, 20, 692704.
Bremser, W.G. and Barsky, N.P. (2004) Utilizing the
balanced scorecard for R&D performance measurement. R&D Management, 34, 229238.
Brown, M.G. and Svenson, R.A. (1988) Measuring
R&D productivity. Research-Technology Management, 31, 1115.
Brown, M.G. and Svenson, R.A. (1998) Measuring
R&D productivity. Research-Technology Management, 41, 3035.
R&D Management 39, 5, 2009

503

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini


Chenhall, R.H. and Morris, D. (1986) The impact of
structure, environment, and interdependence on the
perceived usefulness of management accounting systems. The Accounting Review, 61, 1635.
Chiesa, V. (1996) Separating research from development: evidence from the pharmaceutical industry.
European Management Journal, 14, 6, 638647.
Chiesa, V. and Frattini, F. (2007) Exploring the differences in performance measurement between research
and development: evidence from a multiple case
study. R&D Management, 37, 283301.
Chiesa, V., Frattini, F., Lazzarotti, V. and Manzini, R.
(2008) Designing a performance measurement system
for the research activities: a reference framework and
an empirical study. Journal of Engineering and Technology Management, 25, 213226.
Cooper, R. (1993) Winning at New Products: Accelerating the Process from Idea to Launch. Reading:
Addison-Wesley.
Davila, T. (2000) An empirical study on the drivers of
management control systems design in new product
development. Accounting, Organisations and Society,
25, 383409.
Driva, H. and Pawar, K.S. (1999) Performance measurement for product design and development in a
manufacturing environment. International Journal of
Production Economics, 60, 6168.
Driva, H., Pawar, K.S. and Menon, U. (2000) Measuring product development performance in manufacturing organizations. International Journal of
Production Economics, 63, 147159.
Eisenhardt, K.M. (1989) Building theories from case
study research. Academy of Management review, 14,
4, 532550.
Eisenhardt, K.M. and Graebner, M.E. (2007) Theory
building from cases: opportunities and challenges.
Academy of Management Journal, 50, 2532.
Godener, A. and Soderquist, K.E. (2004) Use and
impact of performance measurement results in
R&D and NPD: an exploratory study. R&D Management, 32, 191220.
Gordon, L.A. and Miller, D.A. (1976) A contingency
framework for the design of accounting information
systems. Accounting, Organizations and Society, 1,
5969.
Gordon, L.A. and Narayanan, V.K. (1984) Management accounting systems, perceived environmental
uncertainty and organizational structure: an empirical investigation. Accounting, Organizations an Society, 9, 3347.
Hauser, J.R. (1998) Research, development and engineering metrics. Management Science, 44, 16701689.
Kaplan, R.S. and Norton, D.P. (1992) The balance
scorecard measures that drive performance. Harvard Business Review, 70, 7179.
Kerssen-van Drongelen, I.C. and Bilderbeek, J. (1999)
R&D performance measurement: more than choosing a set of metrics. R&D Management, 29, 35
46.

504

R&D Management 39, 5, 2009

Kerssen-van Drongelen, I.C. and Cook, A. (1997) Design principles for the development of measurement
systems for research and development processes.
R&D Management, 27, 345357.
Kerssen-van Drongelen, I.C., Nixon, B. and Pearson,
A. (2000) Performance measurement in industrial
R&D. International Journal of Management Reviews,
2, 111143.
Kim, B. and Oh, H. (2002) An effective R&D PMS:
survey of Korean R&D researchers. International
Journal of Management Science, 30, 1931.
Kodama, F. (1995) Emerging Patterns of Innovation.
Boston: Harvard Business School Press.
Loch, C.H., Stein, L. and Terwiesch, C. (1996) Measuring development performance in electronics industry. Journal of Product Innovation Management,
13, 320.
Loch, C.H. and Tapper, S. (2002) Implementing a
strategy-driven performance measurement system
for an applied research group. Journal of Product
Innovation Management, 19, 185198.
Locke, E.A. (1968) Toward a theory of task motivation
and incentives. Organizational Behaviour and Human
Performance, 3, 157189.
Miles, M.B. and Huberman, A.M. (1984) Qualitative
Data Analysis. Newbury Park: Sage.
Moizer, P. (1991) Performance appraisal and rewards.
In Ashton, D., Hopper, T. and Scapens, R.W. (eds.),
Issues in Management Accounting. Englewood Cliffs:
Prentice Hall, pp. 126144.
Neely, A. (1999) The performance measurement revolution: why now and what next? International Journal
of Operations and Production Management, 19, 205
228.
Nevens, T.M., Summe, G.L. and Uttal, B. (1990)
Commercializing technology: what the best Companies do. Harvard Business Review, 68, 154163.
Nixon, B. (1998) Research and development performance measurement: a case study. Management
Accounting Research, 9, 329355.
Ojanen, V. and Vuola, O. (2006) Coping with the
multiple dimensions of R&D performance analysis.
International Journal of Technology Management, 33,
279290.
Ouchi, W. (1979) A conceptual framework for the
design of organizational control mechanisms. Management Science, 25, 9, 833848.
Pappas, R.A. and Remer, D.S. (1985) Measuring R&D
productivity. Research Technology Management, 28,
1522.
Pawar, K.S. and Driva, H. (1999) Performance measurement for product design and development in a
manufacturing environment. International Journal of
Production Economics, 60-61, 6168.
Pillai, A.S., Joshi, A. and Rao, K.S. (2001)
Performance measurement of R&D projects in a
multi-project, concurrent engineering environment.
International Journal of Project Management, 20,
165177.
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

Performance measurement in R&D


Poh, K.L., Ang, B.W. and Bai, F. (2001) A comparative analysis of R&D project evaluation methods.
R&D Management, 31, 6376.
Pritchard, R.D. (1990) Measuring and Improving Organizational Productivity: a Practical Guide. New York:
Praeger.
Rockness, H.O. and Shields, M.D. (1988) An empirical
analysis of the expenditure budget in research and
development. Contemporary Accounting Research, 4,
568581.
Simons, R. (1994) How new top managers use control
system as levers of strategic renewal. Strategic Management Journal, 15, 169189.
Simons, R. (1995) Levers of control How Managers Use
Innovative Control System to Drive Strategic Renewal. Boston: Harvard Business School Press.
Simons, R. (2000) Performance Measurement & Control
Systems for Implementing Strategy. Upper Saddle
River: Prentice Hall.
Sivathanu, P.A. and Srinivasa, R.K. (1996) Performance monitoring in R&D projects. R&D Management, 26, 5765.
Tipping, J.W., Zeffren, E. and Fusfeld, A.R. (1995)
Assessing the value of your technology. ResearchTechnology Management, 38, 2239.
Tuomela, T.S. (2005) The interplay of different levers
of control: a case study of introducing a new performance measurement system. Management Accounting Research, 16, 293320.
Werner, B.M. and Souder, W.E. (1997) Measuring
R&D performance State of the art. ResearchTechnology Management, 40, 3442.
Wind, J. and Mahajan, V. (1997) Issues and opportunities in new product development: an introduction
to the special issue. Journal of Marketing Research,
34, 112.
Wolf, M. (2006) The world must get to rips with seismic
economic shifts. Financial Express, February, 7.
(available at: http://www.nancialexpress-bd.com).
Wouters, M. and Wilderom, C. (2008) Developing
performance-measurement systems as enabling formalization: a longitudinal eld study of a logistics
department. Accounting, Organizations and Society,
33, 488516.
Yin, R.K. (2003) Case study Research: Design and
Methods. London: Sage.

Notes
1. Kodama (1995) classies industrial sectors into
dominant design, high-tech and science-based industries. These three clusters differ with respect to
their level of risk, dened as the probability that an
R&D program expenditure is frozen in an intermediate phase of the R&D process (freezing rate).
Dominant design industries are characterised by a
freezing rate that dramatically decreases from basic
through applied research, coming to zero in the
development phase. The dominant design industries
r 2009 The Authors
Journal compilation r 2009 Blackwell Publishing Ltd

are: food, textile, pulp and paper, printing and


publishing, oil and paints, petroleum and coal,
rubber, ceramics, iron and steel, transportation,
energy. High-tech industries are characterised by a
freezing rate that decreases throughout the R&D
process, but remains 40 even in the late development. High-Tech industries are: ordinary machinery, electrical machinery, communications and
electronics, precision equipment and aerospace.
Science-based industries are those in which the
freezing rate always remains high throughout the
entire R&D process. Science-based industries are
basically pharmaceuticals and industrial chemicals.

Vittorio Chiesa is Professor of R&D Strategy and


Organisation at Politecnico di Milano. He is
member of the Management Committee of MIP
the Business School of Politecnico di Milano,
where he is responsible for the Technology Strategy area. His main research interests concern
R&D management and organisation, technology
strategy and international R&D. He has published 6 books and more than 100 papers, including 40 articles in leading international journals
such as the Journal of Product Innovation Management, IEEE Transactions on Engineering Management and International Journal of Operations
and Production Management.
Federico Frattini (Corresponding author) is Assistant Professor at the Department of Management, Economics and Industrial Engineering of
Politecnico di Milano. His research interests
concern R&D performance measurement, the
organisation of R&D activities and the commercialisation of innovation in high-tech markets. He
has published more than 30 papers, including
articles in R&D Management, Journal of Engineering and Technology Management, and International Journal of Technology Management.
Valentina Lazzarotti is Assistant Professor at
Universita` Carlo Cattaneo LIUC (Castellanza,
Varese). She teaches Economics and Business
Organization and Management Control Systems
at LIUC. She obtained her Master Degree in
Economics at Universita` Bocconi (Milano). Her
research interests concern R&D performance
measurement and organization. She has published more that 30 papers including articles in
Journal of Engineering and Technology Management and International Journal of Technology
Management.
R&D Management 39, 5, 2009

505

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini


Raffaella Manzini is Associate Professor at
Universita` Carlo Cattaneo LIUC (Castellanza,
Varese). She teaches Economics and Business
Organization and Technology Strategy at LIUC
and Politecnico di Milano. Her research interests
concern technology strategy and planning, R&D
management and organization. She has published
more than 60 papers including articles in leading
journals such as Long Range Planning and R&D
Management.

Appendix A. Interview protocol


 What kind of products or services does your
company supply? Which are the distinctive
characteristics of the industry in which it
operates? How high is the probability that
an R&D expenditure is frozen in an intermediate phase of R&D process? How long
does this failure rate remain higher than zero?
Is it higher than zero also in the downward
phases of the R&D process, i.e. development
and testing? How do these characteristics
affect the challenges your rm is confronted
with in R&D?
 Which are your companys core competences?
Which are the most critical for your competitive advantage?
 What percentage of your companys annual
turnover is invested in research and/or development activities?
 What kind of R&D activity does your company carry out? Is it basic and applied research or new product development? Is it
undertaken in distinct organisational units?
Which are the main characteristics of your
R&D organisation? Which is the scope of the
activities undertaken in the R&D unit we are
focusing on?

506

R&D Management 39, 5, 2009

 How many people does your company employ? How many scientists and researchers are
employed in R&D? And how many in the
units we are focusing on?
 How long have you been measuring the performance of your R&D unit? Which are the
main characteristics of the performance measurement system you have been employing?
Have they evolved over time?
 Why have you decided to measure R&D
performance? Which are the main objectives
that you pursue through R&D performance
measurement? How would you rate them on
the basis of their importance for your R&D or
competitive strategy? Are measurement objectives mutually exclusive to some extent?
 What kind of performance dimensions does
your company measure in order to pursue the
established objectives? Are these performance
dimensions completely controllable by researchers and engineers? Is the choice of the
performance dimensions inuenced by the
objectives you wish to pursue? Are there any
other reasons behind the choice of these
dimensions?
 Which indicators are employed to operatively
measure performance? Do you use different
indicators to measure different performance
dimensions? Is the choice of the indicators
constrained by any reasons (e.g., incompatibility with the corporate-level PMS)?
 Did your company suffer from any constraints of human and nancial resources in
designing and implementing the performance
measurement system? Is the set of objectives
that your rms pursue, or the performance
dimensions that it employs, somehow limited
by a lack of resources? And what about the
degree of measurement formalization?

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

Performance is measured mainly


on the basis of the patent
applications submitted by
researchers to the internal
intellectual property department.

Monitored dimensions
of performance

Main objectives for


R&D performance
measurement

Type of R&D
Size of R&D unit

Multinational rm operating in
the highly dynamic electronics and
semiconductor industry
Annual turnover of about h8bn,
15% of which is invested to R&D,
where thousands of people are
employed
Basic and exploratory research,
which is believed to be critical for
the rms competitiveness, is
undertaken at the corporate level
Basic and applied research
About 700 people employed in the
basic and applied research unit
Managers believe that in their
context, the capability to
adequately motivate scientists and
researchers is of foremost
importance, together with the
assessment of their capacity to
contribute to the rms
innovativeness
In a very large R&D unit, with
people working in different
departments and collaborating on
long-lasting projects, the PMS
should also serve the important
purpose to enhance coordination,
communication and collective
learning

General characteristics

Company A

Company C

Company D

Multinational rm operating
mainly in the electronics for
industrial applications industry. Its
core product line comprises
uninterruptible power supplies (or
UPS)
Annual turnover of about h120
million, 5% of which is invested in
new product development activities

Medium Italian manufacturer of


advanced machining centres, which
employs 150 people and has a
turnover of about h15 million
The competitiveness of the rm
mainly depends on its ability to
employ novel and state-of-the-art
technologies to customise and
tailor products to the needs and
technical requirements of its
established client basis
New product development
New product development
About 100 engineers and
15 technicians and engineers are
technicians work in the NPD unit employed in the NPD unit
Managers state that the
Managers believe that NPD
performance measurement should measurement of the performance in
the development unit is undertaken
be unavoidably aimed at
mostly in an informal and
monitoring and controlling the
unstructured manner, although
progress of project activities, so
that possible variances from time they are very much interested in
studying other formal approaches
and quality targets are timely
adequate to their needs
spotted and corrective actions
The measurement of the
introduced
Performance measurement is also development unit performance is
basically aimed at motivating
necessary here to keep under
control product development costs, engineers and technicians;
which is a pre-requisite for the in- according to the companys
managers, this is critical in small
progress assessment of the new
organisations where personal
product protability and a basis
forms of control are predominant
for possible discontinuation
decisions

Large Italian rm (it employs


about 1,800 people) operating in
the aerospace industry
In the last years, it has been
developing a new military trainer,
which was particularly critical for
its competitive position and
required the establishment of a
new NPD unit, where more than
300 engineers and technicians have
been employed
New product development
More than 300 engineers are
employed in the NPD unit
A formal and very articulated
performance measurement system
was designed and is still used in
this department. The head of the
development unit explains that the
measurement system was designed
with the idea to pursue several
objectives
First, there was the need to
monitor the progress of activities
and to continuously evaluate the
respect of target costs and hence
protability objectives
Another critical purpose was to
stimulate coordination and
communication among the several
people belonging to different
functional areas who are put
together to pursue such complex
tasks
For each NPD project which is to The measurement system assesses The new airplane project was
be launched, a joint team made of innovative capability and rewards decomposed in the whole set of
the NPD head, representatives
the number (and potential value) of work packages that it entailed. To
each work package specic target
new ideas for product
from the sales and marketing
costs, resource consumption
improvements put forward by
functions, and from the

Company B

Appendix B. Summary of case study evidence. As far as performance indicators are concerned, this appendix contains only some examples of the metrics used in each rm

Performance measurement in R&D

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

R&D Management 39, 5, 2009

507

508

Indicators used to
measure dimensions
of performance

Appendix B. (Contd.)
Company D
objectives and temporal
milestones were associated
The key point to favour
coordination and learning,
according to the head of the
development unit, is the
interactive use of these
measurement dimensions, which
entails a shared denition of the
objectives to be achieved and a
continuous discussion and face-off
with engineers about the measured
performance and the possible
variance in respect to standards

Efciency in undertaking NPD


activities is measured through a
very high number of indicators
that change along the phases of
the process: (i) for the
development phase, the number of
approved drawings/month, the
number of stipulated purchasing
contracts/month, the average
delay in achieving a milestone
were used; (ii) for the assembling
and ground testing phase, the
number of built components/
week, the number of functional
tests/week, the number of ground
tests/week, the number of
milestones where target costs are
met were employed; (ii) for the
ight test phase, the number of

Company C
engineers
Moreover, the efciency with
which they carry out specic tasks
in the NPD process is evaluated in
terms of consumption of resources
and respect of scheduled
milestones. The achievement of
these targets is assessed mainly on a
subjective basis

Innovative capability is evaluated


measuring the following main
indicators: the number of ideas for
new products that are submitted
for evaluation; the % of these ideas
that pass formal evaluation and
turn into R&D projects; the
scientic excellence of these ideas,
assessed through a peers
evaluation; the average product
life-cycle length
Prociency with which NPD
process are carried out is evaluated
mainly on the basis of: the % of
projects for which resource
consumption targets are satised;
the average delay in completing
temporal milestones; the % of
projects that use a common design

Company B
manufacturing and operations one,
meet to agree on a product plan
that establishes target costs,
temporal milestones and technical/
functional requirements. At each
milestone, the achievement of the
intermediate targets is measured
These requirements are dened on
the basis of the clients needs that
are identied through market
analyses and focus groups. The
capability to identify and take into
proper account these customers
requirements is an important
performance for Company B
Financial evaluation of NPD
projects is undertaken with the
purpose to support prioritisation
and discontinuation decisions
To measure efciency with which
NPD processes are carried out, the
following indicators are used: % of
milestones respected; average delay
in the completion of project
milestones; % of projects that are
delayed or postponed; % of
projects that run out of budget;
degree to which technical
specications are met
The capability to take into account
the voice of the customers in the
execution of NPD projects is
evaluated on the basis of: the
number of meetings with clients;
the number of complaints that are
received from the products early
customers; a subjective evaluation
of the quality of project

Company A

Sometimes it happens that


patents are registered with the
only aim to motivate researchers,
although they are not useful for
the rms technology strategy.
Researchers contribution to the
rms innovative capability is
therefore the most important
dimension of performance in the
PMS used by Company A
Managers established to evaluate
also the efciency with which
specic business processes (e.g.,
identication, analysis and
implementation of customers
requirements) are carried out
Similarly, the capability to acquire
new bodies of competencies,
dened on an annual basis, is
evaluated through peers
assessment
To measure the innovation and
learning capability, the following
main indicators are employed:
number of patents applications
submitted to the intellectual
property department, % of
patents applications that result in
registered patents; number of
scientic publications; number of
citations of the researchers
publications; number of
improvement suggestions per
employee
To measure prociency in
undertaking business processes,
the following main indicators are
employed: quality of project
documentation; % of agreed
milestones, % of projects

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini

R&D Management 39, 5, 2009

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

Critical aspects
associated with the
use of the PMS

Appendix B. (Contd.)

ight tests/month was the main


indicator

Since the beginning of the


development project of the new
aircraft, a strong commitment
from top management toward the
design and introduction of a
complete and sophisticated PMS
existed. This helped mobilise the
nancial and human resources
required to design and run an
appropriate performance
measurement system
The PMS was positively accepted
by the development team, perhaps
as the result of a high number of
training sessions and kick-off
meetings where the use and
functioning of the system were
explained. Since the beginning,
engineers perceived the PMS as a
very useful instrument to keep
everyone informed about the
development project, and they
particularly appreciated the
interactive approach with which
standards and frequencies were
dened

platform; the average annual


improvement in process
parameters (e.g., lead time); the
degree to which advanced
managerial tools (e.g., project
management techniques) are
employed

The rms management has


struggled for years to introduce a
PMS that could motivate the
engineers working in the small
NPD unit. However, the rst
attempts have completely failed,
with some technicians leaving the
rm because of the contrasts that
aroused after the introduction of
the rst performance measures.
They believed that using a PMS
will compromise the comfortable
working place that a small NPD
unit ensured
Different changes to the set of the
employed measures have been
undertaken over the years, with the
aim to overcome engineers
complaints (e.g., substituting
individual measures with projectbased or departmental indicators)
Generally speaking, the rms
management is not satised with
the current characteristics of the
PMS, and is still searching indeed
for opportunities to further
improve it.

documentation that is delivered to


internal customers; % of
engineering hours dedicated to
troubleshooting; response time to
customers request for specials
The nancial evaluation of NPD
performance is undertaken taking
into account: the net present value
of the new product under
development, carried out under
different scenarios; the total costs
for each project; the % of prots
due to R&D
The PMS has been in place, albeit
with different degree of
completion, for more than 20
years. Over the years, the set of
performance dimensions and
metrics used to assess performance
has evolved to mirror the evolution
of the rms competitive strategy
and environment. In particular,
customer-related and nancial
measures have been recently
introduced with the intensication
of competition and the
introduction of a customer-centric
product development strategy
No signicant problems are
encountered in the use of the PMS,
which is very well accepted by
engineers. They believe in fact it is
an important instrument to
objectively evaluate their efforts.
The only complaints are about the
frequency with which certain
indicators are measured, which
sometimes convince managers to
relax these requirements of the
PMS

respecting target costs and


budgets, average annual
improvement in process
parameters
The capability to acquire new
bodies of competencies is assessed
through a peers evaluation

The PMS has been in place for


about 5 years. The rst attempts
to introduce performance metrics
in the corporate R&D unit were
strongly opposed by researchers,
who perceived their autonomy
could be signicantly hampered.
Moreover, they believed that the
outcome of their activities was too
much uncertain and distant in
time to be objectively measured.
Therefore, managers decided to
gradually introduce the PMS,
starting from the departments and
projects that have shown a lower
resistance toward performance
measurement
Today managers are very satised
with the PMS, although they
acknowledge that it is very much
expensive to run, since it requires a
highly interactive use of
performance metrics and a
continuous bargaining of targets
and standards. They believe it
would be hardly impossible to run
such a complex system in a smaller

Company D

Company C

Company B

Company A

Performance measurement in R&D

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

R&D Management 39, 5, 2009

509

510
Company F

Management is however very much


satised by the use of the PMS

rm, with more limited nancial


and human resources

Company E

Company B

Company A

Company G

Company C

Company H

The management team is very


satised with the PMS, which is
being transferred to other parts of
the organisation

Company D

Firm operating in the cosmetics


and non-prescription drugs
industry with a workforce of 500
people
The largest part of development
projects is dedicated to testing
activities through which new
combinations of compounds and
active ingredients are evaluated in
terms of effectiveness, toxicology
and other properties

The rm was founded in 2001 and


it operates in the eld of
pharmaceutical research, mainly
offering services with a high
technical and scientic content
and capable of supporting the new
drugs R&D process
It is a typical Contract Research
Organisation (CRO) with a strong
specialisation in lead optimisation
and competencies in
combinatorial chemistry

The rm is a multinational
chemical enterprise with more
than 20,000 employees that is
specialised in the development,
manufacturing and supply of
additives, polymers, pigments and
colouring materials for a wide
range of applications
Basic and applied research is
undertaken in 12 different
countries and it employs more
than 1,600 researchers. It is
organised around a number of
thematic platforms

One of the largest Italian


aerospace rms, it employs more
than 9,000 people and is
specialised in the development and
manufacturing of helicopters for
civil and military purposes
The rm started in 2005 the
development of a new helicopter
for military use that represents an
important project for its
competitive position. An
organisational unit was
established to pursue the
development of this new product,
that was organised around ve
Integrated Product Teams (Basic
Vehicle, Missilisation, SEIT, Test
Evaluation, ILS) coordinated by a
Program Manager and divided
into operative IPTs.
Type of R&D
New product development
Basic and applied research
Basic and applied research
New product development
About 200 people work in the
Size of R&D unit
More than 50 people employed in About 60 people (mainly chemists, Research is organised around
organisational unit devoted to the
more than 20 units worldwide.
the unit where new products are the largest part of which own a
developed and tested
PhD) work in the organisational The focus of the case study is the development of the new helicopter
Italian unit, where more than 200
unit where research for the
optimisation of novel molecules is researchers are employed
carried out
Managers acknowledge that
Managers believe that in a context The main purpose for which
Main objectives for R&D Managers acknowledge the
performance measurement in a
where results are distant in time performance measurement has
performance measurement importance of measuring
been introduced and is currently very complex NPD environment
and tangible outcomes often
development activities
used in the rms research unit is has the main purpose to keep
impossible to see, performance
performance in order to closely

General characteristics

Appendix B. (Contd.)

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini

R&D Management 39, 5, 2009

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

Monitored dimensions of
performance

Appendix B. (Contd.)
Company F

Company G

monitor their progress and to


support decisions regarding the
termination and prioritization of
projects
Nevertheless, they believe that it is
vital to motivate researchers
working in the development
department, in order to prevent
lack of commitment during the
long phases of the development
process

measurement should be dedicated


rst and foremost to motivate and
stimulate scientists and
researchers.
This is critical also to direct
researchers efforts toward
innovation objectives that are
relevant for the whole
organisation. In this respect,
performance measurement is
adopted also with the purpose to
support the adoption of an MBO
rewarding system
Overall, the ultimate goal of the
performance measurement system
if to improve the performance of
research activities

to motivate researchers toward


the long-term objectives of the
different research platforms they
are currently working on. This is
critical for tunnelling individual
creativity toward organisationwide objectives and hence to
improve the whole R&D
effectiveness
Performance measurement is also
critical to support the allocation of
resources to the different
platforms where the rm is active
The size of the R&D unit and the
heterogeneity of the competencies
that it includes, makes it also
important that the PMS serves the
purpose to streamline a process of
collective sharing of new
competencies and knowledge, as
well as an effective ow of
communication between
organisationally separate
departments
Motivation of researchers is
The capability to nd an
The satisfaction of established
achieved measuring their
innovative solution to the
target costs and the respect of
contribution to the innovative
technical problems that surface
scheduled milestones are
during research activities and to results obtained in each thematic
measured and researchers
platform
add to the rms innovative
rewarded on the basis of their
potential, are the main dimensions Nevertheless, since this
results along these dimensions.
performance is not completely
Furthermore, the achievement of along which the performance of
controllable by the researchers,
R&D is evaluated.
the established quality levels for
their capability to deliver outputs
the cosmetic under development is The need to use performance
with the required quality
assessed, although this has not a measurement also as a basis for
specications and within the
rewarding researchers made it
direct inuence on the
established budget is also
necessary to introduce
remuneration of employees.
Therefore, focus in company E is performance dimensions that are considered
more directly controllable by
on performance measurement
along an internal business process individuals (or small groups of
researchers). Therefore,
perspective

Company E

In order to control the progress of


project activities, it is necessary
rst of all to focus on the efciency
with which critical NPD processes
and activities are carried out.
Moreover, nancial evaluation of
the project protability is adopted.
Nevertheless, an internal-oriented
perspective implies the risk that
product development looses
contact with the needs of the
customers, that are the ultimate
recipients of the output of the
activities. Therefore, a customerrelated perspective in performance
measurement is added

under control the progress of the


project, to evaluate on an ongoing
basis its protability, and to avoid
that it runs out of budgets or
temporal deadlines
Nevertheless, it emerges that it has
another important purpose, i.e.
improve the coordination among
the different people that take part
to the development project coming
from different departments

Company H

Performance measurement in R&D

R&D Management 39, 5, 2009

511

512
Company F
performance in undertaking
specic business processes and
activities (e.g., acquiring new
technologies and competencies,
scouting licensing and partnering
opportunities, collaborating with
external organisations) is also
evaluated

As far as innovative capability is


concerned, the following
indicators are employed: hours
dedicated to staff training;
international relevance of
technologies/competencies
acquired during 1 year; number of
citations of the publications of
companys researchers; % of the
novel identied molecules with the
required technical characteristics
Performance in undertaking
business processes is instead
measured on the basis of: % of
projects concluded on time;
average service cost variance;
number of collaboration
stipulated/number of
collaboration opportunities
identied; % of collaboration
objectives fully satised

Company E

Motivation and alignment of


individual and corporate
objectives are pursued measuring
innovation and learning
capabilities. The number and
quality (evaluated by peers) of
new product ideas developed by
engineers is used as an important
measure in this respect

Indicators used to measure Efciency in carrying out


development activities is measured
dimensions of
mainly on the basis of: quality of
performance
documentation produced during
testing activities; the average delay
in achieving project milestones;
the % of milestones where target
costs are not met; the number of
employees involved in goal setting
and planning activities; the
average experience of R&D
employees
Innovative capability is evaluated
through the following indicators:
hours dedicated to staff training
on new methods and technologies;
number of meetings dedicated to
the analysis of the reasons behind
project failures; number of ideas
for new products submitted to a
dedicated commission; % of these
ideas that turn into projects for
new products development;
subjective evaluation by peers of
the technical feasibility and
commercial value of these ideas

Appendix B. (Contd.)

Contribution to the innovative


outputs of each research platform
is evaluated through: nonnancial scoring methods based
on peers judgment; number of
novel ideas for technical problems
submitted to the head of the
platform; % of these ideas that are
selected and implemented; number
of improvement suggestions per
employee; number of products in
development or projects in course
Efciency in carrying out specic
activities is evaluated on the basis
of: quality of the delivered
documentation; % of technical
requirements that are met at each
project milestone; % of activities
that run out of budget; rate of
successful projects

Company G
Finally, managers note that
coordination and information
sharing among the members of the
NPD teams does not need
different performance dimensions
to be monitored. It requires
instead that engineers and
technicians are deeply involved in
the evaluation process (e.g.,
denition of the targets, analysis
of the results) so that internal
communication and visibility over
other technicians activities is
favoured
Business processes efciency is
measured using two major
quantitative indicators: (i) the
Cost Performance Index (a
numerical indicator which
evaluates the efciency of the
NPD project in terms of costs and
resource consumption); (ii) the
Schedule Performance Index (a
numerical indicator which
determines the efciency of the
project in terms of punctuality)
Financial evaluation employs the
following main indicators: total
cost of each sub-project; IRR of
the project under development;
expected prots from NPD
activities; contribution of each
sub-project to the NPD overall
protability
The ustomer perspective is
evaluated considering the number
and the frequency of meetings
undertaken with the purchasers
during the different phases of the
development project; the number

Company H

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini

R&D Management 39, 5, 2009

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

Company E

Company F

Company G

Company H

of complaints that are received


and processed; time-to-market,
satisfaction of early customers %
of budget dedicated to customer
requirements analysis and
assessment
The PMS was introduced since the
Critical aspects associated The PMS used in the development The PMS was introduced since the The PMS has been in place for
about 5 years. It was introduced establishment of the
with the use of the PMS
unit has been recently introduced foundation of Company F and
organisational unit devoted to the
when the size of the R&D unit
(3 years). Managers note that in associated with an MBO
grew at a level that coordination design and engineering of the new
rewarding system. The top
the rst year researchers ercely
helicopter. The rms commitment
and communication between
opposed the use of the system, that management was strongly
researchers working on different toward this project made it
was considered as a danger for the committed toward its use. They
thematic platforms became hard available a large amount of
believed indeed that, in a highly
autonomy of their activities.
human and nancial resources to
uncertain context, instruments to to achieve through informal
Therefore, it was initially
design and run a very complex
mechanisms
employed to monitor the progress stimulate scientists motivation
At the beginning, the introduction PMS. In particular, two people
of ordinary and repetitive tasks, and innovative behaviour are
were necessary to ensure an
of the PMS was resisted by
and over time expanded to assess critical for the rms
scientists and researchers, who felt interactive use of the measurement
competitiveness. Therefore,
also the performance of
system, sharing with engineers the
development and testing through dedicated human resources were too much constrained in their
targets costs and schedules,
hired and employed to design and activities by the measurement
the use of more sophisticated
collectively analysing results and
ease the acceptance of the PMS. instrument. Managers realised
measures of internal efciency
the opportunities for corrective
that the operation of the PMS
Nowadays, the PMS is considered No particular difculties were
actions
required that more human
as a useful instrument especially to encountered in the introduction
resources were devoted to dene The PMS was very well accepted
and use of the measurement
direct peoples attention toward
shared objectives, to analyse the since the beginning of its use, as it
the critical goals for the company, system, that was very well
was perceived as an important
accepted since the outset of its use obtained results and to discuss
although it is believed to be too
instrument for supporting
The PMS was heavily redesigned with researchers the most
much expensive to run
everyday work and
appropriate corrective actions
when Company B, in 2003,
communication. Managers are
This interactive approach made
decided to expand its activities
the PMS much more costly to run, very much satised with it, they
beyond Contract Research to
but it signicantly improved the believe it has signicantly
include also the research and
improved timeliness and cost
development for new drugs. This satisfaction of both researchers
efcacy and has stimulated a
and top managers with it
required that new performance
relevant process of organisational
dimensions were included to
learning
mirror different Critical Success
The PMS has not undergone
Factors in the area of internal
signicant changes in its structure
research
so far

Appendix B. (Contd.)

Performance measurement in R&D

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

R&D Management 39, 5, 2009

513

514

R&D Management 39, 5, 2009

Company M
Leading Italian producer of
thermoelectric power plants, it
operates on international markets
for customers ranging from Public
Administration to Independent
Power Producers and Industrial
Clients
It employs more than 2,000 people
and invests about 5% of its annual
turnover in development activities,
that are fundamentally aimed at
satisfying the specic
technological and business needs
of its existing and prospective
clients

Company N

New product development


Development activities are carried
out by a controlled company with
about 100 employees, working in
the development and
demonstration of innovative
products in areas such as
innovative dc distribution systems,
high-efciency gas recuperators,
components and systems for
nuclear fusion
A structured system for measuring Company N has been using a PMS
The PMS used in the rms
and controlling the performance for more than 20 years to control
research laboratories has been
its development activities. The
introduced with the main purpose of NPD activities has been
designed and is currently used. It main purpose for which this
of motivating scientists and
researchers and to improve their has the main purpose to monitor system has been introduced and
the progress of NPD activities and used is to maintain under close
commitment toward the
control the progress of
to continuously evaluate the
achievement of tangible results
that are very distant in time (the protability of the several projects development projects, that are
particularly costly and hence
that are contemporarily under
average development time for a
development, in order to timely should absolutely not run out of
new drug is between 10 and 15
introduce corrective actions and budget.
years)

Small rm with a total workforce


of about 150 employees, funded in
1996 and focused on the
development of new drugs based
on the properties of nitric oxide
(NO)
The rm has two subsidiaries, one
where the lead identication and
lead optimisation activities for the
new drug are carried out, the other
one dedicated to development and
commercialisation activities

Company L

Multinational rm that is a
worldwide leader in the household
electrical appliances and home
automation industry, with a total
workforce of about 60,000
employees
The rms competitive advantage
strongly depends on its product
development capability. It uses a
structured NPD process that is
carried out in a very decentralized
product development organisation
and allows to rapidly turn
customer latent needs into new
products and services
Company M invests more than
4% of its sales (that where about
11 billion h in 2007) in product
development
Basic and applied research
Basic and applied research
New product development
About 70 people (the largest part Thousands of people are
About 280 scientists and
of which own a PhD) work in the employed in the product
researchers are employed in the
development units dispersed in the
research laboratory of the
Oncology Division where Basic
business units and countries where
and applied research activities for subsidiary where the lead
Company M operates. The focus
identication and optimisation
new drugs are carried out
of the case study is the Italian unit,
activities are carried out
where more than 200 researchers
are employed

Biotech rm specialised in
oncology research and born in
2004 as a spin-off of a Pzers
laboratory. It employs about 700
people working in three divisions:
Oncology, Pharmaceutical
Development and Pre-clinical
Development
Basic and applied research is
undertaken in the Oncology
Division, which has a matrix
organisation with the following
departments: chemicals,
pharmacology, clinical
development, regulatory activities

Company I

Main objectives for R&D The head of the Oncology


performance measurement Division, who has been directly
involved in the design and
introduction of the PMS, states
that performance measurement is
necessary in Company I chiey to
motivate the rms researchers, to
support the adoption of an MBO
rewarding system and,
consequently, to give the
opportunity to link career paths

Type of R&D
Size of R&D unit

General characteristics

Appendix B. (Contd.)

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

Monitored dimensions of
performance

Appendix B. (Contd.)

The PMS fundamentally measures


the innovative capability of each
single researcher and the scientic
departments around which the
research division is organised (i.e.
medicine, chemistry,
pharmaceutics, biology,
intellectual property)
On average, 45 research projects
are contemporarily in place, which
last between 3 and 5 years. Each of
them employs about 10 people.
The rewarding system for the
members of the project teams and,
especially, for the project leaders,
is linked to the prociency with
which specic project activities are
carried out

In order to control the progress of


development activities, the
efciency with which the different
tasks of the NPD process are
undertaken is the critical
performance dimension
Moreover, nancial evaluation
of the project protability is
carried out
Managers acknowledge that an
internal-oriented perspective
implies the risk that product
development looses contact with
the needs of the industrial
customers. This is particularly
dangerous for Company N,
considered that its development
projects are activated typically in
response to specic requirements
of its customers that are the
ultimate recipients of the output of
the activities. Therefore, a
customer-related perspective in
performance measurement is also
used

Managers note that this on-going,


diagnostic, control system is a prerequisite for an appropriate
evaluation of projects
protability and for taking timely
discontinuation decisions

take adequate go/no-go decisions


It often happens that project
activities involve a very high
number of engineers, sometimes
working in different laboratories.
The PMS is useful also to gather
and disseminate project
information, keep people update
about projects advancement and
more deeply involve them in the
development activities of the
whole rm
The progress and advancement of
project activities is evaluated at
different organisational levels. At
the level of the development unit
responsible for project activities,
nancial dimensions are mainly
used. The performance of the
single development project is
evaluated taking into account the
efciency with which specic
processes are carried out and the
prociency with which it takes
into account the needs of extant
and prospective customers.
Finally, the performance of
the single engineer is
assessed considering its
efciency in undertaking
specic tasks
Managers acknowledge that
nancial and internal efciency
performance are useful for
favouring communication and
coordination of dispersed
engineers. However, in order to
involve them in the whole rms
development activities, a shared
denition of objectives and a
The PMS is linked with and
nurtures an incentive and
rewarding mechanism as well as a
dual-ladder career systems for
high-performing researchers
Overall, the ultimate goal of the
performance measurement system
is to improve the performance of
research activities

(in a dual-ladder scheme) with the


achieved performance
Support of decision making and
the analysis of projects progress
are second-order objectives,
considered the high-level of
uncertainty surrounding research
activities, that makes it unlikely
to use structured approaches to
support decision-making

The motivational purpose of the


PMS is achieved mainly through
the assessment of the quality and
innovativeness of the outputs (e.g.,
molecules or screening platforms)
delivered by people working in the
different departments (e.g.,
chemicals, pharmacology, clinical
development) as well as their
capability to contribute to develop
and grow the rms body of
knowledge
The introduction and use of the
MBO system requires however
that more tangible performance
are measured. Therefore, each
member of the project teams that
are activated in the matrix
organisation is also evaluated on
the basis of the efciency with
which specic process tasks and
activities (e.g., lead optimisation,
high-throughput screening) are
performed at both the individual
and project level

Company N

Company M

Company L

Company I

Performance measurement in R&D

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

R&D Management 39, 5, 2009

515

516

Company I

R&D Management 39, 5, 2009

Critical aspects associated Company I has introduced a PMS


with the use of the PMS
only very recently (about 2 years),
with the aim to support the
operation of an MBO rewarding
scheme. The rms researchers
were (and still are) very sceptical
about the advantages of this
instrument, and hence they are
ercely opposing its operations
Managers state that no tangible
results have been achieved yet

Indicators used to measure Innovative capability is measured


dimensions of
through the following main
performance
indicators: number of
publications; number of novel
identied molecules; % of the
novel identied molecules with the
required technical characteristics;
number of citations of the
researchers publications; peer
evaluation of the market
attractiveness of the novel
identied molecules
Business processes performance is
evaluated at the project team level
through the use of the following
key metrics: % of completed
project milestones; % of
collaboration objectives fully
satised; success rates for new
projects; % of projects delayed or
cancelled due to lack of funding

Appendix B. (Contd.)

The PMS was introduced about 4


years ago, after the managers had
realised a steady decline of
performance in lead identication
and lead optimisation. Scientists
and researchers ercely oppugned
the use of any type of measures
that was not linked with their
innovative outputs and
capabilities. Measures of
timeliness, respect of budgets, etc.,

The following main indicators are


used to assess the innovative
capability of individuals and
departments: hours dedicated to
staff training; international
relevance of technologies/
competencies acquired during 1
year; number of citations of the
publications of companys
researchers; % of the novel
identied molecules with the
required technical characteristics
The efciency with which specic
processes are carried out is instead
measured on the basis of: the % of
projects concluded on time; yes/no
evaluation of respect of budgets;
the rate of successful projects

Company L

Company N

Financial evaluation uses mainly


the following indicators: R&D
annual investments; Present Value
of R&D accomplishment/R&D
expenditures; Expected prots
from NPD projects; Cost/benet
ratio of completed development
projects
Business processes performance is
measured taking into account the
costs and resource consumption of
each development project (e.g.,
time spent on changes to original
product specication; % of project
milestones completed; % of
on time specications
delivery to purchasing and
manufacturing)
The customer perspective is
evaluated considering basically: %
of budget dedicated to customer
requirements analysis; number
and frequency of meetings
undertaken with clients during
the different phases of the
development project;
number of complaints that are
received and processed;
time-to-market
The PMS used in Company N is
The PMS has been in place in
not very much structured and
Company M for a long time to
sophisticated. The management
measure development activities
team decided to use a limited
performance. However, it has
number of performance indicators
received a particular impetus in
to reduce the risk that engineers
the last years, after the rms
opposed the introduction of the
competitive strategy has paid
system
stronger emphasis on customercentricity and time-to-market. As It has been subject to a redesign in
the last years, where a number of
a result, the PMS has been
indicators related to customer
redesigned to include customer-

continuous feedback
on their behaviour is a
critical factor.
Financial performance are
measured on the basis of the
following key metrics: R&D
annual spending, total cost of each
R&D project, IRR or NPV for
development projects
Internal efciency of each
development project and
individual is basically measured
taking into account: % of projects
respecting costs and budget;
number of agreed milestones/
objectives met; % of R&D
expenditures that lead to
new or enhanced products or
processes; % of project
milestones completed
The customer perspective is
evaluated considering as main
indicators: the number of
complaints that are received and
processed; time-to-market;
engineering hours on projects/
engineering hours on problems
and troubleshooting; % of
budget dedicated to customer
requirements analysis
and assessment

Company M

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

The rm is an Italian group operating in the


energy conversion and battery technologies, with
a focus on automotive applications and
components
It operates, directly of through sales agents, in
more than 60 countries with a workforce of
about 2,600 employees, and it realises more than
60% of its turnover abroad. The critical success
factor for the rms competitiveness is
represented by technical excellence, time-tomarket and after-sale services
New product development
NPD activities are carried out in a department
with a matrix structure where about 150
engineers are employed. NPD projects are
Italian company founded in 1935 operating in
the pharmaceutical industry, with particular
emphasis on respiratory and cardiovascular
diseases
It employs a workforce of about 3,000 people
and has three research centres where about 300
scientists are employed
In 2007, the rm invested more than 14% of its
annual turnover in R&D
Basic and applied research
About 300 people work in the three research
centres where basic and applied research
activities for new drugs is carried out

Company Q

Company P

satisfaction have been added. This


was suggested by the intensied
competition also in a very
consolidated industry such as
power generation technologies,
that required Company N to more
closely monitor and take into
account customer satisfaction
Managers complain about the
costs that running a PMS require,
and are thinking about further
reducing its complexity and
reporting practices. However, they
believe that the use of the
measurement system has had a
positive impact on the cost
reduction and the respect of
budgets for NPD projects,
especially those involving radical
technologies

related performance dimensions


and more nancial and human
resources have been dedicated to
its operation
No particular difculties in the
introduction and use of the PMS
emerged, and the management
team believes that the system has
had a fundamental role in the
implementation of the new
competitive strategy and, in
particular, to improve time-tomarket for new products and
timeliness of development
activities

were perceived as absolutely too


much constraining. Some
researchers have even left the rm
as a result of the use of the PMS
Over time, the number of
measures related to internal
efciency has been reduced, and
more attention has been devoted
to illustrate the meaning of the
PMS and its real usefulness.
Managers believe that the use of
the PMS has signicantly helped
focus researchers efforts toward
the innovative goals that are
critical for the rms
competitiveness. The
measurement of internal efciency
is acknowledged as more
problematic, and especially very
much expensive to operate

through the use of the PMS, and


they are even considering the
opportunity to abandon it. They
realize that, in order to properly
operate it, a complete redesign of
its underlying procedures and
indicators would be necessary, and
especially a larger availability of
resources should be ensured to
more deeply involve managers
into the decisions of scientists and
engineers.

Company O

Company N

Company M

Company L

Company I

General characteristics The rm is one of the world leading companies


in the diagnostic imaging industry, offering
products for the three main market segments,
i.e. Magnetic resonance, X-rays and nuclear
medicine
It employs about 1,000 people and has three
main R&D centres, where applied research for
new products and development activities are
carried out. The rm invests about 11% of its
annual turnover in R&D and it holds
approximately 1,500 patents worldwide
Type of R&D
Basic and applied research
Size of R&D unit
More than 100 people are employed in the
R&D centre devoted to basic and applied

Appendix B. (Contd.)

Performance measurement in R&D

R&D Management 39, 5, 2009

517

518

R&D Management 39, 5, 2009

research in the eld of X-rays and electronic


resonance
The rm uses a structured system to measure
research projects performance that has been
introduced with the main purpose to monitor
the progress of activities and support the go/
no-go decisions taken the gates of the stagegate process. Project leaders are evaluated on
the basis of the nancial performance of the
project they are responsible for
At the project portfolio level, the PMS is used
to compare the protability of different
projects and therefore to select the most
promising areas for future investments
Performance measurement is used to improve
the overall rms R&D performance

Company O

Financial performance is measured mainly


Indicators used to
measure dimensions of through the NPV and IRR of the project; the
% of annual sales due to new products and
performance
technologies, a subjective evaluation of the

Monitored dimensions The PMS used by Company O is very simple


of performance
and it is fundamentally based on the use of
nancial performance dimensions. Managers
believe that go/non-go decisions and project
prioritization should be based on an accurate,
quantitative assessment of each projects
impact on the rms protability. This is also a
pre-requisite for an objective ex post evaluation
of R&D projects and for ensuring
accountability of managers and team leaders
However, especially at the earlier stages of the
process, nancial evaluations are accompanied
by a qualitative analysis of the projects
capability to target the needs of a promising
and relevant market segment

Main objectives for


R&D performance
measurement

Appendix B. (Contd.)

Managers believe that the PMS should serve the


main purpose of favouring coordination and
communication among the researchers with
different competencies and roles taking part to
the R&D projects. It is believed that this use of
the PMS if critical for stimulating learning about
past projects and activities across different
scientic departments and, ultimately, for
reducing uncertainty surrounding decisions
about prioritization of projects
Motivating researchers and maintaining their
commitment toward the rms strategic
objectives is acknowledged to be a critical
purpose for performance measurement,
although very difcult to achieve because of the
counterproductive effects the PMS might have
on the creativity of scientists.
The most critical performance dimension around
which the PMS is built is innovativeness, and in
particular the capability of individual
researchers and departments to nd an
innovative solution to the technical problems
that might emerge during research activities and
to add to the rms innovative potential,
Performance in undertaking specic business
processes and activities (e.g., acquiring new
technologies and competencies, scouting
licensing and partnering opportunities,
collaborating with external organisations) is also
evaluated, although it serves more the purpose to
ensure coordination and information sharing
among researchers rather than
The rm also takes into account the nancial
performance of R&D projects
Innovative capability is measured taking into
account basically: number of registrations for
new chemical entities made per year; hours
dedicated to training activities; peer evaluation

Company P

Efciency of internal business processes is


evaluated through the following main indicators:
Program Status Review, which is a modied
version of the Earned Value metric which allows

managed according to a stage-gate system and


consistently with the Six Sigma approach
The rm has introduced and used a PMS for its
development activities after the standard ISO
TS/16949 for the automotive industry was
released in 1999. Conforming to this voluntary
standard required in fact that a number of
control activities were established also for the
NPD process. As a result, a PMS was designed
and used with the main purpose to monitor the
progress of activities, document their
advancements and hence conform to the
standard requirements
However, managers took the opportunity to put
in place a more complete performance
measurement system that allows to rapidly
introduce corrective actions and prioritize
projects through the evaluation of their
protability
Control of activities is achieved mainly through
an in-progress monitoring of the prociency with
which specic tasks and activities dened by the
stage-gate NPD process are carried out
(performance of internal business processes)
A nancial evaluation has been introduced to
allow for a prioritization and evaluation of
projects, especially those involving incremental
technologies

Company Q

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

for a synthetic evaluation of the project


completion; % of projects respecting costs and
budget schedules; quality of documentation
delivered to manufacturing; rate of re-use of
standard designs or proven technologies;
number or % of certied processes; % of
projects that lead to new or enhanced products
or processes
Financial evaluation is undertaken through the
assessment of: annual spending in development
activities; costs required to acquire or internally
develop a new technology; % of cost reduction
allowed by new projects

of the international relevance of the


technologies/competencies acquired during 1
year; average product life-cycle length
The level of prociency with which internal
processes are carried out is assessed on the basis
of: the number of critical observations received
after an external control; number of questions
made by the regulatory authority after the
submission of the registration package; % of
projects respecting costs and budget schedules;
average annual improvement in process
parameter (e.g., lead time and technical quality)
The only nancial indicators used by Company
P is the ROI associated to each R&D project
The introduction of the PMS has been
particularly critical in Company P, with delays in
the use of the selected measures because of the
opposition of researchers and chemists. In order
to ease the acceptance of the system, the rm has
simplied its structure, reduced the number of
indicators and the measurement frequency,
dedicated much more attention to more deeply
involve scientists and researchers in the
denition of the performance targets and in the
analysis of the results, and organised dedicated
training sessions to explain that the PMS could
be an important empowerment mechanism,
through which researchers become more directly
involved in the rms decision making processes
Managers are nowadays satised with the PMS
because it favours a pronounced crossfertilisation of competencies belonging to
traditionally distinct disciplines, which is a
critical success factor for pharmaceutical
research

market share from new products resulting from


research projects
Customer perspective is assessed using the
following indicators: time-to-market; % of
customer-driven projects; % of budget
dedicated to customer analysis and verication

The decision to introduce a PMS in the rms


development unit was determined by the
voluntary compliance to the ISO TS/16949
standard for automotive components suppliers.
As a result, the set of data that should be
collected and presented was in large part dened
by the norm
Managers decided however to take advantage of
this opportunity to devise a more complete
system that allowed to appropriately support
decision making and project prioritization
through nancial indicators
No particular difculties were encountered in the
introduction and use of the PMS, that was
positively accepted by engineers. Managers are
satised with the PMS, which is believed to
improve the timeliness of project prioritization
and the accurateness of decision making

Company Q

Company P

Company O

The rm has been using a PMS for more than 2


Critical aspects
associated with the use years. At the beginning, the measurement
instrument received a strong opposition from
of the PMS
researchers, who feared their autonomy could
be hindered. Therefore, managers decided to
simplify the structure of the PMS, reducing the
number of indicators and, in particular,
eliminating those related to the efciency of
internal business activities. This improved the
acceptance of the PMS, and nowadays
managers are satised with it, because in their
opinion it improves the outcome of investment
decisions without constraining the creativity of
individuals and professionals. Moreover, the
costs associated with running a simplied
system are not particularly high

Appendix B. (Contd.)

Performance measurement in R&D

r 2009 The Authors


Journal compilation r 2009 Blackwell Publishing Ltd

R&D Management 39, 5, 2009

519