March 2010

Towards a conceptual framework for measuring public sector innovation

Module1 Conceptual Framework

Carter Bloch

The paper is part of the output from the joint Nordic research project Measuring innovation in the public sector in the Nordic countries: Toward a common statistical approach (Copenhagen Manual). This project is supported by the Danish Agency for Science, Technology and Innovation, the Nordic Innovation Centre, Innovation Norway, the Research Council of Norway, VINNOVA, the Swedish Association of Local Authorities and Regions, and the Finnish Ministry of Employment and the Economy.

viden skaber muligheder

Table of contents
1 2 INTRODUCTION 3 INNOVATION INDICATORS: REVIEWING THE MAIN ISSUES AND EXISTING WORK 5 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 Recent studies of public sector innovation Innovation: elements of the definition and types of innovations Inputs to innovation Innovation culture and capability Dissemination of innovations and learning Types of interaction Procurement for innovation Innovation drivers Barriers to innovation 5 8 13 15 17 17 18 18 19 20 22 22 25 25 27 32 35 37 38 39 41 42 43 44 46

2.10 Outputs and objectives of innovations 2.11 Summing up 3 4 A MODEL FOR THE MEASUREMENT OF PUBLIC SECTOR INNOVATION INDICATORS AND DEFINITIONS A PRELIMINARY FRAMEWORK 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 5 Defining innovation Individual types of innovations Innovation activities Linkages Organising innovation: strategy, management and competences Measuring the role of ICT in innovation in public sector organisations Measuring innovation performance Barriers to innovation

DISCUSSION AND FUTURE DIRECTIONS 5.1 Comparison with the Oslo Manual

5.2 Heterogeneity of the public sector implications for innovation measurement REFERENCES

Page 2

Introduction

This paper examines key issues in the development of a conceptual framework for measuring public sector innovation. In doing so, we develop a measurement framework for innovation in the public sector, propose definitions for key concepts and outline sets of indicators that are important for measuring public sector innovation. The paper presents and discusses the preliminary framework that has been tested with respondents in all five of the Nordic countries. The framework will subsequently be revised and further tested through a large scale pilot study during 2010. The paper is part of the output from the joint Nordic research project Measuring innovation in the public sector in the Nordic countries: Toward a common statistical approach in which a questionnaire will be developed to collect data on innovation in the public sector. The objective of the project is to develop a measurement framework for collecting internationally comparable data on innovation in the public sector, which will contribute to our understanding what public sector innovation is and how public sector organisations innovate and will develop metrics for use in promoting public sector innovation. The work of the first stage of this project is documented through, in all, six papers1: Innovation and heterogeneity in the public sector (Markus M. Bugge, Johan Hauknes, Stig Sliperster) Towards a conceptual framework for measuring public sector innovation (Carter Bloch) Survey methodology for measuring public innovation (Peter S. Mortensen) Mapping user needs (Lydia L. Jrgensen) Feasibility study of public sector organizations (Per Annerstedt and Roger Bjrkbacka) Nordic survey on public innovation 2009 draft pilot questionnaire

The development of a measurement framework for innovation in the public sector has been based on a user-driven approach. Development work has drawn on four main sources: existing theoretical and empirical studies, studies of users and respondents, and guidelines for measuring innovation in the business sector (the Oslo Manual, OECD/Eurostat 2005).

In addition to this, some countries have published national results of their feasibility studies. See www.mepin.eu.

Page 3

Review of existing studies Respondent interviews and testing

User needs

Oslo Manual

Bugge et al. (2010) discusses public sector innovation and its measurement from a theoretical perspective. The empirical approach of the Nordic project has been to engage in dialogue both with key stakeholders from policymaking organisations and interest groups, and with potential respondents. These studies were conducted in two stages. For stakeholders, initial exploratory meetings were held in each country to discuss the conceptualisation of public sector innovation and user needs. In the second stage, drafts of the questionnaire were discussed both in meetings and through written consultations. The first stage of the respondent studies was also conducted using an open approach. Respondents from different sectors and levels of government participated in focus groups to discuss their perceptions of innovation, key elements in the innovation process, and what types of data would be feasible to collect or measure. The second stage of the respondent studies involved cognitive testing in all five Nordic countries. A wide range of public sector organisations were asked to fill in the draft questionnaire and thereafter participate in an interview on their perception of the questions asked. While it was viewed as important to have an open approach to examine public sector innovation, the Oslo Manual and the Community Innovation Survey were also used as reference points for much of this work. This was done in particular for two reasons. First, the Oslo Manual builds on years of experience in measuring innovation in the business sector. Second, there is interest among many stakeholders in achieving as much comparability as possible with innovation data in the business sector. There are, however, questions to be discussed regarding the relevance and possible outcomes of such comparisons. Hence, comparability or compatibility with Oslo Manual will be an important consideration for indicator development, and will be discussed over the course of this paper. Section 2 examines key areas of the measurement framework, discussing relevant issues and reviewing existing work. Based on this and work in Bugge et al (2010), section 3 builds a general model of innovation in the public sector, taking account of key aspects of public sector innovation. The main objective of this model is to introduce key areas for measurement and their role in an overall framework that seeks to capture the entire innovation process in public sector organisations. The final section provides initial proposals

Page 4

for elements of the measurement framework, including preliminary definitions of key concepts and indicators.

Innovation indicators: reviewing the main issues and existing work

2.1 Recent studies of public sector innovation

This section describes recent survey-based studies of public sector innovation2. We provide only a general description in this section, and include discussion of specific aspects of the studies in the following sections. To our knowledge, only a select few surveys of public sector innovation have been conducted, and many of these have been modelled as innovation prize competitions as opposed to sample surveys. An early attempt at measuring public sector innovation was undertaken by Statistics Canada in 2000 (Earl, 2002). The study, which was part of a Survey of Electronic Commerce and Technology 2000, inquired about the introduction of organisational and technological changes in both public sector organisations and private businesses. They found that very high shares of public sector organisations had implemented these changes, even among the smallest administrations. As part of efforts to promote public sector innovation, the Korean Government developed the Government Innovation Index (GII)3. The GII, which was implemented over 2005 and 2006, seeks to: Measure government innovation efforts, develop a tool for autonomous diagnosis and improvement of innovation capabilities at each institution, and to benchmark government institutions. The GII is a web-based tool, where organizations can enter their own innovation

This paper will also draw on other studies of public sector innovation, particularly work from the Publin project (Koch and Hauknes, 2005). See also the recent Inno Grips Mini Study of public sector innovation, Thenint and Basset (2010) This short description of the Korean GII is based on Powerpoint presentations by the Korean Ministry of Government Administration and Home Affairs (2005) and Yoon (2006).

Page 5

data, and then the tool conducts a series of diagnostic analyses based on the organizations data and a previously collected set of data that is used as a comparison standard. The GII consists of a survey covering four areas of organisational capability (Innovation leadership, Vision and Strategy, Personnel capability and Systemisation of management), the degree of adoption and implementation of pre-specified management systems and other programs, and barriers to innovation. These data both generate indicators directly, and are used to construct a series of diagnostic indicators (or indices) to assess innovative capability in individual public sector organisations. The GII includes diagnostic measures that are used to analyse and benchmark public sector organisations: Foundation readiness for innovation, Consistency of innovation activities, Internalisation of innovation, Barriers to innovation, Innovation Activation index and Performance Enhancement level. The National Audit Office has studied innovation in central government and results have been published in the reports Achieving innovation in central government organizations (2006) and Innovation across Central Government (2009). Both reports are based on a survey of innovation in central government departments, executive agencies and nondepartmental public bodies, though they differ in terms of focus and content. The 2006 survey asks for nominations of innovations and each questionnaire is focused on a single innovation. The survey in particular deals with the driving forces behind the innovation, both internal and external, potential positive and negative impacts of the innovation, and barriers to achieving innovation. The object of the 2009 survey in contrast is the public sector organisation as a whole. The survey inquires about organisations own conceptualisation of innovation, and on how the organisation innovates, including culture and capabilities, risk management and the role of selected barriers and incentives. Borins (2006) discusses innovation surveys conducted in the US, Canada and other Commonwealth countries. These surveys are all based on innovation award programs4 in which nominees to the innovation award are required to fill out an innovation survey questionnaire. These questionnaires were generally open-ended and focused on the individual innovation being nominated for the award. The Audit Commission has also conducted a survey of innovation among Local Authorities in England (Audit Commission, 2007). The study covers attitudes to innovation, the role of organisational structure and staff, barriers and enabling conditions, and learning activities. It also highlights a number of specific examples of innovations in local government.

US: Ford-KSG awards; Canada: IPAC awards; and countries of the Commonwealth: CAPAM awards.

Page 6

NESTA commissioned two studies in 2008 (Dunleavy et al., 2008; Clark et al., 2008) to review literature on public sector innovation and develop initial proposals for the Public Sector Innovation Index. Dunleavy et al. (2008) proposes a range of aggregate indicators across the dimensions of R&D activities, Consultancy and strategic alliances, Intangible assets, ICT infrastructure, Human resources, Institutional performance, E-Government, Origins of Innovations, Innovation outputs, impacts and scope. A cornerstone of the proposal in Clark et al. (2008) is to conduct an innovation survey along the lines of that done for the business sector, modified to capture the particularities of public sector innovation. The UK Department of Health and NHS have initiated a project (Ayling et al., 2009) to measure and value innovation in the NHS. The project seeks to develop indicators of innovation at the three stages of the innovation process (ideas, growth, diffusion) and indicators of innovation culture. Data on these indicators will be collected from health institutions and benchmarked against targets for the generation and subsequent development of new ideas, and their adoption and dissemination. OECD Education has initiated a project (OECD, 2009) on the measurement of innovation in education. The group has established a forum for investigating relevant work in this area and discussing key questions on how to measure innovation in education and benefits and limitations of different approaches. The UK Centre for the Measurement of Government Activities (UKCeMGA), under the ONS, was established to implement the recommendations of the Atkinson Review, an independent review of the future development of government output and productivity (Atkinson, 2005). Both the Review and subsequent work by UKCeMGA reflect an increased governmental and public interest in measures of public sector performance. In addition to a general framework and principles, the intention is to focus on practical solutions for measuring the key functional areas of health, education, public order and safety and social protection. This work has made considerable progress in developing aggregate measures of public sector output within these key sectors. Though, at the same time this work reveals the complexity of measuring economic and social outcomes of public services. The OECD is currently working on a project on a set of indicators (Government at a Glance) to measure six stages of government activities (revenues, inputs, public sector processes, outputs, outcomes, antecedents) (OECD, 2007). The working and technical papers of this project (see reference list below) provide a broad overview of the use of quantitative indicators, along with some key issues. In general, the work by Government at a Glance does not deal with public sector innovation explicitly, though knowledge of output and other measures of government activity is very relevant for innovation measurement.

Page 7

2.2 Innovation: elements of the definition and types of innovations

An important issue for the conceptual framework is how to define innovation in the public sector. This includes examining the main elements that characterise an innovation, types of innovations, distinguishing between different types, and innovation novelty. There are a number of criteria that are used to define innovations in general, such as implementation, success, impacts, and degree of change. Added to this is the question of breadth of the definition in terms of what types of changes are included.

2.2.1 Examples of definitions of innovation

Innovation in the public sector has been defined in a number of ways in existing work. Before discussing some key elements in defining innovation, this subsection reviews examples of innovation found in the literature and studies of public sector innovation. A number of typologies of public sector innovations have also been proposed. These will be reviewed below. The Oslo Manual definition of Overall innovation: The latest (third) edition of the Oslo Manual (OECD/Eurostat, 2005) defines innovation as the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new organisational method in business practices, workplace organisation or external relations. Here an innovation is defined as being one of four types of innovations - product, process, marketing or organizational though without any additional description of what an innovation is overall. This essentially means that the overall definition is fully dependent on the definitions of the individual types of innovations to characterize what an innovation is. The Publin project (see eg. Koch and Hauknes, 2005) proposes the following definition of innovation: Innovation is a social entitys implementation and performance of a new specific form or repertoire of social action that is implemented deliberately by the entity in the context of the objectives and functionalities of the entitys activities. Innovations are thus new, implemented, and intentional, and can refer to any type of change in the organization. Audit Commission (2007) conducted an innovation survey of local government authorities in the UK. They defined innovation as: practices undertaken by organisations in order to improve the product or service they provide, characterised by: Change step-change and impact Novelty new to the organisation in question Action completed, not just an idea

They then ask To what extent, if at all, is innovation taking place within your organisation?: A great deal / Some / Hardly at all / Not at all / Dont know Hence, while the criteria are similar to the Oslo Manual in terms of novelty and implementation, they define innovation in

Page 8

terms of practices instead of the actual change/output, and thereby do not identify any specific types of innovations. NAO (2006a) surveys central government organisations in the UK. The survey invites nominations of up to 3 specific innovations. The survey definition of innovation: Innovation is having new ideas, developing the best ones, and implementing them in such a way that there is (at least) a good chance that they will improve the methods in which your organisation operates and/or performs. New ideas without some degree of implementation are not enough. The definition is thus very open (in a broad and accessible way), and respondents are simply asked to describe their nominated innovation (no types are defined). The survey thus uses the object approach of focusing on single innovations (in line with early business innovation surveys, but in contrast to the Oslo Manual). The authors find (contrary to prior expectations) that no policy innovations (not explicitly defined) were nominated. The authors discuss this, and also interviewed respondents on these results, but there does not appear to be any single clear reason for this. All 126 examples of innovations are listed in NAO (2006a), which provides useful background in examining what organisations themselves consider to be innovations (though only for central government). NAO has more recently conducted a new survey on Innovation Across Central Government (NAO, 2009). Here, the survey does not explicitly define innovation, and examples of innovation are asked for at the end of the questionnaire. The text used is: Please fill the following table with details of 5 projects or programmes, which are currently under way, and which you consider to be your best examples of successful innovation. Please consider customer facing innovations, including changes to the ways your organisation communicates with customers or delivers services, as well as internal process innovations, such as the procurement of goods or services or the development of more efficient processes. However, while no definition is given, respondents are asked to answer questions on how they conceptualize innovations themselves, in particular whether they agree with the following: Innovation is adopting successful practices from other organizations Innovation is being the first to do something Innovation and creativity are the same thing Innovation is problem solving Not every organization can be innovative Not every organization should be innovative For something to be innovative, it must be successful Innovation must add value

From the UK National Health Service (NHS) project on Measuring & Valuing Innovation in the NHS (Ayling et al, 2009) innovation is defined as The adoption of new-to-theorganisation or new-to-the-NHS technology products and/or service delivery processes, comprising incremental or disruptive change, and resulting in a significant improvement in patient outcomes, experiences, safety and potentially cost effectiveness. H.M Moore et al. (1997) defines innovation in terms of novelty and the degree of change in relation to the organisation: Changes worth recognising as innovation should benew to the organisation, be large enough and durable enough to appreciably affect the operations or

Page 9

character of the organisation. This is a very general definition, but it puts particular focus on criteria for when a change is significant enough to be considered an innovation; this is not in terms of novelty compared to other organizations, but changes need to be significant in terms of the organisations overall operations. According to Mulgan (2007): The simplest definition is that public sector innovation is about new ideas that work at creating public value. The ideas have to be at least in part new (rather than improvements); they have to be taken up (rather than just being good ideas); and they have to be useful. This definition thus requires that innovations should be new, implemented and have a positive impact on public value creation.

2.2.2 Key elements in defining innovation

Implementation. An important characteristic of innovations as defined in Oslo Manual and elsewhere is that they must be implemented in some way. This requirement is also made in essentially all the definitions examined here. Good ideas, even if they are developed, are not enough; they must have also been taken into use. Success. Essentially all definitions of innovation require implementation; that an innovation is not just an idea but also has been taken into use. However, many conceptualisations of innovation go one step further, and argue that innovations are not just implemented changes, but also have had a positive impact. Mulgan (2007) uses the generic impact creating public value, which implies value creation but does not necessarily specify for whom. There are a number of difficulties with defining innovations in terms of their success or positive impact. The first is that there may be substantial ambiguity with relation to an innovations success; by some criteria (or for some stakeholder) the innovation may be a success, but may have had a negative impact for others. This is also a point where the public sector arguably differs greatly from the business sector, due to the fact that public sector organisations may have a much wider range of objectives (see Bugge et al, 2010). A second problem with this is that the impact of innovations may not be known until some time after the innovation. Furthermore, one can also question whether innovations should be defined as successes, as opposed to focusing on the actual change itself. It can also be noted that in Oslo Manual for business sector innovation explicitly states that innovations need not be a success. Degree of change required: Implementation concerns whether a change has been taken into use, and success refers to the impact of changes. However, a key element of innovations is the actual changes to the characteristics of the object itself that is being innovated. The question is how large do changes need to be in order to be considered an innovation, and how can they be measured. A fairly common (minimum) requirement is that innovations should be distinguishable from minor changes that are part of daily operations, or take place on a continual basis. For example, the Oslo Manual states: The minimum requirement for an innovation is that the product, process, marketing method or organisational method must be new (or significantly improved) to the firm. Note that this criteria places great importance on the word significant and how it is interpreted; i.e. significant compared to what? Is a major change to a very small part of the organization enough? Or should the change also be large in relation to the organizations overall operations?

Page 10

Source of the innovation: An additional element that can be taken into consideration is the source of the innovation and whether this should be used to characterise innovations. The main question here is whether politically mandated changes which are not a result of decisions made by the organisation itself and potentially can be politically changed again before they can have any impact should be considered as innovations.

2.2.3 Types of innovations

What types of innovations should be considered innovations in the public sector? A number of the definitions above suggest that any type of change can be considered, as long as it constitutes a major change to the organizations activities. This subsection looks at classifications of types of innovations in the public sector context, drawing on the classifications used in the Oslo Manual. The definitions of types of innovations in the Oslo Manual are: Product innovation: the introduction of a good or service that is new or significantly improved with respect to its characteristics or intended uses. This includes significant improvements in technical specifications, components and materials, incorporated software, user friendliness or other functional characteristics. Process innovation: the implementation of a new or significantly improved production or delivery method. This includes significant changes in techniques, equipment and/or software. Organisational innovation: the implementation of a new organisational method in the firms business practices, workplace organisation or external relations. Marketing innovation: the implementation of a new marketing method involving significant changes in product design or packaging, product placement, product promotion or pricing. How well do these definitions relate to public services and their operations? We will examine this in detail in section 4. Windrum (2008) proposes the following typology of innovations, which draws on Koch and Hauknes (2005): Service innovation Service delivery innovation Administrative and organizational innovation Conceptual innovation Policy innovation Systemic innovation Service innovations are the introduction of new service products or improvements of existing service products. Service delivery innovation is new ways of delivering services to and interacting with the users. Administrative and organizational innovation involves introducing

Page 11

new organizational principles for the production and delivery of service products. These first three have formerly been covered by studies of innovation in the private sector, however, according to Windrum (2008) the last three types of innovations; i.e. conceptual innovation, policy innovation and systemic innovation, have not received that much attention in studies of innovation in the private sector. Conceptual innovation is the development of new world views that challenge assumptions that underpin existing service products, processes and organizational forms. (Windrum, 2008) In terms of an individual organization, this would refer to a change in overall strategy or the organisations social objectives. At least at the individual level, conceptual innovations can be related to strategic or business model innovations for firms. Overall strategies are often stressed as central to successful innovation. However, the drawback in terms of measurement is that a new strategy does not necessarily imply an implementation of changes. Policy innovations are new policy concepts, and can either be due to policy changes, or changes in policies, new programmes, large reforms, etc.. NAO (2006) also discusses policy innovations, as do other studies, though without providing an actual definition of what they are. What is notable here (and was pointed out by the authors of the NAO 2006 study) is that none of the 126 examples of innovations provided by respondents in the NAO 2006 study were policy innovations. Furthermore, these would mainly be relevant for a smaller group of public sector organizations. Systemic innovations involve new or improved ways of interacting with other organizations and knowledge bases. These are thus somewhat related to organizational changes (and essentially are included in the definition of organizational innovation used in the Oslo Manual). The Audit Commission (2007) places examples of public sector innovations into the following categories: Service design or delivery innovation providing a new service to users, or delivering existing services in a new way Process or managerial innovation changing the processes, managerial structure or organisational structure of an authoritys back office or service delivery functions Democratic innovation implementing new practices in the pursuit of renewed democratic engagement with citizens Strategic innovation re-positioning the authority in line with new corporate objectives or new customers, including using alternative service delivery models.

The first two types are fairly standard, while strategic innovations resemble conceptual innovations described above. Democratic innovation involves practices to improve engagement with citizens, for example allowing for greater citizen participation or increasing transparency of public sector activities. To a large degree, this type appears to be characterized by a specific objective of innovations, and in terms of the actual changes could consist of new services, processes or organizational practices. Technological innovation. Statistics Canada has surveyed public sector institutions in the Survey of Electronic Commerce and Technology 2000, and asked on two types of innovations (or changes), organisational and technological: (Organisational change) During the last three years, 1998 to 2000, did your organisation introduce significantly improved organisational structures or implement improved management techniques? The following

Page 12

two questions determined if firms were involved in technological and, if so, how were they involved: During the last three years, 1998 to 2000, did your organisation introduce significantly improved technologies? If yes, how did you introduce significantly improved technologies? (Check all that apply) by purchasing off-the-shelf technologies? by licensing new technologies? by customising or significantly modifying existing technologies? by developing new technologies? (either alone or in conjunction with others). The focus on the introduction of new technologies follows the overall scope of the Survey of Electronic Commerce and Technology, though in general this reflects a narrow coverage of, for example, product and process innovations in public sector organisations. User-driven innovation and Employee-driven innovation. User driven innovation involves using advanced, systematic methods to examine, uncover user/customer needs. Employee driven innovation seeks to increase the individual employees involvement in firm innovation activities, gearing management and organizational structure to encourage and capitalize on employee-driven innovation: (through) norms/culture, work teams, devoting resources (worker time) to innovation, included as part of overall innovation strategy, systematic process, contacts and information flows. Both these types of innovations are in policy focus. However, both these types are identifying by activities or driving forces, and not by the actual characteristics of the innovations. Hence, these types are less relevant in determining types of innovations, but are more relevant in motivating the measurement of driving forces of innovation. Social Innovation. Social innovation is another concept that can be found often in discussions of public sector innovation. From Wikipedia: Social innovation refers to new strategies, concepts, ideas and organisations that meet social needs of all kinds working conditions and education to community development and health and that extend and strengthen civil society. By this definition, social definition is not defined by the characteristics of its changes, but more in terms of its objectives or intended outcomes. It thus goes across types of innovations, and thus can be argued to point towards gathering information on the importance of societal goals in organizations innovation.

2.3 Inputs to innovation

2.3.1 Innovation activities
There will in general be great interest in obtaining quantitative measures of innovation for public sector. This includes both input and output measures. However, these are very challenging for the business sector and potentially even more so for the public sector. With the exception of R&D surveys for universities and government research institutes, there is very little experience with the measurement of innovation expenditures in public sector organizations. For example, none of the surveys described above have collected detailed information on innovation expenditures. And, experience in collecting data on R&D expenditures from public research institutions cannot be carried over to other public sector institutions, given the great differences in types of activities.

Page 13

Dunleavy et al (2008) proposes a series of indicators of public organisations R&D/innovation activities, similar to that in the Oslo Manual/CIS; e.g. R&D unit, strategy unit, expenditure on market research, innovation capital costs, in-house R&D, skills training, consulting expenditures (either measured on binary or ordinal scale).

2.3.2 Human resources and skills

Skills and human resources are widely recognized as central to innovation in all sectors. However, the area of HR has received very little attention in business innovation surveys such as CIS. Given the nature of public services, which are generally people-oriented, skills are arguably of even greater importance to public sector innovation than for businesses. The MEADOW project (Measuring the Dynamics of Organisations and Work) examines the measurement of human resource management for both businesses and public sector organisations5. They identify in particular three areas: Investments in skills enhancement; Reward systems for individuals or groups; and dialogue between employees and managers. These aspects thus both refer to training activities and organisational aspects (which are covered below). This has in some cases been supplemented by information on employee characteristics, for example education level or involvement in innovation or research. Dunleavy et al (2008) propose the following indicators on HR: percentage of staff with a graduate education, percentage staff with post-graduate education, employee satisfaction, is innovation an employee performance goal?, training, promotion related to high performance?

2.3.3 The role of ICT in public sector innovation

There is a substantial amount of policy focus on E-government and other forms of ICT-based innovation for the public sector. This motivates taking a careful look at how the role of ICT could be measured in public innovation surveys. ICT is also important for the business sector, though the topic receives little attention in business innovation surveys. A main argument for this is that given space constraints and that ICT usage surveys are already conducted (for businesses, but also in some cases for public sector institutions), analysis of the role of ICT in innovation can be conducted using matched data. However, recent attempts to do this at the OECD level have shown that this is more difficult than first thought.

Meadow Consortium (2010). See http://www.meadow-project.eu/.

Page 14

A major challenge here would be developing indicators that are truly useful. This will likely require a combined effort of reviewing current experience, consultation with key users, and discussions under the feasibility studies. Dunleavy et al (2008) proposes a set of ICT indicators: Share ICT expenditures, share ICT exp spent on website, avg age of ICT equipment, replacement time for PCs, development and mgmt of ICT services done in-house?, intranet system?. And: percentage services that can be requested online, and percentage services that can be delivered online. However, these indicators are fairly standard, raising the question whether they are really able to capture the degree of ICT-based innovation. Moon (2002) (see Bertok et al 2006) identifies 5 stages of e-government development, which may (indirectly) be helpful in trying to capture different levels of ICT development and implementation: One-way communication (billboard stage) Two-way communication (request and respond) Transaction: service delivery and financial transaction (e.g. tax online) Vertical and horizontal integration: intergovernmental and intra-governmental integration (e.g. data sharing) Political participation (e.g. voting online)

While helpful, this typology includes a mix of ICT-based activities that may potentially affect different types of innovations. For example, we are likely interested in the role of ICT for service innovations, organisational innovations, and also ICT that raise the general technological level of the organisation (and might thus impact process innovations). The European Commission collects data on the supply of online public services (see Wauters et al, 2007), in particular indicators on the availability of public services online, the level of online sophistication, and user centricity. Somewhat related to the list above, five levels of sophistication are identified: Information; One-way interaction (downloadable forms); Two-way interaction (electronic forms); Transaction (fully electronic case handling); and Personalisation (pro-active, automated).

2.4 Innovation culture and capability

How do public sector organizations innovate? How do they structure, organize and promote innovation? These questions are central to the innovation process. We have some information from case studies, but in general we have insufficient knowledge about how public sector units organise and promote innovation activities, and which elements or factors appear most often to produce successful results. Arguably, these questions are just as relevant for much of the business sector (potentially even for high tech manufacturing firms with separate R&D departments). A number of general (non survey-based) studies provide recommendations on how successful public sector units organise their innovation activities, along with key

Page 15

organisational/management factors that promote innovation. These provide some helpful inspiration for designing indicators on this topic. An example here is Eggers and Singh (2009), who place focus on the culture of innovation and a need for a more systematic focus on the innovation process and how it is organised. In addition, the survey behind the Korean Government Innovation Index inquires about a broad range of aspects concerning the organization of innovation, such as: Innovation leadership, Vision and Strategy, Personnel capability and Systemisation of management. Some general results concerning elements of innovation organisation, drawing on ReD Associates (2005), Mulgan and Albury (2003) and Audit Commission (2007), are: New ideas generated by possibilities (e.g. due to new technology, changes in regulations, etc.) and less by pressing needs Importance of willingness to accept risk Project leaders (often middle mgmt) as driving force for implementing innovations Allocating time and resources to innovation important, but not enough; need middle mgmt to push good ideas through Cross-disciplinary teamwork Active support of top mgmt (though not necessarily as a driving force for actual work) Staff flexibility and competence development Low user involvement (common finding, though all papers argue that more focus should be placed on users) Lack of measurement of project results (perhaps intentionally resistance to this may be due to concerns that savings will be taken away from organisation budget. Incentives seldom used to promote innovation (though large effects experienced when they are used) Clear targets and goals for innovation activities (i.e. that blue ocean is less suitable for the public sector) Need for flexibility in innovation processes Importance (and lack) of mechanisms for learning and dissemination

Learning and interaction are a part of the organisation of innovation, and are covered separately below (though without being able to completely separate these topics). In many of the studies mentioned in section 2.1, there are a number of examples of questions on the attitudes or values of organisations, such as attitudes towards risk-taking, value of learning from success and failures, looking for new ideas or new ways of doing things, whether the organisation encourages innovativeness or creative thinking. However, innovation surveys generally strive to keep focus on objective characteristics, and thus questions on attitudes or values may be too subjective (or at least would need to be approached in an alternative way).

Page 16

2.5 Dissemination of innovations and learning

Dissemination of innovations and learning activities are an important part of innovation in public sector institutions, and it also has direct policy relevance in a number of ways: how do public sector institutions encourage learning as an organisation? And how do they seek and access information from other actors? In terms of policy relevance, interest here stems both from a desire to increase learning, exchange of best practices, etc. across public institutions, and a strong interest in increasing public sectors contribution to business sector growth and innovation: public-private interaction; procurement policies, etc. There are two directions at play here: dissemination (outbound) and learning (inbound). Business innovation surveys generally focus on inbound diffusion and collaboration, though there are some examples here of surveying dissemination methods. Audit Commission (2007) examined both directions, with more or less the same list of methods: special launch events, local seminars, sending guest speakers, regional networking meetings, own website, national seminars, journal articles, central website, mailshot, hardcopy newsletter, email newsletter. Dissemination and learning are very much related to knowledge management (KM). A series of Knowledge management surveys were conducted around 2001, and a smaller set of KM questions have been included in business innovation surveys. The Canadian KM questionnaire has a number of questions on KM practices under the headings of: policies and strategies, leadership, incentives, knowledge capture and acquisition, training and mentoring, and communications. Some of these questions may provide inspiration for any attempts to develop a set of indicators covering the organisation of innovation, etc.

2.6 Types of interaction

A natural starting point in examining linkages indicators are those used in CIS for businesses, i.e. where firms are asked on the importance of a variety of information sources for their innovation activities, and on innovation cooperation partners. As for businesses, there are a number of different internal and external sources of knowledge for public sector organisations innovation activities. And, organisations may draw on this knowledge in a number of ways, one of which is through active cooperation. Despite increasing focus on renewal within public sector institutions themselves, there is still great interest in what the public sector means for business innovation, and on public-private interaction in general. There might be a number of different types or forms of public-private interaction (regardless of whether this involves innovation), including: Businesses as suppliers Outsourcing of activities previously done by organisation itself Privatisation of public services previously provided by organisation itself.

Page 17

Public-private collaborations in providing a service or completing a task Businesses as users, involved in public sector activities (user groups, etc) PPP Public Private Partnerships, collaborative projects that involve both development and implementation

2.7 Procurement for innovation

Public procurement is becoming an important issue for innovation policies, based on the idea that public procurement can be used to promote innovation in businesses. In terms of measurement, procurement can potentially impact innovation in 2 directions: contributions to innovation in the organisation itself and promoting innovation in other organisations. While there is ample experience with measurement of acquisitions of goods and services for innovation, there is little if any experience with the development of indicators concerning procurement practices in particular and their role in innovation. An expert group report on public procurement for innovation (European Commission, 2005) examined the role of procurement for innovation. Among the main issues identified were: Whether a systematic approach is used in gearing procurement practices to innovation Intelligent customers (i.e. trained purchasers that have the knowledge needed to demand innovative solutions) Early engagement of suppliers (getting their feedback on what is feasible and how tenders and projects should be designed) o Tenders and contracts (role of EU directives and other regulations; specification of tenders and contracts) Details on actual contracting work o Interaction with suppliers, mgmt of contracts o Objectives of procurement (e.g. eco-innovation) o Types of procurement

2.8 Innovation drivers

Innovation drivers key factors that drive innovation processes have received sizable attention in policy circles. Examples here are user-driven innovation and employee driven innovation. In addition almost all existing studies of PI, in particular those based on some form of PI survey, devote substantial space to factors driving public sector innovation. This may in part reflect that drivers are seen to have a greater influence on public innovation than for business innovation. An example here is that many public sector innovations may simply be dictated either directly or indirectly from external sources (policy changes, regulations, etc). In essence, drivers may be different types of factors, for example: Sources of new ideas

Page 18

Pressures for change Changes that present new possibilities

The NAO 2006 study considers a long list of drivers (triggers): Efficiency drives, new government priorities, other factors, response to crisis, change in ministerial priorities, new technology, work with peer organisations, change in policy environment, changes in resource use, implementing EU policies, change of function, private sector, direct action by citizens/campaigns, spin-offs from other work. In addition to these triggers, origins of innovations were also examined: such as levels of management, levels of government, private sector, other public sector organisations, EU. Innovation drivers are a central aspect in a number of other studies. For example, all three surveys mentioned in Borins (2006) examine the initiators of innovation (Politicians, Agency heads, Middle management, Frontline staff, Interest groups, Citizens, Clients, Others). Dunleavy et al (2008) propose the following list of indicators for Origins of innovations: how many innovations due to EU regulations, how many due to ministerial or political suggestions, due to senior staff suggestions, due to middle and/or frontline staff suggestions, due to customer suggestions, due to other public sector organisations.

2.9 Barriers to innovation

There is also substantial focus on barriers in existing studies of PI. Questions on barriers to innovation are also common in business innovation surveys. Below, some selected studies are described, that provide a number of examples of barriers to public sector innovation6. Mulgan and Albury (2003) identify a number of factors that may act to hinder public sector innovation. Among these are: risk aversion, delivery pressures and administrative burdens (i.e. no time to think about innovation), poor risk management skills, short term horizons and budgets, reluctance to close down failing programmes or organisations, constraining cultural or organisational arrangements, lack of incentives, over-reliance on high performers as source of innovation Clark et al. (2008) suggest examining the following barriers: Bureaucratic culture, risk aversion, entrenched practices and procedures, trouble/busy coping with large scale reforms, lack of capacity for organisational learning.

See also the review in Thenint and Basset (2010).

Page 19

The NAO (2006) study includes the following in their survey: reluctance to new ways of working, stakeholders with different interests, getting the resources, fragmentation, coordinating stakeholders, lack of agreement on objectives, risks of audit, political uncertainty, lack of leaders, lack of training, working with contractors. Borin (2006) lists the following potential obstacles to innovation in the public sector: regulations and laws, funding, cooperation partners both internal and external, technology, union opposition, public opposition, concerns about loss of control and responsibility, opposition from private sector businesses, lack of clear vision or political support.

2.9.1 Unsuccessful innovations or abandoned projects

As discussed above, the willingness to take on risk, and the ability to learn from failures are important issues for public sector innovation. This motivates gaining more information on failures, both implemented changes (innovations) that did not meet their objectives or had negative impacts, and innovation projects that were abandoned. In addition, this issue relates to the important aspect of implementation; how are good ideas implemented into actual innovations and what are the main barriers to implementation? To our knowledge, these are not issues that have received much coverage in existing surveys, though there are a variety of types of questions that could be considered, such as: Ask for examples of innovation projects that were abandoned or failed to be implemented. Give reasons why not implemented or why a failure. Impacts of these unsuccessful innovation projects. These may be both negative and positive (e.g. learning effects vs. no-more-experiments attitude).

2.10

Outputs and objectives of innovations

Objectives and impacts of innovations are at opposite ends of the innovation process; objectives are at the beginning and shape how innovation processes are conducted, while effects are the actual outputs at the end of the process. However, they both concern the same aspects, thus making sense to consider them together (where objectives can be considered as measures of intended outputs). Output indicators range from output or performance measures (where the main business sector example is the share of innovative sales) to statements on the impacts of implemented innovations or innovation activities to broader objectives. Key questions here are: What are the most important impacts of innovation activities? What output or performance measures currently exist? Van Dooren et al (2006) discuss the use of output measures in OECD countries and related issues:

Page 20

Output measures are typically very specific to individual sectors (and are very seldom as complete a measure as sales revenue is for businesses). ReD Associates (2005) and Van Dooren et al (2006) distinguish between two broad types of services: o o Individual services that are consumed i.e. services are provided to a specific user. These services are thus comparable to private sector services. Collective or public good services that have to do with administration, monitoring, policy/regulation development. These differ greatly from the private sector and have a high public good aspect to them.

There are a number of potential (incentive) problems with the use of output measures, such as gaming, distortion of overall objectives (over emphasis of the ones that can be measured). Output/performance measures often concern the following types of measures: o o o o o Number tasks completed Time for completing a single task Percentage with success (i.e. where the service activity achieved its purpose or goal) Service quality Share of users with access or use of new service

Given the lack of a common output measure (such as sales revenue or profits, etc), it would appear to be very difficult to construct generic quantitative measures that could be applied across different public sector institutions. Quantitative output measures are likely only possible for surveys designed for specific sectors, such as health, education or elderly care. Work is however being undertaken to develop comparable output measures in selected public sectors, and it may be worthwhile to gain a better idea of how far along this work is, and if there is user interest in more detailed examinations of selected sectors (e.g. with survey modules that are specific to a single sector). Concerning qualitative impact measures (such as those found in business innovation surveys), there is little experience from existing public sector innovation surveys. However, one way to approach this topic is to look in detail at the types of objectives public sector institutions may have (both large and small), along with work in business innovation surveys. Qualitative impact measures (or expected outcomes) are key sources of information on what types of objectives, organisations sought to achieve in their innovation activities. Given that types of questions are very similar for objectives and effects, there is often a trade off between the two which can potentially mean that only one is included in a survey. Effects are arguably more useful as measures for policymaking as they provide information on actual effects and not just intentions. However, effects may be difficult to answer for organisations, which in turn affect the reliability of these measures. First, due to time lags, effects may not have materialised within the reference period of a survey. Second, some impacts may require analysis and evaluation to discern whether they have actually taken place. However,

Page 21

even if this is the case, information on expected effects after the implementation of an innovation may be quite different from intended effects (i.e. objectives) at the beginning of an innovation project.

2.11

Summing up

Existing studies of public sector innovation provide a number of insights on public sector innovation, and suggest a number of areas for focus, examples and useful discussion of key issues. This, together with standards on measuring innovation in businesses, provides a point of departure for project work in developing a measurement framework for public sector innovation. An important part of this development process is dialogue and testing with users and respondents in each of the Nordic countries. The discussion above motivates the examination of a number of key issues in development of definitions and indicators. For example: How is innovation conceptualized? Is it possible to measure innovation expenditures? How is the role of ICT important for innovation, and how can this be characterized? Is knowledge on innovation culture and how organizations innovate important for users? What key elements of their innovation activities can organizations themselves identify? What do users need to know about external interactions? How do organizations view the importance of external activities for their innovation? How can data on drivers and barriers to innovation be used? What do organizations themselves identify as most important for them? How can innovation in the public sector be related to that in the business sector? How does the heterogeneity of public sector organisations impact measurement?

A model for the measurement of public sector innovation

This section outlines a model for measurement of public sector innovation, in order to identify key areas for indicators and how they fit into an overall innovation framework. We have discussed above a number of important differences between businesses and public sector organisations. However, in its most generic terms, a model of innovation in public organisations has the same basic elements as for businesses. Public organisations need to specify their objectives, invest in innovation activities, organise the process of innovation that

Page 22

includes the generation and adoption of new ideas, their implementation, and depends both on internal capacity and linkages with external actors. Innovation may yield outputs in terms of implemented innovations and a broad range of potential outcomes. And finally, this entire process is influenced by a number of drivers and barriers (framework conditions). An overall model is important for setting individual indicators in the appropriate context. While these generic elements may be similar to that for businesses, there may be very important differences in the details of these elements; for example in the characteristics of services provided, in the decision making and organisational structure, general framework conditions, objectives, and the overall process by which ideas are created, transformed and implemented as innovations. This suggests that while the overall model may be very similar, when specified in detail the measurement framework will need to be different to capture the specificities of public sector innovation. The figure below illustrates the main elements of the model.

A model of innovation in public sector organisations

External actors (users, businesses, universities, other public organisations) Outcomes (societal, economic)

Inputs (investments, training, competences)

Innovation process (organisation, culture, interfaces)

Outputs (innovation, impact on organisation itself)

Inputs to the innovation process in a public sector organisation include the organisations investments in research, development and innovation, such as knowledge purchases from consultants or salary and training expenditures for staff members working with innovation. Another important input is the actual staff of the organisation. Characteristics such as the educational background, experience and diversity of the employee base can shape the propensity of the organisation to innovation and the type of innovative activities that it undertakes. Finally, another key input to innovation in organisations is the availability of an effective technological infrastructure that can catalyse and facilitate innovative activities.

Objectives

Framework conditions (Policy, rules, budget, governmental structure)

Page 23

The second element in the model deals with the actual innovation process in public sector organisations; i.e. how organisations innovate. The innovation process in a public sector organisation is also affected by its culture; for example, attitudes towards risk and change, incentive structure and perception of barriers to innovation. On a broad level, we can identify four elements of how the innovation process is organised in public sector organisations: first, the placement of innovation in the organisations overall strategy (or business model); second, the role of management in promoting innovation in the organisation; third, the structuring of innovation processes; and fourth, the competences within the organisation. These elements work and potentially have different roles at different stages of the innovation process: initial generation of new ideas, the subsequent development of good ideas, and actual implementation of innovations. Incentive structures are both a factor influencing innovation processes within the organisation and as part of framework conditions, determining incentives for the organisation as a whole. Incentive structures are made up of the direct rewards (or penalties) to individuals and the organisation for innovation, but also a broader set of internal and external factors that promote (or hinder) innovation activities. An additional aspect is linkages and knowledge flows, both within and outside the organisation. How organisations seek external information, cooperate with others, and diffuse their own innovative ideas, are characteristics that shape the innovation process. As discussed in Bugge et al. (2010), organisations may have a number of different interfaces with for example businesses, citizens and other public sector organisations. These three general types of interfaces may take on a variety of forms and place differing demands on the organisation in capitalising on them for their innovation activities. These innovation activities can result in a number of impacts within and outside the organisation. These can be distinguished between outputs (goods and services delivered or other activities by public sector organisations) and outcomes (broader end-outcomes of public service activities, where these outcomes will generally also be affected by a variety of other factors) (Van Dooren et al, 2006) A main output is innovations themselves actual implementation of changes to services or other parts of the organisations operations. Some organisations may be more prone to incremental improvements in their organisational processes while others seek to develop new types of services. Successful innovation is associated with some form of performance improvement, either in terms of higher quality in the organisations activities, increased efficiency, or both. In addition, innovation efforts can be associated with greater satisfaction among both employees and users. Social outcomes (social cohesion, equality, reduced crime, poverty reduction, better educated population, improved health, etc.) are also important, as they represent central aims of public services. Moreover, successful innovation can carry other, intangible benefits, such as improving the image of the organisation and the services it delivers, thus strengthening its legitimacy and trust from users or other stakeholders.

Page 24

Indicators and definitions a preliminary framework

The development of international guidelines for measuring public sector innovation is a lengthy trial and error process, where the end product of this process would be a set of definitions, indicators and classifications that have been accepted by working parties within the OECD (NESTI) and Eurostat (WPSTI). The indicators and definitions here can be considered a step in this process. This section presents the preliminary framework that was developed and used to conduct testing studies with respondents for the Nordic project. With respect to the further development of this framework, the rationales and discussions that lie behind this framework are at least as important as the definitions and indicators themselves. Hence, we will draw extensively on results and insights from user and respondent studies and discussions of indicators based on them. An important issue that goes across all indicators in this framework is the degree of comparability, both across public sector organisations and with guidelines used for businesses. Therefore, we will discuss the issues of heterogeneity and comparison with the Oslo Manual repeatedly over the course of discussion and conclude with an overall assessment on both these issues.

4.1 Defining innovation

An important issue for the conceptual framework is how to define innovation in the public sector. This section discusses and presents the definitions used for cognitive testing in the respondent study, and the implications of testing for their further development. Implementation. As discussed above, a common characteristic of definitions of innovation in both the public and business sector is a requirement that innovations must have been taken into use by the organisation. There was also broad agreement to this in the user and respondent studies. Nature of public services. For the public sector, we have mainly people-oriented services. This puts the process at the forefront; many can be social, deal with work methods, or methods involving what people do or the procedures they follow. There are clearly also technologically oriented innovations, and organisational innovations along the same lines as for businesses; but it is important to ensure that purely non-technological product and process innovations are viewed to fit within the definitions (i.e. to the extent that they meet the criteria specified). This point was also brought up in meetings with users and respondents; that innovations should also include softer innovations that involve significant changes but no new technologies or equipment.

Page 25

The nature of many public services, where product, process and other methods are less distinct from each other, argues for a greater reliance on an overall definition of innovation in surveys, which influences how it should be defined. An overall definition will still depend on the definitions of individual types of innovations; this is unavoidable unless one was to fully exclude them and only use a single definition (which has not been considered here). However, the definition should to a certain degree be able to stand alone (i.e. have at least partial meaning on its own, and not defined solely in terms of the 4 types). In addition, it will be important that criteria require that changes are significant; more consideration needs to be placed on this aspect for the overall definition, along the same lines as is done for the individual types. Perceptions of what an innovation is. How innovations are perceived by respondents and thus also the understanding of definitions are important to take into account here, and perceptions may differ between the private and public sector. For example, the notion of improvement may be understood differently in public sector contexts, and depend on the perspective of the respective stakeholders. Whereas companies in the private sector operate with a single objective, public organisations count on a multiplicity of objectives, e.g. increase quality, equity and efficiency. As a result, improvements in e.g. education can be perceived differently based on the respondent. On comparability with the Oslo Manual. In broad terms, the preliminary definitions of innovation shown below seek to maintain some degree of comparability with the Oslo Manual, but at the same time attempt to take account of the nature of innovation in the public sector. There are a number of reasons for this. First, while user needs differ across stakeholders, some stakeholders stressed comparability with businesses as being important for the usefulness of public sector innovation data. Second, in a general sense, both respondents and users considered the approach used in the Oslo Manual (i.e. the criteria and types of innovations) to be applicable to the public sector provided the wording was made less business-oriented. Third, maintaining a degree of comparability with definitions for business innovation may aid in forming a common understanding of what an innovation is. Change vs. improvements. There is wide acceptance that while innovations must have been implemented, they need not be successes. Despite this, definitions of individual innovations in the Oslo Manual require improvements, on the argument that these definitions deal with the characteristics of the innovations and not their effects. However, it is unclear whether this argument is also applicable for the overall definition, since it covers a variety of innovations. The proposed definition of overall innovation (and accompanying text) is stated below:

Page 26

An innovation is the implementation of a significant change in the way an organisation operates of in products provided. Innovations comprise new or significant changes to services and goods, operational processes, organisational methods, or the way the organisation communicates with customers. The innovation must be new to the organisation, but it may have already been implemented by other public organisations or businesses. The innovation must constitute a significant change for the organisation. It must significantly affect the operations or character of the organisation. An important requirement is implementation. Innovations much have been taken into use by the organisation. However, organisations do not need to have developed the innovations themselves.

The first sentence of the definition provides a definition of an innovation on its own. And, importantly, attaches much greater weight to the word significant, measuring significant in relation to overall operations. However, the overall definition must correspond to individual types, so these are listed in the second part of the definition.

4.2 Individual types of innovations

Types of innovations. At least in general terms, the four types of innovations product, process, organisational and marketing appear acceptable to public sector organisations, with respect to the scope of innovations. In initial discussions, both users and respondents were asked how they understood innovation and also how they viewed the four types of innovations (product, process, organisational and marketing)7; whether each type was relevant and whether they thought other types of innovations should also be included. The general view in most cases was that all four types seemed relevant and adequately covered innovations in public sector organisations. The term, marketing innovations does not seem

In this first phase of the studies, participants were not provided with a formal definition beforehand, but instead were asked to comment on these general concepts. And exception here is Finland, where participants were presented with the definitions from the Oslo Manual beforehand.

Page 27

appropriate for the public sector. On the other hand, there appears to be fairly wide recognition that marketing is important for the public sector; promotional and other types of initiatives are important for the success of public sector organisations. These types of innovations have been termed communication innovations. The preliminary definitions given below include the four types of innovations, with the central requirements being that innovations must have been implemented, be new to the organisation itself and constitute significant changes. The discussion above also noted a number of other types of innovations that have been proposed in earlier studies, such as social innovations, policy innovations, strategic innovations and systemic innovations. In general, we have not found good reasons to include these additional types of innovations, with the possible exception of strategic innovations. Social innovation is arguably very relevant to public sector innovation, however it is mainly characterised by the objectives of the innovations as opposed to the characteristics of the innovations themselves. Hence, this would argue for covering social innovation by collecting data on the importance of social objectives for innovations. Concerning policy innovations, we note that respondents in the NAO (2006) study (where innovations were not defined in the survey itself) did not provide any examples of policy innovations, suggesting that they do not consider these as innovations. Our discussions with Nordic users and respondents provide additional support to this. The basic message here was that policies are changed all the time, but should not in themselves be considered innovations, although they may be an important driver of innovation in other organisations. And, changes to the way that policies are developed or implemented can be more appropriately considered as organisational innovations. Systemic innovations essentially involve changes in organisations relations with others. Innovations in external relations are actually already included in the definition of organisational innovations used in the Oslo Manual. Hence, it would seem most appropriate to follow this approach for the public sector. Strategic innovations involve changes in the overall strategies and objectives of organisations. These could also be considered business model innovations. While we do not include them here in these preliminary definitions, the case for not doing so is less clear than for the other types discussed here. Strategic innovations are at the core of much of the management literature on innovation8. While these managerial theories of innovation can be related to innovations in specific areas or activities, the focus is still on implementing a new overall strategy or business model. The most appropriate business models would of course be different for public sector organisations, but strategic innovations are still arguably of great

Examples here are Chesbrough (2003) and Pralahad and Krishnan (2008), but there are many others.

Page 28

interest in the public sector. One problem with measuring strategic innovations is that the introduction of a new strategy does not necessarily imply that actual changes will be made throughout the organisation as a result of it. And, if the new strategy does result in concrete changes in the organisations operations, we should be able to capture these through the other types of innovations. Our approach to capture aspects concerning strategic innovation is instead to ask a series of questions on innovation strategy and how innovation is organised in the public sector. More details on this are provided below.

4.2.1 Product innovations

Concerning the definition of product innovations, the first part of the definition used in the Oslo Manual (OECD/Eurostat, 2005; definitions also shown in section 2.2 above) would also seem applicable to public sector organisations. However, the second part is very technical and also has a primary focus on manufactured goods. What are the main types of characteristics that are central to new public services? Some elements could be: New combinations of services ICT-oriented services Improving user-friendliness (customer access, ease of use, individualisation to user needs, improving services provision to target groups)

Drawing on this, the following definition of product innovations is proposed: A product innovation is the introduction of a good or service that is new or significantly improved with respect to its characteristics or intended uses. This includes significant improvements in customer access, ease of use, technical specifications or other functional characteristics that improve the quality of the good or service offered.

4.2.2 Process innovations

Concerning the definition of process innovations, there are a number of issues that can be considered. First, it is important to move away from a description of processes as manufacturing production methods. Second, the definition of process innovations should be distanced from service innovations on the one hand, and from organisational innovations on the other hand. Concerning service innovations and process innovations, it is mainly a question of whether new methods affect the characteristics of the actual product that the user experiences (service) or if it affects the delivery/creation process itself (process). This distinction can partly be accomplished by the aim of the innovation; process innovations should mainly have to do with efficiency and cost. Though, both products and processes have to do with quality in different ways the quality of a product is part of its characteristics while quality for the process is more concerned with consistency (maintaining a certain level).

Page 29

A process innovation is the implementation of a new or significantly improved method for the creation and provision of goods and services. This includes significant changes in methods, equipment and/or skills with the aim of improving quality or reducing costs or time of delivery.

4.2.3 Organisational innovations

In a broad sense, organisational innovations would appear to be fairly similar in the business and public sectors. However, there are differences in which terminology is most appropriate. Questions can also be raised concerning the Oslo Manual definition. For example, business practices is a broad term that is used in the definition to cover a number of different types of changes; and could both imply small and large operations. In addition, it is somewhat difficult to consider all types of potential organisational innovations as methods, which makes the wording of the Oslo Manual definition somewhat difficult. Finally, clarification of how large the innovation should be for it to be considered an innovation is not included in the actual definition (instead given in accompanying text). Among the types of changes that could be considered for the public sector are: New incentive structures New practices for gathering and disseminating knowledge Changes in organisational structure New management systems (e.g. Lean management, Strategic Performance management, Quality management) New programs for developing staff competences or strengthening innovative capability New external relations

The following definition is proposed for the public sector: An organisational innovation is the implementation of significant changes in the way work is organised or managed in your organisation. This includes new or significant changes to management systems, workplace organisation and/or programs to improve learning and innovative capacity.

4.2.4 Communication innovations

Generally, there isnt a market in the public sector. However, many respondents stated that promotion (i.e. marketing) is very important for their activities, and an area where innovative approaches are needed. The public sector is often just as reliant on good methods to launch and promote their products.

Page 30

In addition, public sector organisations may make a number of campaigns or promotions that essentially dont provide a service to users. Their goal is instead to influence behaviour in some way. Is this a service or a promotional campaign? Some respondents considered this type of new initiative to be a service innovation, but most categorised it in terms of a marketing or communications innovation. An additional issue is commercialisation i.e. launching new products, implementing them on the market; getting them taken into use. This type of activity is relevant for the public sector, but the term commercialisation was deemed by many to be to business-oriented. And, can commercialisation by distinguished from promotionor can it fit under promotion? If commercialisation is understood as described above, then it would appear that it could very well fall under the heading of promotional methods. These considerations motivate the following definition for communication innovations:

A communication innovation is the implementation of a new method of promoting the organisation or its goods and services, or new methods to influence the behaviour of individuals or others.

4.2.5 Discussion
This section discusses issues related to the definitions above, drawing in particular on the results of cognitive testing of the draft questionnaire. In general, the respondents found the definitions easily understandable, but some expressed criticisms on the scope of the definitions and how the concepts should be applied when answering the questions. By scope, respondents mainly referred to the degree of change required. A number of respondents argued that the definitions were too inclusive in this respect without a clear distinction from day-to-day routine changes, which could risk that all public sector organisations are found to be innovative. In some cases this also related to whether changes to small parts of the organisations operation should be included as innovations. When asked, respondents recognized that the term significant improvement was used to distinguish from minor changes, but there was uncertainty as to how this should be understood. In addition, respondents did not appear to give much attention to this criterion. Almost all respondents provided examples of innovations (see Annerstedt and Bjrkbacka, 2010) which, at least based on the short descriptions given, appear to fit the criteria given by the definitions. Possible exceptions here are some of the examples of IT-based applications which could be better characterised as routine updates. Furthermore, examples of all four types were given. These examples provide some further support for comments given by respondents, though they suggest that only minor modifications may be needed to strengthen definitions. In preliminary (open) discussions, a number of respondents and experts pointed out that it is very difficult to distinguish one type of innovation from another, and that if innovations are only defined in terms of subtypes (i.e. product, process, etc) respondents may feel that the definitions do not really capture their innovations. If many public sector innovations span

Page 31

more than one type, then it would be constructive to consider how definitions can best reflect this. These views motivated the inclusion of an overall definition of innovation in the questionnaire (in addition to the four types of innovations). This need was only partially confirmed in the cognitive testing; respondents noted that many innovations span more than one type, but respondents were for the most part able to identify individual types of innovations. Respondents also stated that the language and type of questions (sometimes explicitly and sometimes implicitly) are somewhat skewed towards the business world. This relates to the types of innovations and to the formulations. This essentially suggests that the way innovation is defined and classified does not fully fit with the way public sector organisations view innovation. This is clearly an issue that should be kept in mind in future work, however it should also be balanced against the fact that these definitions are new to public sector organisations and respondents were able to provide relevant examples.

4.3 Innovation activities

A second key element of the conceptual framework is innovation activities. These comprise the main inputs to the innovation process, but also provide information on the innovation process itself; i.e. what types of activities are being undertaken and also what types of work or technology is outsourced or acquired. The definition of innovation activities in the Oslo Manual is: Innovation activities are all scientific, technological, organisational, financial and commercial steps which actually, or are intended to, lead to the implementation of innovations. Some innovation activities are themselves innovative; others are not novel activities but are necessary for the implementation of innovations. Innovation activities also include R&D that is not directly related to the development of a specific innovation. (OECD/Eurostat, 2005) In looking at this definition from the viewpoint of the public sector, it can be considered to place too much emphasis on technical aspects, on R&D (a concept which may not be that familiar in many public organisations9) and on production innovation processes. At the same time, one could argue that it doesnt clarify enough the knowledge building activities t hat may be needed for innovations.

In fact many users and respondents from the public sector have implicitly used the term development to indicate a very weak form of innovation; day to day, incremental changes that lie at the borderline of being considered innovation activities.

Page 32

Based on this, the following definition was proposed for the draft questionnaire:

Innovation activities are all steps which actually, or are intended to, lead to the implementation of innovations. These include development and implementation activities conducted in-house, in cooperation with others, or externally through acquisitions of external knowledge, machinery, IT or know-how related to innovations. Innovation activities also include training and other competence building activities necessary for the development and implementation of innovations.

This includes a great variety of different activities, both in-house and external. Gaining information on which types of activities public sector organisations engage in can thus be useful in characterising their innovation processes. And given that it may often be the case that individual innovation projects may include more than one type of innovation, the list of activities should reflect not just work on new products or processes, but also organisational and communication innovations. While there is potential for creating a more detailed classification of innovation activities, the following initial list of activities has been used in testing with respondents: In-house activities: o In-house research and development (creative work to increase the stock of knowledge towards the development of innovations) o Other in-house innovation activities (Planning and design; market research and other user studies; other activities to implement innovations, such as feasibility studies, testing and other preparatory work) o Internal or external training and education of staff for innovation activities (development and introduction) External activities: o External research and development o Other consultancy services (management, market research, technical, etc.) o Acquisitions of other external know-how (patents, licenses, other types of knowledge) o Acquisitions of machinery, equipment and software

These categories are thus very broad, breaking down in-house activities in terms of R&D, other in-house activities and training, and external activities in terms of extramural R&D, other external knowledge, consultancy services and machinery, equipment and software. These categories also attempt to take account of the broader, less technological nature of public sector innovation activities. The significant policy focus on public procurement motivates obtaining information on what types of innovative procurements public sector organisations undertake, and also from what types of suppliers the purchases are made. A simple breakdown of suppliers for acquisitions is:

Page 33

Private businesses Public research (universities and government research institutes) Other public sector organisations

Initial work with focus groups suggested that data on innovation expenditures would be very difficult to collect. In addition, it was viewed that many potential respondents would be unfamiliar with the concept of R&D and likely not maintains accounts of R&D expenditures. This motivated the use of an even simpler classification into three categories: In-house expenditures External expenditures on services and other immaterial goods (extramural R&D, other external knowhow and other consultancy services) Acquisitions of machinery, equipment and software

4.3.1 Discussion
In general, the results from tests concerning innovation activities and innovation expenditures were not very encouraging. Concerning the actual definition of innovation activities, there were only limited comments. The basic message from these few comments was that the definition was understandable, but abstract, which made it difficult to have in mind when answering other questions that are based on innovation activities (such as linkages and barriers). The results of the study showed that almost all respondents stated to have conducted inhouse R&D. This, combined with confusion in earlier discussions about the meaning of development, leads us to question whether respondents understand the concept of R&D in the same way as defined in the Frascati Manual (OECD, 2002). This suggests that either public sector innovation surveys should include a lengthier explanation of what R&D is and how it is defined, or alternatively, they should not isolate R&D activities in questionnaires. Finally, all respondents stated that collecting data on innovation expenditures would be very difficult, with some stating that they would not be willing to devote the time needed to produce a good estimate and others stating that it is not possible to calculate the expenditures. This suggests that questions on innovation expenditures may suffer from lack of accuracy and their inclusion in questionnaires may have a negative impact on response rates. At the same time, it should be noted that this experience with measurement of innovation expenditures might not differ greatly from that for businesses. One potential option that might make it easier for organisations to respond is to ask for total expenditures devoted to innovation projects. Given that projects may be better registered and have set budgets, organisations may be able to provide accurate data on these. However, such an option would exclude innovation activities that are not part of projects, and could potentially include some expenditures within projects that do not fall under the definition of innovation activities.

Page 34

4.4 Linkages
4.4.1 Innovation cooperation and information sources
Questions on innovation cooperation and on the importance of information sources for innovation have been extensively used in business innovation surveys and are outlined in the Oslo Manual. They are equally relevant for the public sector, though the list of sources or partners should be modified to fit the public sector. The following list has been used in this project: Enterprises o As suppliers (incl. consultancy services) o As clients / users Public organisations o As suppliers o As Clients / users o Universities / gov. research institutions (not as clients) o Other public organisations Citizens o As clients / users (i.e. user surveys, observations) Other sources o Conferences, seminars, other gatherings o Scientific journals; specialist journals o Professional and industry associations

Respondents generally were positive on the above list of sources and partners. However, there were a number of criticisms of the question on information sources. Many respondents found the concept of information sources to be very vague and also had difficulties distinguishing it from innovation cooperation. Information sources essentially cover all forms or channels of knowledge transfer, both formal and informal, where innovation cooperation focuses on more formal forms of interaction. A potential alternative to information sources would be to focus instead on more concrete forms of informal information channels. Examples here are the internet, networks, hiring of specialised personnel, or evaluations.

4.4.2 Who developed the innovations?

Innovation cooperation can refer to cooperation at any stage of the innovation process and may or may not be concern activities that are directly linked to a specific innovation. Questions that relate directly to the role of external actors and the organisation itself in developing innovations complement data on cooperation. The Oslo Manual outlines a simple breakdown of whether (product or process) innovations were developed by the firm itself, together with others or mainly be others. However, for the public sector, policy needs suggest that there is great interest in identifying whether others are private businesses, public research or other public sector organisations.

Page 35

4.4.3 Innovative procurement

We discussed above measures of external acquisitions for innovations. These provide some useful information on the role of procurement in innovation, however these do not examine whether procurement practices are consciously used as tools to promote innovation, either in the organisation itself or in others. Both the knowledge on this focus on innovative procurement and on which types of practices are employed would provide a more detailed picture of the role of procurement in innovation. The following list of procurement practices was employed (where organisations can be asked whether the practices are used to promote innovation in their own organisation or in other organisations): Acquisition of services, components or software from ICT-suppliers Acquisition of other machinery and equipment Contracting management consultancy services Contracting user studies Acquisition of other consulting services Outsourcing of service provision Collaboration with others in the provision of services Collaboration with contractors on development projects Competitive contracting procedures

Respondents had some difficulties in understanding this question, more specifically on what was meant by using to promote innovation and how they could use these practices to promote innovation in other organisations. Respondents understanding of this question can potentially be improved by reformulating the question.

4.4.4 Innovation drivers

As discussed above, there has been a large amount of focus on the drivers of innovation activities in public sector organisations, in part due to the fact that public sector organisations typically have less autonomy in their decision making and are thus subject to greater influence by external forces. Drivers can be either people, organisations or other factors that push organisations to innovate. However, in order to avoid overlap with information sources, attempts were made to distinguish drivers from information sources. Drivers here are driving forces that push organisations to innovate as opposed to knowledge that organisations find useful for their innovation activities. This distinction or refinement of what a driver is has also influenced our proposed list of driving forces: Internal driving forces o Management o Staff External or political forces o Mandated changes in budget for your organisation o New laws or regulations o Changes, innovations implemented in partner or higher level organisations (e.g.. new procedures or services, organisational changes, deregulation)

Page 36

o New policy priorities Public organisations Enterprises o As suppliers o As clients / users Citizens - As clients / users (i.e. feedback, complaints; influence from associations of users)

The majority of these driving forces are actors, internal and external, with the exception of political factors. A key characteristic of public sector innovation is that it may be greatly influenced by changes from higher level organisations. These include changes in the organisations budget, new regulations, new political agendas, and potentially also innovations in higher level organisations. In all cases, these changes can have a significant impact on the organisations operations, either creating new possibilities or forcing innovation in order to adjust to the changes. Respondents generally did not have difficulties in answering this question, and also found it to be relevant in characterising their innovation activities.

4.5 Organising innovation: strategy, management and competences

We look at 4 elements here: innovation strategy, the role of management, organising innovation activities, and competences. And furthermore, these elements seek to examine the different stages of the innovation process: idea generation, development, and implementation. The list builds on both examples from other studies (for example the Korean GII), and the user and respondent studies which for example emphasized covering the implementation of innovations. Innovation strategy o An innovation strategy is included in the overall vision or strategy of the organisation o The organisation has specific goals/targets for innovation activities o Part of the strategy is to ensure specific funding in the budget for innovation activities o Evaluations of strategy and specific goals/targets for innovation are conducted regularly Innovation management and leadership o The political level supports changes o Managers give high priority to developing new ways of working o Managers support trial-and-error testing of new ideas o Top management takes an active role in leading the implementation of innovations Organising innovation o The organisation has a development department/section

Page 37

Innovation activities are mainly organised as projects, steered by a dedicated group o Members of staff have part of their time devoted to development/innovation projects o Staff have incentives to identify new ideas and take part in their development o Evaluations of the innovation processes are conducted regularly o Users are involved in the implementation process Staff capabilities o The staff has a diversity in terms of background (demographic, educational) o Staff comes up with new ideas on the design and delivery of services o Staff time devoted to learning about new technologies or other new knowledge

Concerning strategy, items here include whether the organisation has an explicit innovation strategy, whether specific targets are set for innovation activities, whether financial resources are earmarked to these activities, and whether strategies or goals are evaluated. These aspects indicate whether innovation is a planned part of the organisations vision and operations, and how committed the organisation is to implementing these plans. For example, dedicated funding indicates a stronger commitment to realising plans. There may also be different degrees of planning; for example, some organisations may have set targets for innovations, but where innovation is not an integrated part of their overall strategy. The main focus for questions on innovation management is the degree of focus on developing new ideas, supporting experimentation and leading implementation of innovations. Questions on the organisation of innovation deal with the structuring of innovation activities, for example in separate departments or concrete projects, incentive structures, evaluation, and the role of users. Final questions concern broad aspects of staff competences and focus on building knowledge capacity. Respondents generally also found this question to be relevant for characterising their innovation activities. At the same time, there were criticisms that this part contained too many questions, and that there was some overlap between some of the questions and questions on innovation drivers.

4.6 Measuring the role of ICT in innovation in public sector organisations

Our work in this area is influenced to a large degree by efforts to balance between obtaining useful information on the role of ICTs in innovation while at the same time avoiding an over emphasis on technological aspects in the innovation survey. In general, project examination of user needs confirmed the strong policy interest in the role of ICTs. However, considerably less emphasis was placed on ICTs by potential respondents, even those where ICT is a central part of their innovation activities.

Page 38

An additional question was whether the aspect of ICTs should be covered in a separate module. One difficulty with this approach is that ICT is a cross cutting themes that is relevant for all areas in an innovation survey; a separate module would mean revisiting each of these areas. Based on these considerations, a fairly compact set of questions was developed within the project, addressing: The degree to which innovations were ICT-related (E-Government) ICT related innovation activities (both in-house and acquisitions) ICT-related effects, such as improved ICT capabilities, improved user access and improved online delivery of services Lack of ICT know how as a barrier to innovation ICT-related acquisitions as a procurement tool

Given the above concerns with excessive technological focus, we have not developed questions that seek to measure how advanced organisations ICT-capabilities are. However, this remains an area of potential relevance if ICT is to be given extra focus in public sector innovation surveys. Respondents did not appear to have any problems in understanding these questions, however it is difficult based on the small number of responses to determine whether these questions will be able to provide useful information. This will depend to a large degree on how much variation can be captured across organisations.

4.7 Measuring innovation performance

In terms of generic measures that are applicable to all types of public sector organisations, our work focused on two types of measures; a list of impact measures that can be formulated both as effects and objectives, and a quantitative measure of the scope of innovations.

4.7.1 Effects and objectives

The question of measuring effects or objectives (in the case where it is not feasible to cover both) is difficult to resolve completely, as both have advantages. However, both these options will be tested in order to gain more information on the feasibility of both types. With minor modifications, we have for both cases developed the following list: Efficiency o Reduced labour costs per service/good o Reduced other costs per service/good o Increased capacity of the services/goods provision o Simplified administrative procedures Quality o Improved quality of services/goods o Services or goods offered to more or new types of clients

Page 39

o The services or goods has become more visible for potential clients o Improved customer satisfaction Information and communication technologies (ICT) o Improved ICT-capabilities o Greater user access to information o Improved delivery of services online Organisation and staff o Improved communication or information sharing within the organisation o Improved communication or information sharing with others o Improved capacity to innovate o Improved employee satisfaction o Reduced staff turnover Other o Improved environmental conditions o Improved health and safety

The above list was formulated as effects in Sweden, Iceland, Norway and Denmark. In general, respondents did not feel it was a problem to answer. This question tended to have an impression of either the actual or expected effects of their innovations. Though, a number of respondents stated that this question was too long and should be shortened. However, some respondents noted problems with linking them to more than one innovation project. This was deemed as problematic by some do they relate to one specific innovation project, or to different projects depending on the effect, or to an average assessment (none of the respondents appeared to do thisin all cases, and given the complexity of the definition of innovation, the tendency was to visualize a specific innovation project). At the same time, a number of respondents stressed the importance of effects of innovations this was what mattered in assessing and measuring innovations. One area that is difficult to cover as effects is broader societal impacts. However, these can potentially be covered as objectives, in order to measure how much importance (if any) organisations attach to broader societal goals for their innovation activities.

4.7.2 The scope of innovations

Two ways of measuring the impacts of innovations are in terms of effects on operations (e.g. increased efficiency) or what share of operations are influenced by the innovations; for example do the innovations only involve a small share of overall operations or the entire organisation. The former is difficult to develop as a general measure. It is also challenging in designing measures for specific sectors, but the potential in such case is greater to find suitable output measures. We have pursued the latter type of measure in the project, by asking respondents to estimate what share of their operations were affected by the innovations that they had implemented over the period. However, the results from this were not positive. There widely different

Page 40

interpretations of what the question was actually asking for, and in some cases respondents were not able to answer the question.

4.7.3 Qualifying innovations novelty and other potential measures

We have discussed above measures of innovative performance in terms of their impacts, both qualitative and quantitative. Other potential approaches to measure innovative performance lie in seeking to qualify the innovations themselves; how innovative the innovations are. One dimension is the novelty of innovations in relation to other public sector organisations and to businesses. Another dimension is the degree of change with respect to existing products and methods in the organisation itself. For example, how great was the change in comparison to existing practices: A significant improvement to existing practices, or completely new to the organisation, or a systemic change that represented a visible change in the organisations operations as a whole?

The latter was not attempted in the pilot questionnaire that was tested in the feasibility study, but was brought up by respondents in interviews. Measures of innovative novelty have been used extensively in business innovation surveys, and provide an important means to distinguish between organisations that have introduced innovations that are adoptive from those that are more inventive. We proposed using a concept of novelty that is similar to that used in the Oslo Manual. Instead of using new to the market, we are using new to the sector. Sector was not defined in the questionnaire, but we instead relied on organisations own perception of the term. In addition to new to the sector, the concept world new was also introduced, which refers to innovations being new in relation to all sectors including businesses. This would thus allow three levels of novelty and a more detailed measure of innovative performance. However, defining sector is not straightforward. Municipalities may for example consider all local government as their sector while many central government units may consider themselves as the only institution of its kind in the country. Some respondents indeed questioned what a sector was, in lesser cases also what was meant by world new. Results of the testing study suggest that public sector organisations will likely understand the term sector differently, which complicates use of the measure.

4.8 Barriers to innovation

There is extensive experience in collecting data on barriers to innovation in business innovation surveys, and also in a number of existing public sector innovation surveys. In general, the question is equally applicable for both sectors, mainly requiring modifications to

Page 41

lists of specific types of barriers. The following list was developed in the project, drawing in part on existing work: Political factors o Rigid laws and regulations o Lack of political commitment or opposition o Uncertainty about political environment o Budgetary rules Bureaucracy o Risk of failure o Rigid hierarchy o Lack of cooperation between departments Other internal conditions o Lack of agreement about objectives/goals for innovation o Inadequate time being allocated to innovation o Inadequate know how on new technology (ICT) o Lack of incentives for staff to innovate o Lack of incentives for organisation as a whole to be innovative o Lack of leaders or champions to support innovation o Lack of tools to measure expected benefits External conditions o Difficulty coordinating activities with other stakeholders o Problems involving private contractors o Lack of main suppliers' capability to provide innovative solutions o Conflicting private sector interests o Resistance of users to changes

Discussion and future directions

This paper has sought to develop a preliminary framework for measuring public sector innovation, based on existing work and on the results from studies of users and respondents in the Nordic countries. What has been presented here is the framework that has been used in cognitive testing with respondents. These are thus results from the first stage of work; based on these results revisions will be made in preparation for the pilot study to be held over the course of 2010. And, following this the project will provide recommendations based on the results of this study. This paper will conclude by looking at the issue of comparability both across the public sector and with businesses in order to put the results in a broader context and to suggest directions for further work.

Page 42

5.1 Comparison with the Oslo Manual

Can public sector innovation be measured with the same framework as for businesses? Our answer to this based on project work is that they cannot be measured in the same framework. However, the differences are not fundamental. As we have argued above, a generalised conceptualisation of public sector innovation is similar to that for businesses, but many definitions and questions need to be different to reflect the specificities of the public sector. In general terms, the four types of innovations in the Oslo Manual were viewed as relevant for the public sector, though definitions required modification to better suit the public sector. Main changes here were changing marketing innovations to communication innovations and modifying all four definitions to better reflect the nature of public sector services and processes. However, the definitions shown here were argued to still be too business oriented, suggesting that further modification is needed. An additional difference concerning the innovation definitions is perceptions. Cognitive testing results suggest that public sector respondents do not attach the same weight to the term significant improvement as for businesses. Definitions were thus viewed by most respondents as being too inclusive, failing to exclude minor routine changes. This also motivates further modification. Politically mandated innovations present a difficult issue. Definitions of innovation for businesses focus on change and implementation; where it is taken as given that changes are the result of decisions made by the business itself with the aim of improving its operations. For the public sector, changes may be politically mandated and potentially reversed within a short period. Our view is that it is not advisable to exclude politically mandated changes, but it may at the same time be important to identify whether innovations in the public sector are based on internal decisions or politically mandated. The concept of innovation activities used here is also quite similar to that in the Oslo Manual, though with changes to the specification of types of activities included. Where the Oslo Manual has a fairly clear distinction between activities for product-process and organisational-marketing innovations, activities here relate to all types of innovations. However, a detailed categorization of types of innovation activities has not been attempted here. Respondent studies indicate that the R&D concept is much less known in public sector organisations. At the same time, there are extreme differences between most public sector organisations which likely conduct little if any R&D, and public research. An open question for the latter is whether basic research should be included as part of innovation expenditures. It was found very difficult to measure innovation expenditures in public sector organisations. However, this is also a difficult issue in the business sector, so it is unclear whether there are significant differences between the two sectors here. Diffusion is an important focus for all sectors, but in comparison with the business sector, there is more focus on the public sectors role in promoting innovation elsewhere (particularly

Page 43

among suppliers). Our focus on the role of procurement for innovation which is not found in the business sector reflects this policy focus. Our treatment of linkages follows fairly closely that of the Oslo Manual, though it should be acknowledged that a number of other approaches are feasible and could be developed. More specifically, we have used modified questions on innovation cooperation and information sources for innovation. However, the question on information sources was found to be very vague by many respondents. Hence, in further work we will consider instead using a more concrete question on types of informal information channels. Innovation drivers have been covered in selected business innovation surveys, but receive little emphasis in the Oslo Manual or CIS. There is arguably greater focus on this aspect in the public sector, potentially due to a number of reasons. Among these are: the view that external actors play a relatively larger role for public sector innovation than business innovation; we have less knowledge on innovation processes for the public sector; and the important role of political drivers for public sector innovation. An additional aspect that is distinct to this work on the public sector is on the organisation of innovation. This topic was initially motivated by discussions with users and respondents, though there is little focus in existing surveys on how innovation activities are organised. At the same time, innovation culture and the organisation of innovation are key focus areas in many non survey-based studies of public sector innovation. This raises the question of whether this topic fits within the scope of surveys of public sector innovation. The answer to this is arguably yes, if the goal is to gain more information not just on whether public sector organisations innovate, but also on how they innovate. Coverage of the effects and objectives of innovation, as well as barriers to innovation, are similar to that in the Oslo Manual. Our initial treatment of questions on objectives or effects has been much generalised, but has not included broader objectives which are often sector specific. This aspect will be important to consider in further work. Barriers to innovation may differ greatly between the private and public sector; in particular, political and structural factors have been highlighted here.

5.2 Heterogeneity of the public sector implications for innovation measurement

As has been discussed above and in greater detail in Bugge et al (2010), the public sector is very heterogeneous. This is of course also the case for the business sector, but this diversity is arguably greater for public sector organizations. In particular different levels of government and different types of outputs both individual and collective services play a large role. There are at least two aspects to the issue of heterogeneity. The first is the public sector so heterogeneous that a generalised survey is not possible? The second - what types of aspects would be most important to examine in terms of heterogeneity? Our view based on preliminary testing with respondents is that it a generalised survey covering a broad range of aspects - is very much possible. The study of respondents (see

Page 44

Annerstedt and Bjrkbacka, 2010) covered a very diverse group of public sector organizations. We did not find any particular problems across government levels or sectors that would give an indication that a generalized approach is not possible. The overall number of participating institutions is however still fairly small (under 40), so these preliminary results should not be considered as fully conclusive. The upcoming pilot study will provide an opportunity to examine this issue on a much larger scale. However, this does not mean that specific aspects of individual sectors arent important. On the contrary, there is great interest in capturing these specific aspects. Hence, there is a need for developing questions specific to individual sectors. We have only been able to look briefly at this issue in the Nordic project, but it is clear that this is an important area for further work. Two main dimensions for classifying organisations are sector and level of government. There are without doubt important differences in innovation, organisation, etc. across both of these dimensions. However, the overriding objective here should be to capture as much of this diversity as possible in a common framework. This allows greater comparison, a common understanding of key aspects of public sector innovation and some common messages. Levels of government may differ in terms of their sphere of influence; whether they are concerned mainly with innovation in their own organisation or for others. Central government organisations are concerned with promoting own innovation, but also in promoting innovation in local government and frontline services. Hence, the framework for central government could include this specific aspect; actions to promote innovation in these two groups of organisations. Local and regional authorities will act to promote innovation in those frontline services that are part of (or administered by) local and regional government, and their own operations and innovative capability may be impacted by central government. Frontline services in contrast are mainly focused on innovation within their own organisation, but this innovative capability can be influenced by both central and local authorities. The issue of addressing differences across sectors is more difficult to address. While we stress again the importance of maintaining a common core of indicators across organisations, it can also be relevant to focus on aspects that are specific to individual sectors. These can concern effects of innovations, barriers, specific organisational issues or types of collaboration, and could either consist of a separate module or a small set of additional or modified questions throughout the survey. Further work on the issue of heterogeneity in public services could thus include more theoretical work examining case studies and other available material to suggest what form sector specific modules might take in questionnaires; and, which types of organisations should be targeted with these extra modules. It will also be important to engage in dialogue with stakeholders in selected sectors to get their input on what types of measures have the highest priority. Natural first choices would be Health and Education services.

Page 45

References
Annerstedt, P., and Bjrkbacka, R., 2010, Feasibility study of public sector organisations, Nordic project Measuring innovation in the public sector in the Nordic countries: Toward a common statistical approach (Copenhagen Manual). Atkinson, 2005, Atkinson Review of measurement of government output and productivity for the National Accounts: Final Report. Audit Commission, 2007, Seeing the light: innovation in local public services. Ayling, Winn and Wong, 2009, Measuring & Valuing Innovation in the NHS, Working paper, NHS Innovation Centre. Bertok, J., Hall, J., Kraan, D.J., Malinska, J., Manning, N. and Mattews, E., 2006, Issues in outcome measurement for Government at a Glance, OECD GOV Technical Paper 3. OECD, Paris. Borins, S., 2006, The challenge of innovating in government. Bugge, M., Hauknes, J., Bloch, C. and Sliperster, S., 2010, The public sector in innovation systems, Project Measuring Public Innovation in the Nordic Countries: Toward a common statistical approach. Chesbrough, H., 2003, Open Innovation: the new imperative for creating and profiting from technology, Boston: Harvard Business School Press. Clark, Good and Simmonds, 2008. Innovation in the Public and Third Sectors. NESTA Innovation Index Working Paper. Dunleavy et al., 2008. Innovation in Government Organisations, Public Sector Agencies and Public Service NGOs. NESTA Innovation Index Working Paper. Earl, 2002, Innovation and Change in the Public Sector: A Seeming Oxymoron, Statistics Canada working paper. Eggers, W.D. and Singh, S.K., 2009, The public innovators playbook: nurturing bold idea in government, Deloitte Research and Ash Institute for Democratic Governance and Innovation. European Commission, 2005, Public procurement for research and innovation developing procurement practices favourable to R&D and innovation, Expert group report, DGResearch. Jrgensen, L.L, 2010, Mapping user needs, Nordic project Measuring innovation in the public sector in the Nordic countries: Toward a common statistical approach (Copenhagen Manual). Koch, P. and Hauknes, J., 2005, On innovation in the public sector, Publin Report D20. Korean Ministry of Government Administration and Home Affairs, 2005, Government Innovation Index Model and Diagnosis, Powerpoint presentation. Meadow Consortium, 2010, The Meadow Guidelines, Meadow (MEAsuring the Dynamics of Organisations and Work) project, 6th EU FP. Mulgan, (2007) Ready or Not? Taking Innovation in the Public Sector Seriously. NESTA Provocation 03. Mulgan, G. and Albury, D., 2003, Innovation in the Public Sector. Strategy Unit, Cabinet Office, UK. Moon, M.J., 2002, The evolution of e-Government among municipalities: Rhetoric or reality?, Public Administration Review, 62, 4, pp.424-443.

Page 46

Moore, H.M., Sparrow, M, and Spelman, W., 1997, Innovation in policing: from production lines to jobs shops, Chapter 12 in (Eds. Altshuler, A.A. and Behn, R.D.) Innovation in American Government, The Brookings Institution, Washington D.C. Mortensen, P.S., 2010, Survey methodology for measuring public innovation, Nordic project Measuring innovation in the public sector in the Nordic countries: Toward a comm on statistical approach (Copenhagen Manual). National Audit Office, 2006, Achieving innovation in central government organisations: detailed research findings. National Audit Office, 2006a, Achieving innovation in central government organisations (main report). National Audit Office, 2009, Innovation across central government. OECD, 2007, Towards Better Measurement of Government, OECD Working Papers on Public Governance, 2007/1. OECD, 2009, Measuring innovation in education and Training, OECD discussion paper, OECD/CERI, Paris. OECD/Eurostat, 2005, Oslo Manual Guidelines for Collecting and Interpreting Innovation Data, Third Edition. Prahalad, C.K. and Krishnan, K.S., 2008, The New Age of Innovation, McGraw-Hill. ReD Associates, 2005, Bedre innovation i den offentlige sektor. Thenint, H. and Basset, J., 2010, Innovation in the public sector, Inno Grips Mini Study 10. Van Dooren et al, 2006, OECD GOV Technical Paper 2 Issues in Output Measurement for Government at a Glance. Wauters, P., Nijskens, M. and Tiebout, J., 2007, The user challenge - benchmarking the supply of online public services, prepared by Capgemini for European Commission DG for Information, Society and Media. Windrum, P., 2008, Innovation and entrepreneurship in public services. In Innovation in Public Sector Services Entrepreneurship, Creativity and Management, edited by P. Windrum and P. Koch. Cheltenham, UK; Northhampton, USA: Edward Elgar. Yoon, 2006, Government Innovation Index (GII): Concept, Development and Application, Powerpoint presentation.

Page 47