Public Health (2005) 119, 239–245

MINI-SYMPOSIUM — PUBLIC HEALTH OBSERVATORIES

Public health indicators
Julian Flowers*, Pamela Hall, David Pencheon

Eastern Region Public Health Observatory, Institute of Public Health,

Robinson Way, Cambridge, CB2 2SR, UK

KEYWORDS Summary An indicator is a measure used to express the behaviour of a

Indicator; system or part of a system. Indicators are widely used in the public sector, and
Performance there is widespread use of indicators for performance management of public
management health.
In this paper, we define some of the terms used in relation to indicators. We
outline some of the most important issues around selection and construction of
indicators, and we include criteria for developing or assessing indicators. Use of
inappropriate indicators can be misleading and can result in negative con-
sequences for public health, and we point out the potential for pitfalls. Some
misinterpretation of indicators could be avoided by use of better methods of
presentation than the familiar league table. We use the example of a funnel plot
to show a method of summarising indicator data which avoids ranking, and allows
rapid identification of areas functioning outside normal limits.
Q 2005 The Royal Institute of Public Health. Published by Elsevier Ltd. All rights
reserved.

Introduction In this article, we examine three issues:

All health services and systems are increasingly † What makes a public health indicator, and what
subject to scrutiny and public health is no makes it ‘good enough’?
exception. Such scrutiny often relies on summary † Some pitfalls in public health indicators; and
† Some of the ways of presenting indicators.
measures of the system to indicate ‘performance’.
As a result, there is increasingly a culture of
management by measurement, and there is bur-
geoning interest in the development and use of
indicators to guide and support public health
Background
practice. As Deming said, ‘what gets measured
For the purposes of this paper, we define an
gets done’.
indicator as a summary and synthesised measure
that indicates how well a system might be perform-
ing. A useful definition of a public health indicator is
* Corresponding author. Tel.: C44 1223 330348; fax: C44 1223
330345.
‘a summary statistic which is directly related to and
E-mail address: julian.flowers@rdd-phru.cam.ac.uk which facilitates concise, comprehensive, and
(J. Flowers). balanced judgments about the condition of a major
0033-3506/$ - see front matter Q 2005 The Royal Institute of Public Health. Published by Elsevier Ltd. All rights reserved.
doi:10.1016/j.puhe.2005.01.003
240 J. Flowers et al.

aspect of health, or progress towards a healthier † Partnerships; and

society’1,2.
There is no shortage of indicators—they come
in baskets, scorecards, portfolios and libraries.
Virtually every policy initiative is accompanied by
indicators. Given the breadth of public health,
indicators are rarely considered in isolation. In
England for example, the Chief Medical Officer has
developed, with the Public Health Observatories, a
set of regional indicators covering:
† Population health status;
† Determinants of health and risk factors;
† Public health interventions;
† Services;
† Public health capacity3.
Indicator language
Just as there is a profusion of indicators, there is a
profusion (and confusion) of terms used. We offer
some thoughts, and hopefully clarification, on the
use of some of these terms (see Box 1).
An indicator merely indicates—it is a measure of
interest which is used to indicate some concept,
construct or process that we cannot measure
directly. Its value often derives from the context
in which it is used.

Box 1
Data The fundamental components which are processed in many ways, including
collation, contextualisation and interpretation, to reveal or create information/
knowledge.
Information/knowledge/ Processed and accurate data—collated, linked, contextualised, interpreted and
intelligence presented/disseminated in sufficient time to enable a decision-maker to take
whatever action is required.
Research evidence The results of systematic studies that can give important generalisable knowledge on
what causes certain health related outcomes, and knowledge on what works in
diagnosing and managing important clinical and public health problems.
Practice What organisations, teams and individuals actually do.
Guidance Knowledge on best practice that has been published or issued by an authoritative
person or body. It should be assembled by a systematic and explicit process. It is simple
to understand and possible to implement and helps practitioners to make the best
possible decisions on behalf of their patients and the public.
Criteriona An important area of health and quality that can be measured in order to understand
the system better.a
Standarda The level which a criterion should not fall, or should exceed, or aspire below.a
Indicator A summary and synthesised measure that indicates how well a system might be
performing. An indicator merely indicates—it is a measure of interest which is used to
indicate some concept, construct or process that we cannot measure directly. Levels
that exceed or fall short of the expected or desirable level are worthy of further
investigation.
Target A desired change in the value of an indicator over a given period of time. Aspirational
level (e.g. of an indicators) at which to aim. The direction, and rate of progress
(sometimes called the trajectory) are both as important as actually meeting the
target. Targets should focus on direction not just destination.
Surveillance Systematic and continuous collection, analysis and dissemination, that enables
important changes in incidence and prevalence of determinants and diseases, to be
detected sensitively and addressed rapidly and efficiently. It should have the capacity
to detect the unexpected through alerts and alarms.
Monitoring Periodic performance and analysis of routine measurements, aimed at longer
term trends in determinants and health status. Monitoring tends to be
reactive, post hoc, and tracks issues that are already known through alerts and
alarms.
Index A summary measure which often has a built-in standard or baseline. It is a relative
measure usually measured on a ratio scale.
Trajectory The periodic change required in the value of an indicator in order to achieve a target
a
Alternatively, a standard is something set by authority, to be attained or aspired to; and a criterion is a method or measure by
which adherence to (or achievement of) a standard will be measured. Each standard can have one or several criteria that will be
used to assess to what extent the standard has been achieved, or exceeded.
Mini-symposium
What makes a good public health
indicator?
Most policy areas have indicators. Ideally, the
development and construction of indicators
would follow a rigorous and scientific process,
but often there are political and other constraints4.
Surprisingly little work has been done on what makes
a good indicator in public health. For example, a
recent review of the literature identified 18 popu-
lation based indexes which combined mortality and
morbidity data and were largely derived from
routinely available or easily accessible data5. The
indexes were developed to facilitate comparisons
between the health of populations (defined
Table 1 Twenty questions to ask of a proposed indicator.
Title, Rationale, Validity
Declarative title?
Description (with definitions where appropriate)?
From which organisation/unit does this indicator origi-
nate?a
What is the broad policy area to which you would allocate
this indicator to?
Rationale with evidence?
What is this indicator purporting to indicate (e.g. HbA1C-
control; Retinopathy-control/quality of service)?
Face validity?b
Construct validity?c
Data
Numerator (N) and denominator (D) and comparator
(each has a source, or means of data collection and
quality assurance (provenance))?
Routine or special collection?
What is the unit of analysis? What level (place,
institution, person.) is being analysed?
Time? (frequency)
Qualitative/quantitative?
Disease classification (be specific about type of diabetes)?
If calculated—which method?
Miscellaneous
Strengths?
Weaknesses?
Risk of gaming and perverse incentives?
How is this likely to influence (improve) practice/
behaviour?
Is the indicator mainly associated with structure, process
or outcome?
a
Don’t invent new indicators, unless there is a very compelling reason to do so.
b
Does the indicator appear to measure what it intends to in practice—is it intuitive?
c
Is there a sound theoretical basis for constructing the indicator in this way?
241
geographically or using other dimensions such as
ethnicity), but few had been validated.
Drawing on this and the efforts of many other
commentators in this area, we describe in Table 1
a framework which could be used to help develop
indicators or to assess fitness for purpose of existing
indicators2,4,6–10.
Assessing validity
Relevance
There should be a clear rationale for developing an
indicator, which includes a link to current policy.
Good indicators should be timely, and there should
be evidence that the indicator is a plausible proxy
Example
Record of glycosylated haemoglobin (HbA1C) level
% of patients with diabetes who have had a record of
HbA1C in the previous 15 months
GMS new contract dataset28
Clinical and cost effectiveness
(a) HbA1C levels related to outcome
(b) practices which keep good records are practices
are associated with good outcomes
Organisation and commitment of the practice.
Yes
Yes
N, number of patients with diabetes who have had a
record of HbA1C in the previous 15 months.
D, total number of people in practice with diabetes
Routine
General practice
Annual
Quantitative
e.g. Type 1 diabetes as defined by.
Simple division
Good evidence base and high face validity
Risk of denominator under-estimate (see below)
To keep the value of this indicator high, there is an
incentive to under record the denominator (i.e.
total number of diabetics)
Improved records lead to better clinical practice
and to better prevalence estimates
Process.
242
for the underlying measure of interest. For
example, life expectancy is a plausible indicator
of population health.
Validity
An indicator should have face validity in that it
should be likely to measure what it purports to
measure. For example, the Index of Multiple
Deprivation (IMD) 2000 is widely used as an
indicator of the level of deprivation in small areas
in the UK11. It correlates well with previous
measures such as Townsend or Carstairs and
identifies as deprived areas that people accept as
being deprived.
Indicators should also have construct validity.
Many indicators are complex composite measures
combing several elements into a single figure. The
elements should be plausible and the composition
of the indicator should make sense. For example,
the IMD 2000 includes several domains, each of
which contains measures which people would
accept as representing a form of deprivation.
Technical criteria for indicators
Behaviour
Indicators should be ‘well-behaved’, that is, a
change in the value of the indicator should
be interpretable, and, for composite indicators,
the indicator value should change in an appropriate
direction if the underlying elements change.
Murray et al. have recently provided some useful
guidance for summary population health measures
like life expectancy10. In a nutshell, any measure
which is a summary of age-specific rates should
change appropriately if any of the underlying rates
change. For example, if a summary measure of
health is based on age-specific rates (e.g. mortality
or morbidity) and a rate is lower in any age-group,
providing everything else remains the same the
summary measure should improve; if the age-
specific rate of disease frequency or severity is
higher in any age-group, providing everything
else remains the same the summary measure
should worsen. Note that this is not always the
case with some commonly used summary measures
such as Standardised Mortality Rates9.
Clear specification
Clear and comprehensive information should be
available about the construction of an indicator,
including details of numerator and denominator
data and the calculations necessary to derive the
indicator value.
J. Flowers et al.
Repeatability
Most indicators are tracked over a time period. It is
important to consider changes in the components of
the indicator, including changes in collection or
coding of the data underlying indicators. If a change
is significant, it may be necessary to revise the
indicator. For example the shift from use of the
ninth revision of the International Classification of
Diseases (ICD-9) to the tenth revision (ICD-10)
resulted in significant alterations to coding of
clinical data, which has affected monitoring of
mortality data over time12,13.
Construction and deconstruction
For complex measures, e.g. life expectancy, it is
valuable to be able to deconstruct the measure into
its components, e.g. cause-specific or age-specific
death rates. This allows consideration of the source
of any variation, and enables interventions to be
targeted at specific causes. There are no population
interventions for increasing population life expect-
ancy, but there are many interventions targeted at
specific causes of mortality.
Feasibility
Indicators should usually be constructed using
routinely collected data. It is important to consider
the availability and quality of both numerator and
denominator data. For example, it is currently
difficult to produce primary care trust (PCT) level
indicators because recent PCT population estimates
are not available.
Calculations should be transparent; ideally, given
the appropriate data it should be possible to
reconstruct an indicator and derive the same values.
Consequences of indicators
The presence of indicators may have consequences
on the behaviour of a system. These may be
welcome, for example improvement in data collec-
tion. However, other unintended consequences
may be detrimental to the system.
Gaming
This is an issue to do with pursuit of targets, which
can cause an unwanted behaviour change. People
may ‘fiddle the figures’ in order to avoid punishment
or embarrassment, e.g. suspending people from
waiting lists so that they don’t count as ‘breaches’ of
some standard. For example, if general practices are
to be rewarded for completeness of, say, measuring
blood pressure in diabetics, they may create a
register only including those diabetics known to
have had their blood pressure measured, and
Mini-symposium
therefore appear to be doing exceptionally well. The
Audit Commission reviewed 41 NHS acute trusts
thought to be at greater risk of misreporting waiting
list information.14 In three trusts, there was
evidence of deliberate misreporting, and in 19 trusts
reporting errors were identified.
Balance
Ideally indicators should be balanced; they should
not focus attention on one part of a system to the
exclusion of the rest. Improving performance in one
area may have a negative impact in other areas. An
indicator relating to the proportion of people
waiting more than two weeks for an outpatient
appointment for suspected cancer may result in
increased waiting times for treatment of people
with diagnosed cancer, as trusts strive to improve
their performance against the indicator.
Health warnings for public health
indicators
Indicators are not without their pitfalls. We
describe here some of the factors to consider
when interpreting indicators.
Will Rogers phenomenon
This is the paradox observed when moving an item
from one set to another moves the average values
of both sets in the same direction.
‘When the Okies left Oklahoma and moved to
California, they raised the average intelligence
level in both states.’14
Figure 1 Regression to the mean. Areas with an increase in directly age-standardised mortality rates (DSRs) for circulatory
disease between the first pair of years (rate ratio O1) tend to show a reduction in DSRs between the second pair of years (rate
ratio !1), and vice versa. Scatter plots show directly age-standardised mortality rates for circulatory disease in males aged
under 75, primary care trusts in England 1998–2001. Source: compendium of clinical and health indicators 200225.
243
The Will Rogers phenomenon has largely been
described in cancer survival where ‘migration’ is
between cancer stages, but it was originally
described in relation to true population
migration15–17. We know little about the effects of
migration on population health measures. For
example, if the healthiest people in an unhealthy
area move to a much healthier area where they
become the least healthy people, the effect on the
average health in both populations might be to
reduce it. If, however, people move from small,
unhealthy populations to populous, healthy ones,
the health gap between the populations might
increase because the average health in the former
population will fall but there may little noticeable
effect in the latter. There is some evidence for this
in the changes in socioeconomic gradients18–20.
Regression to the mean
Regression to the mean is a very common problem,
where a measurement yielding an extreme value on
one occasion tends to yield a value closer to the
average on the next occasion without anything else
having changed21–24. This particularly affects indi-
cators where year on year change is used to indicate
an underlying trend.
In Fig. 1, we can use PCT data on circulatory
disease mortality to illustrate regression to the
mean25. If we compare the change in circulatory
disease mortality between 1998 and 1999, and
between 1999 and 2000, using the ratios of the
directly age-standardised rates (DSRs) for the two
pairs of years, we produce the graph on the left of.
There is a negative correlation between the two
244 J. Flowers et al.

ratios. If mortality increases between one pair of
years, it tends to fall in the next pair of years, and
vice versa. This phenomenon can be demonstrated
using different data sets; for example, if we plot
data from 1999/2000 and 2000/1, we produce a
similar graph (graph on the right of Fig. 1).
League tables
The presentation of indicators is crucial to their
interpretation, and to what extent (appropriate)
action is taken. Indicators are often presented as
ranks or league tables, often using traffic light
coding (green for satisfactory performance, amber
when there is some cause for concern and red for
unsatisfactory performance). Such methods have
serious limitations (e.g. in ranking, someone has to
be worst regardless of how good they are); the
principal flaw is the implicit assumption that there
is a performance difference between organisations.
Analysing and ranking results on the basis of this
underlying assumption inevitably leads to organis-
ations being compared with each other.
Measuring uncertainty
Ranking, such as in league tables, fails to allow for the
variation associated with measurement that occurs
even in the most stable systems. It is good practice to
include a measure of uncertainty in league tables, for
example presenting confidence intervals for values or
ranks, but this doesn’t solve the problem:
† There is a natural tendency to focus on the rank of
an organisation in a table and ignore the confi-
dence interval.
† Comparison of multiple confidence intervals is a
form of multiple significance testing. Remember
that on average one in every 20 measurements will
fall outside the 95% confidence intervals.
† Confidence intervals are not readily understood
by everyone that might wish to use such data.
It is much better to assess indicators against a
fixed baseline, or to estimate the underlying trend (if
there are sufficient data points), or to create
a control chart, which takes into account the
uncertainty in estimating annual percentage change.
Better presentation methods use techniques such
as ‘statistical process control’. This can be used to
distinguish between those parts of the system that are
operating within normal limits and those parts which
show greater than expected variation, for example
using control charts and funnel plots26,27. Such
methods combine the two most important features

Figure 2 Funnel plot showing 2000–2001 data from Fig. 1 but with rate ratio plotted against the average number of
events over the time period. The dotted lines represent 95 and 99.8% control limits; the solid horizontal line represents
‘no change’, i.e. a rate ratio of 1. Only one area lies above the upper 95% limit indicating probable genuine increase in
rate, and a few lie below the lower 95% limit showing probable genuine decrease in rate. No area gives cause for alarm,
i.e. none lie outside the upper or lower control limits. Source: compendium of clinical and health indicators 200225.
Mini-symposium
of good data presentation: valid construction and
intuitive display. In Fig. 2, we use a funnel plot to
present PCT data on circulatory disease mortality.
Conclusions
Just as better understanding of research methods in
health care and improved understanding in the
consumers of these methods has led to both better
reporting and better quality research, we believe that
a similar approach should be applied to the use and
appreciation of data and indicators in public health.
Attention to the criteria that we have outlined, and to
presentation, may help us to improve our use of
indicators, develop new and perhaps more appro-
priate indicators and find new ways of communicating
the meaning of indicators. There are pitfalls in using
public health indicators, and although this is by no
means an exhaustive review of the subject, it may
serve to remind public health practitioners and wider
audiences of some of the key points to consider when
proposing, preparing and interpreting indicators.
References
1. Deming WE. The new economics for industry, government,
education. Cambridge: Massachusetts Institute of Technology
Center for Advanced Educational Services; 1994.
2. Mathers C. Framework for the development of national
public health indicators. A paper presented to the national
public health information working group meeting of May
1998;1999.
3. Bailey K, Flowers J, Streather M, Wilkinson J. Indications of
public health in the English Regions. Stockton-on-Tees:
Association of Public Health Observatories; 2003.
4. Bird SM, Cox D, Farewell VT, Harvey G, Holt T, Smith PC.
Performance Indicators: Good, Bad, and Ugly. J Royal Stat-
istical Society: Series A (Statistics in Society) 2005;168:1–27.
5. Kaltenthaler E, Maheswaran R, Beverley C. Population-based
health indexes: a systematic review. Health Policy 2004;68:
245–55.
6. Mathers C, Schofield D. Development of national public health
indicators. A paper presented to the national public health
information working group meeting of December 1997 1999.
7. Murray CJL, Ezzati M, Lopez AD, Rodgers A, Vander Hoorn S.
Comparative quantification of health risks conceptual frame-
work and methodological issues. Popul Health Metr 2003;1:1.
8. Gakidou E, Murray CJL, Frenk J. A framework for measuring
health inequality. In: Murray CJL, Evans DB, editors. Health
systems performance assessment. Debates, methods, and
empiricism. Geneva: World Health Organization; 2003.
9. Mathers CD, Salomon JA, Murray CJL, Lopez AD. Alternative
summary measures of average population health. In:
245
Murray CJL, Evans DB, editors. Health systems performance
assessment. Debates, methods, and empiricism. Geneva:
World Health Organization; 2003.
10. Murray CJL, Mathers CD, Salomon JA. Towards evidence-based
public health. In: Murray CJL, Evans DB, editors. Health
systems performance assessment. Debates, methods, and
empiricism. Geneva: World Health Organization; 2003.
11. Department of the Environment, Transport and the Regions.
Indices of Deprivation 2000. London: DETR; 2000.
12. World Health Organization. International Classification of
Diseases, ninth revision. Geneva: WHO; 1977-8.
13. World Health Organization. International Statistical Classifi-
cation of Diseases and Related Health Problems, tenth
revision. Geneva: WHO; 1992-4.
14. Audit Commission. Waiting list accuracy. Assessing the
accuracy of waiting list information in NHS hospitals in
England. London: Audit Commission; 2003.
15. Wikipedia-The free encyclopedia. Will Rogers phenomenon.
Available at from URL: http://en.wikipedia.org/wiki/Will_-
Rogers_phenomenon [accessed 20 September 2004].
16. Feinstein AR, Sosin DM, Wells CK. The Will Rogers phenom-
enon. Stage migration and new diagnostic techniques as a
source of misleading statistics for survival in cancer. N Engl
J Med 1985;312:1604–8.
17. Bodner BE. Will Rogers and gastric carcinoma. Arch Surg 1988;
123:1023–4.
18. Dent DM. Improving cancer survival results by artefact-Will
Rogers and the stage migration phenomenon. S Afr Med J
1996;86:645.
19. Boyle P, Norman P, Rees P. Changing places. Do changes in
the relative deprivation of areas influence limiting long-
term illness and mortality among non-migrant people
living in non-deprived households? Soc Sci Med 2004;58:
2459–71.
20. O’Reilly D, Stevenson M. Selective migration from deprived
areas in Northern Ireland and the spatial distribution of
inequalities: implications for monitoring health and inequal-
ities in health. Soc Sci Med 2003;57:1455–62.
21. Bland JM, Altman DG. Statistics notes: some examples of
regression towards the mean. BMJ 1994;309:780.
22. Bland JM, Altman DG. Statistic notes: regression towards the
mean. BMJ 1994;308:1499.
23. Avery AJ, Rodgers S, Heron T, Crombie R, Whynes D,
Pringle M, et al. A prescription for improvement? An
observational study to identify how general practices vary in
their growth in prescribing costs. Commentary: beware
regression to the mean BMJ 2000;321:276–81.
24. Morton V, Torgerson DJ. Effect of regression to the mean on
decision making in health care. BMJ 2003;326:1083–4.
25. Department of Health. Compendium of clinical and health
indicators 2002. London: National Centre for Health Out-
comes Development, London School of Hygiene and Tropical
Medicine; 2003.
26. Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol,
Shipman, and clinical governance: Shewart’s forgotten
lessons. Lancet 2001;357:463–7.
27. Battersby J, Flowers J. Presenting performance indicators:
alternative approaches. INphoRM 2004; 4.
28. Department of Health. Quality and outcomes framework-
Guidance. London: DH; 2003. [Indicator ref. DM 5].