Journal of Operations Management 21 (2003) 475488

Discipline note
Survey research in operations management: historical analyses
M. Johnny Rungtusanatham a, , Thomas Y. Choi b,1 , David G. Hollingworth c,2 ,
Zhaohui Wu d,3 , Cipriano Forza e,4
a Department of Management, Arizona State University, P.O. Box 874006, Tempe, AZ 85287, USA
b Departments of Management and Supply Chain Management, Arizona State University, P.O. Box 874006, Tempe, AZ 85287, USA
c Lally School of Management and Technology, Rensselaer Polytechnic Institute, 110 Street 8th, Troy, NY 12180-3590, USA
d Department of Supply Chain Management, Arizona State University, P.O. Box 874006, Tempe, AZ 85287, USA
e Dipartimento di Tecnica e Gestione Dei Sistemi Industriali, Universit di Padova, Stradella S. Nicola 3, 36100 Vicenza, Italy

Received 5 July 2001; accepted 18 December 2002

Abstract
Our paper provides a comprehensive assessment of 285 survey research articles in operations management (OM), published
between 1980 and 2000. Six OM journals are included in this study; they are, in alphabetical order: Decision Sciences (DS),
International Journal of Operations & Production Management (IJOPM), International Journal of Production Research
(IJPR), Journal of Operations Management (JOM), Management Science (MS), and Production and Operations Management
(POM). In this paper, we reflect upon the state and evolution of survey research in the OM discipline across a 21-year time
span and the contribution of OM journals that have published these studies. Major changes have occurred in the last 5 years of
our sampling period, and two topics stood out as showing fastest ascendancy to prominenceoperations strategy and supply
chain management. Furthermore, over the years, the Journal of Operations Management appears to have been publishing
more survey research articles and a greater variety of OM topics compared to the other five journals in our study.
2003 Elsevier Science B.V. All rights reserved.
Keywords: Survey research; OM history; Methodology

1. Introduction velopment of the OM discipline (e.g. Hill et al.,

Operations management (OM) scholars often point
to 1980 as a critical junction in the modern de-
Corresponding author. Tel.: +1-480-727-6268;
fax: +1-480-965-8314.
E-mail addresses: j.rungtusanatham@asu.edu
(M.J. Rungtusanatham), thomas.choi@asu.edu (T.Y. Choi),
hollid@rpi.edu (D.G. Hollingworth),
zhaohui.wu@asu.edu (Z. Wu), forza@gest.unipd.it (C. Forza).
1 Tel.: +1-480-965-6135; fax: +1-480-965-8314.
2 Tel.: +1-518-276-6445; fax: +1-518-276-8661.
3 Tel.: +1-480-727-6191; fax: +1-480-965-8314.
4 Tel.: +390-444-998-731; fax: +390-444-998-888.
1988/1989; Filippini, 1997). Before 1980, core is-
sues and problems in OM had traditionally been
addressed via modeling-based research using either
optimization or simulation methodologies, almost to
the point of exclusivity of other research approaches.
However, in 1980, the OM discipline began to un-
dergo fundamental changes that had resulted directly
from self-assessments of research needs and agendas
(Buffa, 1980; Chase, 1980; Miller and Graham, 1981;
Sullivan, 1982; Voss, 1984). Many scholars recom-
mended that OM researchers undertake research ap-
proaches frequented by organizational behavior and
marketing specialists (Sullivan, 1982, p. 214) in order

0272-6963/03/$ see front matter 2003 Elsevier Science B.V. All rights reserved.
doi:10.1016/S0272-6963(03)00020-2
476 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488
to study management-oriented (Buffa, 1980) and more
macro-oriented (Chase, 1980) OM issues and prob-
lems. These realizations eventually led to a conclusion
that the scope of [OM] can not be captured and ex-
plained in its entirety by purely deductive tools such
as mathematics and its extensions such as operations
research or statistics (Swamidass, 1991, p. 794).
Since 1980, the OM discipline has witnessed in-
creased deployment of empirical research designs,
particularly survey research, to understand better
such issues as manufacturing strategy, manufactur-
ing flexibility, just-in-time manufacturing, quality
management, etc. In general, OM researchers, in
light of their traditional training in optimization-
and simulation-based research methodologies, have
demonstrated remarkable progress in comprehending
the complexities of designing and executing empirical
research. Rungtusanatham (1998, p. 10) commented
that this progress is evidenced not only by the quan-
tity, but also the quality and sophistication of the
research endeavors that have been completed.
Our paper seeks to provide a comprehensive assess-
ment of OM survey research. We begin by reviewing
other evaluative articles similar to ours and demon-
strate how we build on these previous works. After dis-
cussing methodological issues, we present the results
of our analyses. Our results will delineate the historical
trends of OM survey research, particularly in the last 5
years of our sampling period, as well as the contribu-
tions that different OM journals have made in develop-
ing and promoting survey research. We end with a gen-
eral discussion and implications for future research.
2. Literature review
In 1997, the International Journal of Operations &
Production Management (IJOPM) published a collec-
tion of articles on survey research. One of the papers
in that issue (i.e. Filippini, 1997) traced the evolution
of OM topics and research approaches over the last 2
decades by analyzing the 244 papers published in the
Proceedings of the Annual Meeting of the Decision
Sciences Institute, 1996, in terms of research content
and methods. The insights were compared to other
insights developed from similar analyses of the 1986
and 1987 proceedings of the same conference (see
Amoako-Gyampah and Meredith, 1989). In his con-
clusions, Filippini brought attention to the need for im-
proving the quality of survey research in OM. In par-
ticular, he encouraged the development of reliable and
valid measures and the incorporation of multiple vari-
ables. In this same special issue, Collins and Cordon
(1997) sought to identify and discuss methodological
issues surrounding the design and administration of
large-scale surveys. Choosing manufacturing strategy
as their content focus, Collins and Cordon com-
pared and contrasted the design and execution of
two completed survey-based research studies that led
to insights concerning sample selection, respondent
preparation, bias, etc.
Malhotra and Grover (1998), seeking to provide
a normative perspective on what constitutes good
survey research, developed a list of 17 ideal survey re-
search attributes. These attributes touched upon issues
such as unit of analysis, triangulation of data sources,
psychometric assessments of multiple-item measure-
ment scales, sampling frame and strategies, response
rates and bias, and substantive analytical approaches
and statistical power. Malhotra and Grover then ap-
plied these 17 attributes to evaluate 25 survey-based
OM papers from four journals between 1990 and
1995. This evaluation led the authors to suggest the
need for OM researchers to build on existing research,
the importance of theory-driven survey research, the
use of triangulation, the need to conduct formal as-
sessments of reliability and construct validity, the use
of existing measures, the application of confirmatory
methods for data analysis, etc.
Finally, Hensley (1999) focused specifically on
the development and use of reliable and valid mea-
surement scales in OM research. Six studies were
reviewed in terms of the approaches to the develop-
ment and validation of multiple-item measurement
scales. Strengths and weaknesses of these six stud-
ies were identified and discussed in order to help
future researchers better understand the various is-
sues involved in creating and/or adapting existing
measurement scales for OM research.
While the fundamental purpose of our paper par-
allels that of these similarly published papers, our
study brings in a decidedly historical perspective.
For instance, Malhotra and Grover (1998) focused
on 25 published survey research studies in OM be-
tween 1990 and 1995. Collins and Cordon (1997)
compared and contrasted two survey research studies
 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488
on manufacturing strategy. Filippini (1997) examined
articles from proceedings of an annual conference,
and Hensley (1999) reviewed and discussed six pa-
pers that reported narrowly on the development of
multiple-item measurement scales.
In order to build on these studies and to frame the
study in a more historical context, we have expanded
the number of articles considered, as well as the time
period under analysis. We have included a more ex-
haustive set of survey research articles published (285
articles) and have expanded the journal list to six ma-
jor journals. Furthermore, we have chosen to cover a
span of 21 years between 1980 and 2000. As a way
of building on past studies, we repeat some of the
same analyses that earlier scholars have conducted
(e.g. measurement issues, topical issues, data analysis
approaches, etc.), but we do that in an expanded con-
text. Particularly, we set out to see what changes have
taken place since the work of Malhotra and Grover
(1998), one that is most similar to our work, but whose
sampling period ended in 1995. In addition, we try to
highlight different contributions that different journals
have made in the evolution and development of OM
survey research.
3. Methodology
In order to fulfill our purpose of providing a compre-
hensive historical perspective on OM survey research,
we address issues of time horizon, journal selection,
article selection, coding, and analysis.
3.1. Time horizon
We chose to anchor our assessment period between
1980 and 2000, a 21-year time horizon. We chose 1980
as the starting point since the modern development of
the OM discipline can be traced to the encouragements
of such scholars as Buffa (1980) and Chase (1980).
We chose the year 2000 as the end point because it
extends prior reviews by 5 years and has symbolic
meaning as the ending of the century.
3.2. Journal selection
We looked for all OM journals that have been
known to publish empirical studies, in particular
477
survey papers. We first selected Decision Sciences
(DS), Journal of Operations Management (JOM), and
Management Science (MS) because previous studies
have consistently ranked these journals in the upper
echelon. Saladin (1985), for example, ranked JOM,
MS, and DS in that order with respect to quality of
journal and appropriateness. Barman et al. (1991)
classified these three journals in the first three of the
six groupings in terms of relevance and in the first
of the five groupings in terms of quality. In Vokurka
(1996), MS, DS, and JOM were ranked first, second,
and fifth, respectively, out of 25 journals. Goh et al.
(1997) considered these three journals to be in the
Elite grouping. Soteriou et al. (1998) reported that
European OM scholars placed JOM and MS in the
top five groupings in terms of both relevance and
quality.
We included three other journals in our review and
evaluationIJOPM, International Journal of Pro-
duction Research (IJPR), Production and Operations
Management (POM). In Goh et al. (1997), IJOPM
and IJPR had the same Elite status as DS, JOM,
and MS. For European OM scholars, both IJOPM
and IJPR placed in the top five groupings in terms
of both relevance and quality (Soteriou et al., 1998).
POM is included because it is generally regarded as
an important emerging journal for OM research; for
instance, in a recent replication study, Barman et al.
(2001) ranked POM as second after JOM in terms of
relevance in OM research.
3.3. Article selection
We applied two criteria to select articles from all
potential articles published in the six journals noted
above. First, the selected articles should address a
problem or an issue that falls under the OM research
agenda, especially since DS and MS also publish re-
search that falls outside of OM (e.g. finance, informa-
tion systems, and organizational theory). Second, the
selected articles should have applied a pen-and-paper
mailed survey as the primary, not necessarily the only,
data collection approach. We wanted to avoid introduc-
ing potential confounding factors by considering dif-
ferent survey methods in one study (e.g. pen-and-paper
mailed study versus in-person survey or telephone
survey), and we believed that the pen-and-paper
mailed survey was the most widely adopted data
478 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488
collection methodology in survey research conducted
in OM.
We also excluded several articles that did not con-
tain sufficient information about four critical aspects
in the execution of the survey methodology (i.e. dis-
cussion of sampling strategy, specification of the unit
of analysis, assessment of measurement quality, and
assessment of common methods/source variance). Ad-
ditionally, we also did not consider a number of arti-
cles that merely described the extent of usage of OM
concepts and techniques, background descriptive char-
acteristics of production managers, or characteristics
of manufacturing in a particular industrial sector in a
country.
3.4. Coding
We coded all 285 OM survey articles to obtain in-
formation regarding several evaluation dimensions
journal outlet, year of publication, main topic of
article, purpose of article (e.g. descriptive versus rela-
tional), sampling approach, unit of analysis, psycho-
metric analysis (i.e. reliability and construct validity),
and common methods/source bias. The working defi-
nitions that were used to code selected articles on each
of the evaluation dimensions are given in Appendix A.
The coding was completed on an individual basis,
such that one member of the research team would
be responsible for at least one journal. One individ-
ual research team member then double-checked the
data coding for the entire set of articles that were
included in our study, correcting for missing or in-
complete information. We conducted random checks
for inter-rater reliability, and we typically obtained
about 85% agreement, which gave us reasonable as-
surance that we were being consistent in our coding
activities.
3.5. Analysis
Our purpose in this paper, being descriptive and
inductive in nature, is not conducive to statistical
methodologies for deductive hypothesis testing. In-
stead, we engaged in trend and pattern analyses in or-
der to shed greater understanding on the development
and evolution of survey research in OM and to iden-
tify potential areas for improvement. We present these
analytical results in the form of tables and graphs.
4. Trend and pattern analytical results
4.1. Survey research growth in OM
We have tabulated the counts of OM survey articles
published in the six selected journals since 1980 in
Table 1. Based on the total number of OM survey arti-
cles published year-to-year across the six journals, we
can observe confidently that there has been an increase
in the quantity of OM survey research since 1980.
Comparing the percentages of survey papers to the
total number of articles published in each journal,
JOM, at 14.4%, clearly appears to be the gatekeeper
of publishing survey papers. In the next tier, IJOPM
and POM have been publishing about the same per-
centages at 9.9 and 8.6%, respectively. What is no-
table, however, is that the total percentage of survey
articles across all journals remains at a mere 3.3%.
However, if we were to examine the percentages since
1992 when POM came into existence, this percentage
increased to 5.1%. Also, JOMs role as the gatekeeper
of publishing survey articles becomes even more pro-
nounced in this time period (27.9% compared to other
journals).
4.2. OM survey research focus
In order to code the primary research focus of
published OM survey articles, we combined the topic
listings published in the submission guidelines for
authors in DS, IJOPM, and JOM. After removing re-
dundancies, we identified five major topic areas as fol-
lows: Just-in-Time (JIT), Management of Technology
(MoT), Operations Strategy (OS), Quality Manage-
ment (QM), and Supply Chain Management (SCM).
Three of these five topic areasnamely, Quality, Op-
erations Strategy, and Management of Technology
have previously been identified in a Delphi study
of top OM research issues (Malhotra et al., 1994).
Appendix B lists examples of subtopics in each of the
five major topic areas.
We then sought to classify the 285 OM survey ar-
ticles into one of the five research topic areas. In in-
stances, where published articles focused on more than
one main topic area, we assigned equal weights to the
different topic areas such that the sum of the assign-
ments did not exceed 1. For example, for Boyer et al.
(1997), we assigned 0.5 to OS and another 0.5 to MoT,
 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488 479

Table 1
Frequency of OM survey articles published by journal and by year
Year Journals

DS IJOPM IJPR JOM MS POM Total survey articles

1980 0 (51) 0 (10) 1 (60) 0 (19) 0 (111) NA 1 (251)

1981 0 (65) 0 (10) 2 (64) 2 (18) 0 (112) NA 4 (269)
1982 0 (55) 1 (12) 1 (58) 2 (17) 0 (122) NA 4 (264)
1983 0 (44) 1 (17) 0 (72) 1 (26) 0 (116) NA 2 (275)
1984 0 (37) 1 (24) 1 (78) 2 (29) 0 (124) NA 4 (292)
1985 0 (32) 1 (24) 0 (87) 1 (40) 0 (134) NA 2 (317)
1986 0 (38) 1 (32) 0 (105) 0 (34) 0 (135) NA 1 (344)
1987 0 (45) 5 (27) 4 (127) 1 (17) 0 (139) NA 10 (355)
1988 0 (61) 2 (40) 1 (132) 0 (20) 0 (125) NA 3 (378)
1989 1 (60) 4 (46) 2 (138) 0 (24) 0 (107) NA 7 (375)
1990 0 (61) 3 (48) 2 (157) 1 (24) 0 (110) NA 6 (400)
1991 0 (78) 7 (65) 1 (164) 4 (25) 1 (118) NA 13 (450)
1992 1 (85) 6 (64) 0 (178) 0 (0) 0 (112) 4 (27) 11 (466)
1993 0 (63) 5 (64) 1 (80) 1 (24) 1 (123) 2 (16) 10 (470)
1994 2 (45) 4 (82) 3 (181) 3 (14) 1 (121) 1 (19) 14 (462)
1995 5 (35) 6 (94) 4 (206) 4 (42) 3 (147) 1 (24) 23 (548)
1996 2 (35) 11 (87) 2 (195) 3 (22) 2 (126) 1 (25) 21 (490)
1997 1 (38) 9 (72) 4 (193) 8 (21) 2 (126) 3 (23) 27 (473)
1998 4 (44) 9 (73) 2 (187) 13 (39) 1 (145) 1 (31) 30 (519)
1999 5 (47) 11 (75) 6 (226) 12 (34) 5 (118) 5 (25) 44 (525)
2000 6 (36) 17 (75) 5 (251) 17 (30) 2 (113) 1 (29) 48 (534)
Total OM survey articles 27 104 42 75 18 19 285
All articles published 1054 1041 3039 519 2584 219 8456
Percentage (%) of survey 2.5 9.9 1.3 14.4 0.6 8.6 3.3
articles (19802000)
Percentage (%) of survey 5.1 11.4 2.6 27.9 1.7 8.6 5.1
articles (19922000)
The total number of articles are given in parenthesis.

since the article contained these two foci. In the end, count of 58.5, and JIT shows the lowest at 27.5, fol-
we were able to classify 202.5 out of the 285 OM sur- lowed by SCM at 33.
vey articles into at least one of the five research topic When the cumulative counts of each of the five
areas. As indicated in Table 2, OS shows the highest topics across all journals were plotted, OS and SCM
showed the steepest positively sloped curve (see
Table 2 Fig. 1), indicating comparatively fast growth. JIT
Classifying OM survey research by research focus showed a slowing of publications with the smallest
Research focus Survey articles slope.
across all journals One striking result was obtained when the five top-
Just-in-Time (JIT) 27.5 ics were compared across different journals as shown
Management of Technology (MoT) 40.5 in Table 3. Based on the earlier observation that por-
Operations Strategy (OS) 58.5 trayed JOM as the gatekeeper of publishing the highest
Quality Management (QM) 43
percentages of survey articles in OM, we expected to
Supply Chain Management (SCM) 33
see JOM emerging as the leader in one or more of the
Total OM survey articles classified 202.5 topic areas. However, across all topics, JOM appeared
All OM survey articles published 285
in middle to low positions in terms of rankings. Upon
480 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488

Fig. 1. Total number of survey articles by major OM topic area by year.

further analysis, we discovered that only 54% of sur-
vey papers published in JOM fell into these five major
topic areas, with the remaining percentage of survey
papers falling into other topic areas such as perfor-
mance assessment, international issues, new product
development, time-based management, etc. For DS,
IJPR, IJOPM, and POM, the percentages were 96, 83,
76, and 74%, respectively.
4.3. Purpose of OM survey research
The OM survey articles generally fall into one of
the two categories, descriptive or relational, in terms
of why the survey research was being conducted. By
descriptive, we refer to studies that are designed to
provide a snapshot of the current state of events
related to an OM phenomenon. Generally, these

Table 3
Comparing journal outlets by the research topic areas
Just-in-Time manufacturing Management of Technology Operations Strategy Quality Management Supply Chain
Management
IJOPM (14.9%) IJPR (19%) POM (36%) DS (37%) DS (20.3%)
MS (13.8%) IJOPM (16%) IJOPM (24%) IJPR (27%) IJPR (16.6%)
IJPR (10.7%) DS (12.9%) DS (18.5%) POM (13%) JOM (12%)
POM (10.5%) JOM (12%) JOM (13%) MS (11%) IJOPM (9.6%)
DS (7%) POM (7%) MS (11%) JOM (10.6%) POM (7.8%)
JOM (6%) MS (5.5%) IJPR (9.5%) IJOPM (8.6%) MS (0%)
Note: Percent (%) refers to the percentage of articles in the given topic area to the total number of survey articles published in a particular
journal.
 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488 481

studies do not conduct formal tests of hypotheses, Table 4

other than tests of differences between groups for de- Classifying OM survey research by research purpose and by year
scriptive purposes. Some examples include the Saraph Year Descriptive Relational Both Total by year
et al. (1989) study defining and operationalizing eight
1980 1 0 0 1
critical factors underlying quality management, the
1981 2 1 1 4
Gilbert (1990) study describing JIT implementation 1982 1 2 1 4
in small- to medium-sized manufacturers, and the 1983 1 1 0 2
Vargas and Johnson (1993) study of the operations 1984 4 0 0 4
functions in maquiladoras. 1985 1 0 1 2
1986 1 0 0 1
By relational, we refer to studies that are designed
1987 5 2 3 10
to empirically examine relationships among two or 1988 2 1 0 3
more constructs or variables, either in an exploratory 1989 6 1 0 7
or a confirmatory manner. Studies that fall into this 1990 4 2 0 6
category specify propositions or hypotheses a priori to 1991 5 6 2 13
1992 3 7 1 11
guide subsequent empirical analyses. Examples of OM
1993 3 6 1 10
survey articles in this category include the empirical 1994 6 7 1 14
test of a Deming-based theory of quality management 1995 4 19 0 23
(Anderson et al., 1995), the performance implications 1996 6 11 4 21
of tooling problems in flexible manufacturing systems 1997 2 22 3 27
1998 3 18 9 30
(Perera and Shafaghi, 1995), and the factors affecting
1999 2 39 3 44
job shop dispatching rules (Green and Appel, 1981). 2000 2 44 2 48
Some OM survey articles, in fact, serve both descrip-
Total count 64 189 32 285
tive and relational purposes (e.g. Ahire et al., 1996;
Tan et al., 1998).
Table 4, which tabulates the 285 survey articles
Table 5
according to research purpose, reveals that OM re- Sampling strategies in OM survey research by year
searchers have been increasingly designing relational
Year Secondary Probability Population Total by
studies using survey methodology for data collection.
data sampling study year
During the first 15 years starting with 1980, de-
1980 0 1 0 1
scriptive articles averaged about 3.4 articles per year 1981 0 0 0 0
and relational studies averaged about 2.8 articles per 1982 0 0 0 0
year. However, this trend changed dramatically since 1983 0 0 0 0
1995. Beginning in 1995, while descriptive studies 1984 0 0 1 1
increased only slightly to 5.0 articles per year, rela- 1985 1 0 0 0
1986 0 0 0 0
tional articles leapfrogged from 2.8 to 27.3 articles per 1987 0 0 0 0
year. 1988 1 0 0 1
1989 0 1 2 3
4.4. Sampling strategies 1990 2 1 0 3
1991 2 3 1 6
1992 2 0 0 2
With respect to sampling strategies, we counted 1993 0 2 3 5
152 articles (53%) that provided sufficient informa- 1994 0 4 2 6
tion about the target population, the sampling frame, 1995 7 2 5 14
and/or the sampling process to understand how the fi- 1996 1 4 6 11
nal sample was constructed. These 152 articles can 1997 5 4 10 19
1998 7 1 11 19
be classified into one of the three categories in terms 1999 5 11 10 26
of sampling strategiessecondary data survey papers, 2000 8 10 17 35
probability sampling survey papers, and population
Total count 41 43 68 152
survey papers. As shown in Table 5, 41 articles re-
482 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488
ported empirical results based on a secondary data
source. These articles specified in the methodology
section that the data were provided by a secondary
source, collected by other researchers and for other
purposes (Babbie, 1992). For example, Anderson et al.
(1995, p. 655) stated, the empirical results reported
here are based on an analysis of an available data
source.
A probability sampling approach allows samples to
be created that are representative of the populations of
interest. For instance, Sakakibara et al. (1993) used a
stratified random sample of US and Japanese-owned
plants. Gupta et al. (1991) used a random sample of
manufacturing organizations.
Population surveys occur when the authors admin-
ister surveys not to a sample but to the entire mem-
ber elements within the target population. As such,
there is no need to deploy any sampling approach to
construct a sample of the target population per se. As
shown in Table 5, this appears to be the dominant ap-
proach in OM survey research. For instance, Kim and
Lee (1989) sent questionnaires to all members in the
membership directories of APICS, IIE, and the Com-
puter and Automated Systems Association of SME.
Likewise, Archer and Wesolowsky (1996) sent surveys
to all vehicle owners in the Hamilton, Ont., Canada,
metropolitan areas.
Looking at the general pattern in Table 5, clearly
there has been an upward trend in more recent years
in terms of specifying the sampling strategy adopted
in OM survey research. Most markedly, the year 2000
showed that 72% of the survey articles published
clearly specified the sampling strategy. When the
overall data were partitioned into different journals,
we found that JOM and IJOPM have historically been
most consistent in reporting sampling strategies
JOM at 61% of the total number of survey articles
published and IJOPM at 60%.
4.5. Unit of analysis
Related to the issue of sampling strategies is the
specification of the unit of analysisi.e. what or who
is [being] studied (Babbie, 1992, p. 92). Out of the
285 papers, only 50 OM survey articles clearly iden-
tified and/or stated the unit of analysis underlying the
reported empirical results. A closer look at these 50
papers reveals that all were published in the 1990s,
with 27 appearing during 1998, 1999, and 2000, the
last 3 years of our analytical time horizon.
Among these 50 survey articles, the unit of analysis
has varied considerably, from a manufacturing plant
or factory (e.g. Ahire et al., 1996; Flynn et al., 1994;
Noble, 1997) to a bank (e.g. Roth and van der Velde,
1991) and to a health maintenance organization (e.g.
Heineke, 1995). Additionally, OM articles have ana-
lyzed a machine shop (e.g. Wisner, 1996), a primary
product line (e.g. Safizadeh and Ritzman, 1997), an
assembly line (e.g. Chase and Singhal, 1981), the
individual employee (e.g. Archer and Wesolowsky,
1996), a relationship between buyers and suppli-
ers (e.g. Bozarth et al., 1998; Hartley et al., 1997),
and a business unit (e.g. Narasimhan and Jayaram,
1998).
4.6. Assessments of measurement quality
Measurement quality, or the reliability and valid-
ity of measurement instruments used to collect data,
is a pivotal concern in any research, particularly in
research that seeks to test relationships among con-
structs (Carmines and Zeller, 1979). As such, for the
issue of measurement quality, we considered only the
221 OM survey articles in the review set with a rela-
tional purpose (see Table 4).
4.6.1. Reliability
Of the 221 relational OM survey articles, we ex-
cluded 46 that predominantly analyzed single-item
measures, since the use of single items does not al-
low for an evaluation of internal consistency reliabil-
ity. These 46 OM survey paper, by the way, did not
provide evidence of other forms of reliability. Of the
remaining OM survey articles, only 31 offered no ev-
idence of reliability. Among these 31 were ones that
claimed to have conducted reliability assessment but
reported no results, as well as ones that simply did not
provide any discussion.
Of the OM survey articles that provided evidence
of reliability, 24 offered other reliability assessment
results in place of, or in addition to, Cronbachs .
These 24 OM survey articles commented, for instance,
on inter-rater reliability, Werts et al. (1974) reliability
coefficient, and/or a composite reliability outlined by
Fornell and Larcker (1981). Twelve of these 24 survey
articles appeared in 1999 and 2000, consistent with
 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488
our previous observation concerning the increasing
methodological rigor of OM survey research toward
the end of our 21-year time period.
4.6.2. Validity
Generally speaking, three different types of validity
can be assessed for any measurement instrument
content validity, criterion-related validity, and con-
struct validity. Of the three, construct validity comes
closest to the general definition of validity and is
the primary concern for research seeking to empiri-
cally test relationships among constructs. Therefore,
we consider, in our evaluation of the 221 relational
OM survey articles, whether or not assessments of
construct validity were conducted in the execution
of the survey research. More particularly, we narrow
our focus to the 164 OM survey articles that used
multiple-item measurement scales, since OM survey
articles that do not use multiple-item measurement
scales generally do not address the issue of construct
validity.
Our analyses found that of the 164 published survey
articles in OM, only 78 (or 46%) acknowledged or
reported on the issue of construct validity. However,
47 of these 78 (60%) were published in the last 2 years
of our sampling period (1999 and 2000), validating,
once again the increasing methodological rigor of OM
survey research.
4.7. Common methods/source variance
The potential for inflated empirical relationships
can occur when data for independent variables and
dependent variables have been collected using the
same method or have been provided by the same
single source (Crampton and Wagner, 1994). Either
situation may introduce a response bias that would
exaggerate the magnitude of the empirical relation-
ships. This issue of common methods/source variance
is of particular concern for survey research wherein a
single organizational informant provides answers to
both independent and dependent variables using the
same data collection approach such as a question-
naire.
As with the issue of measurement quality, we only
consider the 221 OM survey articles with a relational
purpose in terms of the issue common methods/source
variance. Of these 221, only a very small percentage,
483
19% to be exact, explicitly considered or evaluated the
potential of common methods/source variance.
From a historical perspective, no article published
before 1992 even raised or brought this issue into dis-
cussion. In fact, 17 of these 19 OM relational sur-
vey articles appeared only in the last 5 years (since
1996) for which we were collecting dataan indi-
cation of a major change in this trend. Bozarth and
Edwards (1997), for example, conducted a Harman
one-factor test to detect for common method variance.
Sriparavastu and Gupta (1997, p. 1232), in discussing
future research directions, urged that data be collected
by other methods in other industries to eliminate
the potential for biased responses. Choi and Eboch
(1998) attempted to mitigate the concern of common
method variance by interpreting convergent validity
and discriminant validity results in terms of the in-
dependence of the constructs. Finally, Klassen and
Angell (1998) explicitly recognized the potential for
single informant bias in their study.
5. Discussion
Although there have been other evaluative studies
published with a similar purpose (e.g. Malhotra and
Grover, 1998), our paper takes a more comprehensive
and historical look at survey research in the OM dis-
cipline. Our study, in contrast to previously published
papers, included a more exhaustive set of survey ar-
ticles (n = 285), considered a larger set of core OM
journals (DS, IJOPM, IJPR, JOM, MS, and POM), and
spanned a longer time period of 21 years from 1980 to
2000. We believed that working with this exhaustive
set of data was necessary to provide a historical per-
spective and an introspective reflection for future OM
survey research. We chose 1980 as the starting point
for data collection because most OM scholars consider
1980 as the watershed year in terms of empirical OM
research.
Our results clearly indicate that we have witnessed
an increasing proliferation of OM survey research
articles published since 1980. What is striking is that
the field has taken a quantum leap since 1995, the
last year in the study of Malhotra and Grover (1998).
It would seem, from looking at the data, that we
have come through a renaissance in designing and
executing survey research during the late 1990s. For
484 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488
instance, the number of articles that explicitly stated
sampling strategies dramatically increased since 1995
(see Table 5), and 27 of the 50 articles that explicitly
articulated the unit of analysis all appeared since 1997
(see Section 4.5). Furthermore, 12 of the 24 articles
that reported reliability assessment results in place
of, or in addition to, Cronbachs , appeared in 1999
and 2000 (see Section 4.6.1); 47 of the 78 articles
that reported construct validity were again published
in 1999 and 2000 (see Section 4.6.2); and 17 of
the 19 articles that addressed the issue of common
methods/source variance appeared since 1996 (see
Section 4.7).
Looking at all six journals across the 21-year time
span (see Table 1), JOM appears to have been most
active in publishing survey research articles14.4%
of all articles published since 1980 in JOM have been
survey researches; since 1992, this percentage has in-
creased to 27.9%. This finding, by itself, may not be
terribly interesting. What becomes interesting, how-
ever, is that, in rank ordering the six journals in terms
of the percentage of survey articles published in each
major topic area (Table 3), JOM ranks in the middle
or near the bottom for each of these five OM topic
area.
One could potentially argue that it would be natu-
ral for other journals that publish articles at a higher
volume (e.g. IJOPM) to dominate the selection of the
major topic areas and, thus, the ranking. However, the
list of major topic areas was derived a priori based on
the key topics listed in these journals (see Section 4.2),
and the rankings were created based on percentages,
not absolute numbers. Therefore, the only conclusion
we can draw is that over the years, JOM has, know-
ingly or unknowingly, been more open and receptive
to experimenting in non-major or -traditional topics.
Only about half of the survey articles published in
JOM belonged to the five major topic areas, whereas,
almost all survey articles published in DS and about
three-quarters in IJOPM, IJPR, and POM fell under
the five major topic areas.
Among these topic areas, operations strategy (OS)
showed the highest count of 58.5 (see Table 2), while
JIT and SCM totaled near the bottom at 27.5 and
33, respectively. Furthermore, during the last 5 years
between 1996 and 2000, whereas, survey articles in
OS and SCM have shown steep inclines, the number
of JIT survey articles has flattened out (see Fig. 1).
Indeed, OS, as a topic area, has been the most dom-
inant throughout the history of OM survey research
and continues to dominate with OM survey research
addressing such macro issues as manufacturing strat-
egy, strategic taxonomy, and service strategy (see
Appendix B). Like OS, SCM, which also addresses
such macro issues as the dynamics between firms and
strategies for managing a group of companies (e.g.
suppliers), has attracted and continues to attract the
attention of OM scholars.
As discussed in Section 4.3, there has also been a
dramatic increase in the number of relational OM sur-
vey studies published since 1995. The average num-
ber of relational OM survey studies had been 3.4
articles per year for the first 15 years of our sam-
pling period, but from 1995 to 2000, this number
grew to 27.3 articles per year. This is encouraging
because relational studies provide the foundation for
building and testing theories and for growth of scien-
tific knowledge in a discipline. However, we need to
keep such encouraging news in perspective, particu-
larly since the average percentage of survey articles
published over the years across all six journals re-
mains considerably low at 3.3%. If empirical studies
were to play a major role in building the foundation
for the field of OM and if pencil-and-paper survey
were to become the primary methodology for obtain-
ing such empirical data, we need to see this percent-
age grow to a much higher level and at a much faster
pace. Although this percentage went up significantly
to 6.7% in the last 5 years of our sampling period,
we believe this percentage should continue to increase
dramatically and we remain hopeful and convinced
that survey research will become a major method-
ological approach for conducting research in the OM
discipline.
6. Conclusions
The last 2 decades have evidenced remarkable
progress in the quantity and quality of empirical re-
search in OM. Survey research in OM, in particular,
has blossomed, becoming increasingly accepted as a
legitimate methodology for understanding the core
issues and problems that our discipline faces. In our
review, we have empirically substantiated this growth
and have demonstrated the improving nature and the
 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488 485

increasing rigor of OM survey research. However, Acknowledgements

we must acknowledge that opportunities for improve-
ment still existopportunities that, we believe, can
propel our research and results into a greater arena of
acceptance both within and beyond the boundaries of
the OM discipline.
We wish to acknowledge and thank the referees
and Jack R. Meredith, former Editor-in-Chief of the
JOM, for constructive and encouraging comments in
improving earlier versions of this paper.

Appendix A

Coding headings and definitions

Coding Definition

General issues:
Journal outlet What is the journal in which the article was published (DS, IJOPM, IJPR,
JOM, MS, or POM)?
Year of publication What year did the article appear in published form?
Main OM topic What is the main topic area that the article addresses?
Purpose of the article Does the article intend to describe some OM phenomenon (i.e.,
descriptive research) or does it intend to test relationships supported by
OM phenomena (i.e. relational or hypothesis testing research)?
Sample issues:
Sampling approach How is the sample constructed (e.g. random, stratified, etc.)?
Unit of analysis Did the paper clearly specify what the unit of analysis was, or did the
paper allow readers to infer what the unit of analysis was?
Measurement quality issues:
Reliability What approaches, if any, were employed to examine the reliability of the
multiple-item measurement scales or indices in the survey?
Construct validity What approaches, if any, were employed to examine the construct validity
of the multiple-item measurement scales or indices in the survey?
Analysis issues:
Common methods/source bias Did the paper acknowledge and/or address issues of common bias as might
be appropriate?

Appendix B

Top five research foci and sub-topics sampling

Research focus Sub-topics

Just-in-Time manufacturing (JIT) Definition of Just-in-Time manufacturing

Impact of JIT on organizational design
Performance implications
Inventory level determinants
JIT implementation
486 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488

Appendix B (Continued )
Research focus Sub-topics

Management of Technology (MoT) Robotics

Flexible manufacturing systems
Advanced manufacturing technology
Cellular manufacturing
AMT adoption
Operations Strategy (OS) Service strategy
Manufacturing strategy
Relationship between manufacturing and business strategy
Comparative study of manufacturing strategies
Strategy taxonomy
Quality Management (QM) Quality circles
Definition of total quality management
ISO 9000
Deming management method
Impact of design and process management on quality
Supply Chain Management (SCM) Sourcing strategies
Buyersupplier alliances
Supplier selection
Supply cost issues
Logistics partnership

References related journals by academicians. Journal of Operations

Ahire, S.L., Golhar, D.Y., Waller, M.A., 1996. Development
and validation of TQM implementation constructs. Decision
Sciences 27 (1), 2356.
Amoako-Gyampah, K., Meredith, J.R., 1989. The operations
management research agenda: an update. Journal of Operations
Management 8 (3), 250262.
Anderson, J.C., Rungtusanatham, M., Schroeder, R.G., Devaraj,
S., 1995. A path analytic model of a theory of quality
management underlying the Deming management method:
preliminary empirical findings. Decision Sciences 26 (5), 637
658.
Archer, N.P., Wesolowsky, G.O., 1996. Consumer response to
service and product quality: a study of motor vehicle owners.
Journal of Operations Management 14 (2), 103118.
Babbie, E., 1992. The Practice of Social Research, 6th ed.
Wadsworth Publishing Company, USA.
Barman, S., Hanna, M.D., LaForge, R.L., 2001. Perceived
relevance and quality of POM journals: a decade later. Journal
of Operations Management 19 (3), 367385.
Barman, S., Tersine, R.J., Buckley, M.R., 1991. An empirical
assessment of the perceived relevance and quality of POM-
Management 10 (2), 194212.
Boyer, K.K., Leong, G.K., Ward, P.T., Krajewski, L.J.,
1997. Unlocking the potential of advanced manufacturing
technologies. Journal of Operations Management 15 (4), 331
347.
Bozarth, C., Edwards, S., 1997. The impact of market requirements
focus and manufacturing characteristics focus on plant
performance. Journal of Operations Management 15 (2), 161
180.
Bozarth, C., Handfield, R., Das, A., 1998. Stages of global sourcing
strategy evolution: an exploratory study. Journal of Operations
Management 16 (23), 241255.
Buffa, E.S., 1980. Research in operations management. Journal of
Operations Management 1 (1), 17.
Carmines, E.G., Zeller, R.A., 1979. Reliability and Validity
Assessment. Sage Publications, USA.
Chase, R.B., 1980. A classification and evaluation of research
in operations management. Journal of Operations Management
1 (1), 914.
Chase, R.B., Singhal, K., 1981. An exploratory study of
interrelationships among technological and operational variables
in paced assembly lines. International Journal of Production
Research 19 (6), 703708.
 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488
Choi, T.Y., Eboch, K., 1998. The TQM paradox: relations among
TQM practices, plant performance, and customer satisfaction.
Journal of Operations Management 17 (1), 5975.
Collins, R.S., Cordon, C., 1997. Survey methodology issues
in manufacturing strategy and practice research. International
Journal of Operations & Production Management 17 (7), 697
706.
Crampton, S.M., Wagner, J.A., 1994. Perceptpercept inflation in
micro-organizational research: an investigation of prevalence
and effect. Journal of Applied Psychology 79 (1), 6776.
Filippini, R., 1997. Operations management research: some
reflections on evolution. International Journal of Operations &
Production Management 17 (7), 655670.
Flynn, B.B., Schroeder, R.G., Sakakibara, S., 1994. A frame-
work for quality management research and an associated
measurement instrument. Journal of Operations Management
11 (4), 339366.
Fornell, C., Larcker, D.F., 1981. Evaluating structural equation
models with unobservable variables and measurement error.
Journal of Marketing Research 18 (1), 3950.
Gilbert, J.P., 1990. The state of JIT implementation and
development in the USA. International Journal of Production
Research 28 (6), 10991109.
Goh, C.-H., Holsapple, C.W., Johnson, L.E., Tanner, J.R., 1997.
Evaluating and classifying POM journals. Journal of Operations
Management 15 (2), 123138.
Green, G.I., Appel, L.B., 1981. An empirical analysis of job
shop dispatch rule selection. Journal of Operations Management
1 (4), 197203.
Gupta, Y., Lonial, S., Mangold, W., 1991. An examination of
the relationship between manufacturing strategy and marketing
objectives. International Journal of Operations & Production
Management 11 (10), 3343.
Hartley, J.L., Zirger, B.J., Kamath, R.R., 1997. Managing the
buyersupplier interface for on-time performance in product
development. Journal of Operations Management 15 (1), 5770.
Heineke, J., 1995. Strategic operations management decisions and
professional performance in US HMOs. Journal of Operations
Management 13 (4), 255272.
Hensley, R.L., 1999. A review of operations management
studies using scale development scale. Journal of Operations
Management 17 (3), 343358.
Hill, A.V., Scudder, G.D., Haugen, D.L., 1988/1989. The
operations management research agenda for the 80s: a progress
report for Ph.D. dissertation research. Operations Management
Review 7 (1/2), 2432.
Kim, G.C., Lee, S.M., 1989. Impact of computer technology
on the implementation of Just-in-Time production systems.
International Journal of Operations & Production Management
9 (8), 2039.
Klassen, R.D., Angell, L.C., 1998. An international comparison
of environmental management in operations: the impact of
manufacturing flexibility in the US and Germany. Journal of
Operations Management 16 (2-3), 177194.
Malhotra, M.V., Grover, V., 1998. An assessment of survey research
in POM: from constructs to theory. Journal of Operations
Management 16 (4), 407425.
487
Malhotra, M.K., Steele, D.C., Grover, V., 1994. Important strategic
and tactical manufacturing issues in the 1990s. Decision
Sciences 25 (2), 189214.
Miller, J.G., Graham, M.B.W., 1981. Productionoperations
management: agenda for the 1980s. Decision Sciences 12 (4),
547571.
Narasimhan, R., Jayaram, J., 1998. Causal linkages in supply
chain management: an exploratory study of North Ameri-
can manufacturing firms. Decision Sciences 29 (3), 579
605.
Noble, M.A., 1997. Manufacturing competitive priorities and
productivity: an empirical study. International Journal of
Operations & Production Management 17 (1), 8599.
Perera, T., Shafaghi, M., 1995. Analysis of tooling problems
in discrete manufacturing industry. International Journal of
Operations & Production Management 15 (12), 7685.
Roth, A.V., van der Velde, M., 1991. Operations as marketing:
a comparative service strategy. Journal of Operations Manage-
ment 10 (3), 303328.
Rungtusanatham, M., 1998. Let us not overlook content validity.
Decision Line 29 (4), 1013.
Safizadeh, M.H., Ritzman, L.P., 1997. Linking performance drivers
in production planning and inventory control to process
choice. Journal of Operations Management 15 (4), 89
103.
Sakakibara, S., Flynn, B.B., Schroeder, R.G., 1993. A framework
and measurement instrument for Just-in-Time manufacturing.
Production and Operations Management 2 (3), 177194.
Saladin, B., 1985. Operations management research: where should
we publish? Operations Management Review 3, 39.
Saraph, J.V., Benson, P.G., Schroeder, R.G., 1989. An instrument
for measuring the critical factors of quality management.
Decision Sciences 20 (4), 810829.
Soteriou, A.C., Hadjinicola, G.C., Patsia, K., 1998. Assessing
production and operations management related journals: the
European perspective. Journal of Operations Management
17 (2), 225238.
Sriparavastu, L., Gupta, T., 1997. An empirical study of Just-in-
Time and total quality management principles implementation
in manufacturing firms in the USA. International Journal
of Operations & Production Management 17 (12), 1215
1232.
Sullivan, R.S., 1982. The service sector: challenges and imperatives
for research in operations management. Journal of Operations
Management 2 (4), 211214.
Swamidass, P.M., 1991. Empirical science: new frontier in
operations management research. Academy of Management
Review 16 (4), 793814.
Tan, K.C., Handfield, R., Krause, D.R., 1998. Enhancing the firms
performance through quality and supply base management: an
empirical study. International Journal of Production Research
36 (10), 28132837.
Vargas, G.A., Johnson, T.W., 1993. An analysis of operational
experience in the USMexico production sharing (maquiladora)
program. Journal of Operations Management 11 (1), 17
34.
488 M.J. Rungtusanatham et al. / Journal of Operations Management 21 (2003) 475488

Vokurka, R.J., 1996. The relative importance of journals used in
operations management research: a citation analysis. Journal of
Operations Management 14 (3), 345355.
Voss, C.A., 1984. Productionoperations management: a key dis-
cipline and area for research. OMEGAInternational Journal
of Management Science 12 (3), 308319.
Werts, C.E., Linn, R., Jreskorg, K.G., 1974. Inter-class reliability
estimates: testing structural assumptions. Educational and
Psychological Measurement 34 (1), 2533.
Wisner, J.D., 1996. A study of us machine shops with just-in-time
customers. International Journal of Operations & Production
Management 16 (7), 6276.