Feature Articles

Protocol-Based Resuscitation Bundle to Improve

Outcomes in Septic Shock Patients: Evaluation
of the Michigan Health and Hospital Association
Keystone Sepsis Collaborative*
Michael P. Thompson, PhD1,2; Mathew J. Reeves, PhD1; Brittany L. Bogan, MHSA, CPPS2;
Bruno DiGiovine, MD, MPH3; Patricia J. Posa, RN, BSN, MSA4; Sam R. Watson, MSA, CPPS2

Objectives: To evaluate the impact of a multi-ICU quality improve-
ment collaborative implementing a protocol-based resuscitation
bundle to treat septic shock patients.
Design: A difference-in-differences analysis compared patient
outcomes in hospitals participating in the Michigan Health &
Hospital Association Keystone Sepsis collaborative (n = 37) with
noncollaborative hospitals (n = 50) pre- (2010–2011) and post-
implementation (2012–2013). Collaborative hospitals were also
stratified as high (n = 19) and low (n = 18) adherence based on
their overall bundle adherence.
*See also p. 2275.
1
Department of Epidemiology and Biostatistics, Michigan State University,
East Lansing, MI.
2
Keystone Center for Patient Safety & Quality, Michigan Health & Hospital
Association, Okemos, MI.
3
Division of Pulmonary, Critical Care, and Sleep Medicine, Henry Ford
Health System, Detroit, MI.
4
St. Joseph Mercy Hospital, Ann Arbor, MI.
The Michigan Health & Hospital Association Keystone Center receives unre-
stricted donations from Blue Cross Blue Shield Foundation of Michigan.
Supplemental digital content is available for this article. Direct URL citations
appear in the printed text and are provided in the HTML and PDF versions
of this article on the journal’s website (http://journals.lww.com/ccmjournal).
Dr. Thompson disclosed other support and received support for article
research from Blue Cross Blue Shield Foundation of Michigan (The Michigan
Health & Hospital Association [MHA] Keystone Center receives unrestricted
donations). His institution received funding from the Blue Cross Blue Shield
Foundation of Michigan (The MHA Keystone Center receives unrestricted
donations). Dr. DiGiovine received funding from the Michigan Hospital Asso-
ciation, American Board of Internal Medicine, and Law firm of Silver Golub &
Teitell. He has a family disclosure (his wife owns stock in United Medical Sys-
tems). Dr. Posa received other support (Surviving Sepsis Campaign Soci-
Setting: Eighty-seven Michigan hospitals with ICUs.
Patients: We compared 22,319 septic shock patients in collab-
orative hospitals compared to 26,055 patients in noncollaborative
hospitals using the Michigan Inpatient Database.
Interventions: Multidisciplinary ICU teams received informational
toolkits, standardized screening tools, and continuous quality
improvement, aided by cultural improvement.
Measurements and Main Results: In-hospital mortality and hospi-
tal length of stay significantly improved between pre- and postim-
plementation periods for both collaborative and noncollaborative
hospitals. Comparing collaborative and noncollaborative hospi-
tals, we found no additional reductions in mortality (odds ratio,
0.94; 95% CI, 0.87–1.01; p = 0.106) or length of stay (–0.3 d;
95% CI, –0.7 to 0.1 d; p = 0.174). Compared to noncollabora-
tive hospitals, high adherence hospitals had significant reductions
in mortality (odds ratio, 0.84; 95% CI, 0.79–0.93; p < 0.001)
and length of stay (–0.7 d; 95% CI, –1.1 to –0.2; p < 0.001),
whereas low adherence hospitals did not (odds ratio, 1.07; 95%
CI, 0.97–1.19; p = 0.197; 0.2 d; 95% CI, –0.3 to 0.8; p = 0.367).
Conclusions: Participation in the Keystone Sepsis collaborative
was unable to improve patient outcomes beyond concurrent
trends. High bundle adherence hospitals had significantly greater
improvements in outcomes, but further work is needed to under-
stand these findings. (Crit Care Med 2016; 44:2123–2130)
Key Words: evaluation; outcomes research; quality improvement;
resuscitation bundle; septic shock

ety of Critical Care Medicine-ICU Liberation Collaborative Sage Products
Excelsior Medical) and received funding from Michigan Hospital Association,
Johns Hopkins Armstrong Institute, and Missouri Patient Safety Organiza-
tion. She is a Consultant for Advanced Nursing LLC. Dr. Watson’s institution
received funding from Cross Blue Shield of Michigan. The remaining authors
have disclosed that they do not have any potential conflicts of interest.
For information regarding this article, E-mail: mthompson@uthsc.edu
Copyright © 2016 by the Society of Critical Care Medicine and Wolters
Kluwer Health, Inc. All Rights Reserved.
DOI: 10.1097/CCM.0000000000001867
S
evere sepsis and septic shock are common and highly
fatal conditions, which place substantial burden on the
healthcare system (1–3). Great strides in the identifi-
cation and treatment of septic shock have been made in the
years since the formative 2001 study by Rivers et al (4), which
developed the early goal-directed therapy protocol to treat
septic shock patients. Protocol-based therapy—especially
time-dependent treatment bundles—remains the mainstay of

Critical Care Medicine www.ccmjournal.org 2123

Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Thompson et al
clinical guidelines for the treatment of septic shock patients
issued by the Surviving Sepsis Campaign (SSC) (5). A recent
SSC publication stated that increased compliance to a proto-
col-based therapy bundle was associated with a 25% relative
risk reduction in in-hospital mortality (6).
Despite the widespread promotion of protocol-based ther-
apy as the standard of care, there is substantial controversy
surrounding its efficacy. Recent randomized clinical trials have
shown no benefit of protocol-based therapies on mortality
or length of stay compared to usual care (7–9). There is also
concern that the protocol-based therapies may lead to inap-
propriate antibiotic use, unnecessary testing, and overuse of
invasive treatments (10). Additionally, successful implemen-
tation of protocol-based therapies is particularly difficult
(11–17). Common barriers to implementation include diffi-
culty in identifying septic shock patients, lack of knowledge
about septic shock in hospital staff, limited resources, and the
complexity of the protocol itself (18–20).
In 2010, the Michigan Health & Hospital Association (MHA)
Keystone Center developed a multi-ICU quality improvement
(QI) collaborative project called the “MHA Keystone Sepsis
Collaborative.” This program promoted the use of a protocol-
based resuscitation bundle in the ICU. To assess the impact
of the collaborative, we conducted a difference-in-differences
(DID) analysis to compare relative changes in outcomes
(mortality and length of stay) in collaborative hospitals with
changes observed in noncollaborative hospitals. Furthermore,
to explore how the level of resuscitation bundle adherence in
hospitals influenced changes in outcomes, we stratified collab-
orative hospitals into high and low adherence hospitals based
on their hospital-level adherence.
METHODS
MHA Keystone Sepsis Collaborative
In 2010, all Michigan hospitals with ICUs were invited to
participate in the MHA Keystone Sepsis collaborative. Once
enrolled, ICUs formed multidisciplinary teams, which included
physicians, nurses, pharmacists, respiratory therapists, and
other staff involved in patient care, such as clerks and techni-
cians. Teams were provided with an informational toolkit on
the clinical presentation of sepsis, evidence-based treatment of
sepsis, and conceptual framework for the resuscitation bundles.
Clinical screening and data abstraction tools were provided to
standardize septic shock case definitions and data collection
methods between hospitals. Face-to-face workshops followed
by regular coaching calls led by clinical experts educated sepsis
teams on the informational toolkit information and promoted
continuous QI. To foster implementation of the intervention,
MHA Keystone employed two previously successful QI strate-
gies: the four E’s in translating evidence into practice (Engage,
Educate, Execute, and Evaluate) (21) and a Comprehensive
Unit-Based Safety Program (CUSP) (22). CUSP engages and
empowers staff to improve patient safety and has previously
been shown to improve organizational culture and patient
safety and outcomes in ICUs (23).
2124 www.ccmjournal.org December 2016 • Volume 44 • Number 12
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Collaborative hospitals collected patient-level resuscitation
bundle compliance data on patients admitted to the ICU with
a diagnosis of septic shock, which were submitted monthly
through a secure online portal maintained by the MHA. The
eight bundle measures reflected the 2008 SSC guidelines: fluid
bolus administration, lactate measurement, antibiotic admin-
istration, obtaining two blood cultures, blood cultures obtained
prior to antibiotic administration, and clinical achievement in
central venous pressure (> 8 mm Hg), mean arterial pressure
(> 65 mm Hg), and central venous oxygen saturation (> 70%)
(24). No additional demographic or clinical information were
collected. Baseline data collection began in April of 2011, and
full uptake of the intervention was expected by the end of 2011.
The MHA Keystone Sepsis collaborative concluded at the end
of 2013, but a second iteration is ongoing, and is part of the
Blue Cross Blue Shield of Michigan pay-for-performance
program.
Data Source and Sample
Discharge data from the Michigan Inpatient Database (MIDB)
provided outcome and covariate data for hospitals partici-
pating in the Keystone Sepsis collaborative as well as noncol-
laborative hospitals. The MIDB is a comprehensive source
of all-payer, patient-level data on all acute-care hospital dis-
charges in the State of Michigan (25). Between 2010 and 2013,
we identified 49,232 patients from 87 hospitals with ICUs who
had the International Classification of Diseases, 9th Edition
code for septic shock (785.52) listed within the first 10 diag-
nosis codes (1). Patients were excluded from analysis if they
were under the age of 16 (n = 724; 1.5%), left against medical
advice (n = 185; 0.4%), had a hospital length of stay of more
than 90 days (n = 205; 0.4%), or had no information on dis-
charge disposition (n = 8; 0.02%). After exclusions, n equal to
48,110 septic shock cases remained for analysis. Since this proj-
ect utilized existing data with no identifiable protected health
information, it was deemed exempt from institutional review
board review.
Outcomes of Interest
The primary outcomes of interest in this analysis were in-hos-
pital mortality and hospital length of stay (d), obtained from
the MIDB.
Independent Variables
For this analysis, we defined two independent variables: pre-
versus postimplementation period and hospital participation
in the collaborative. Since full implementation of the col-
laborative was expected by the end of 2011, if patients were
discharged in either 2010 or 2011, they were considered preim-
plementation patients, and postimplementation if discharged
in 2012 or 2013. Hospitals were defined as collaborative hos-
pitals if they were enrolled in the Keystone Sepsis collaborative
(n = 37). Using bundle adherence data submitted to the MHA
portal, we calculated the average resuscitation bundle adher-
ence for each collaborative hospital as the sum of all bundle
measures each patient received divided by the sum of all

measures for which each patient was eligible. To explore how
the level of resuscitation bundle adherence influenced changes
in outcomes, we stratified collaborative participating hospitals
as either high or low adherence based on a median split of hos-
pital-level average bundle adherence.
Statistical Analysis
We compared demographic, clinical, and hospital charac-
teristics from the MIDB for patients treated in collaborative
(as well as high and low adherence hospitals) compared to
noncollaborative hospitals in the pre- and postimplementa-
tion periods of the collaborative using chi-square tests and
analysis of variance for categorical and continuous variables,
respectively. Patient demographics included age, gender, and
insurance payer (Medicare, Medicaid, private, other). Clini-
cal information included admission type (emergency, urgent,
elective, unknown), transferred patient status, and Charlson
comorbidity index (26, 27). Hospital characteristics included
rural versus urban location, teaching status, bed size (< 100,
100–300, and 300+), proportion of Medicaid patients, bed size,
and teaching status.
We estimated unadjusted pre- versus postimplementation
effects in collaborative and noncollaborative hospitals, as well
as high and low adherence hospitals using logistic regression
for in-hospital mortality, and linear regression for hospital
length of stay. We then adjusted pre- versus postimplementa-
tion effects for patient characteristics, and then patient and
hospital characteristics. Risk-adjusted in-hospital mortality
rates and hospital length of stay pre- and postimplementation
were estimated using model-based indirect standardization.
To evaluate the impact of participation in the collaborative
on in-hospital mortality and hospital length of stay, we per-
formed a DID analysis. The DID method is a well-cited and
highly robust method that compares relative changes in out-
comes observed in the intervention group (difference 1) with
concurrent changes observed in a control group undergoing
similar temporal trends but are not exposed to the intervention
(difference 2) (28, 29). Conceptually, the independent effect of
the intervention is referred to as “the DID estimator,” which rep-
resents the “DID” between intervention and nonintervention
groups pre- and postimplementation (difference 1 – difference
2 = DID estimator). In practice, the DID estimator is derived
from regression models and is represented by an interaction
term between an intervention indicator (i.e., hospital collabor-
ative status) and indicator for pre- versus postimplementation
period. Using regression models also allow for adjustment on
patient and hospital characteristics. A statistically significant,
nonzero interaction term would indicate that relative changes
in outcomes observed in the intervention group were signifi-
cantly different than relative changes observed in the control
group. Further explanation of the DID analysis can be seen in
the Supplementary Appendix (Supplemental Digital Content
1, http://links.lww.com/CCM/B930).
The DID analysis was performed for both outcomes com-
paring collaborative versus noncollaborative hospitals, as well
as high and low adherence versus noncollaborative hospitals.
Critical Care Medicine www.ccmjournal.org 2125
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Feature Articles
In a sensitivity analysis, we reran the DID analysis for length
of stay excluding patients who died, as length of stay may be
shorter for patients who died in the hospital. We also excluded
2011 data from the preimplementation period to eliminate
any spillover of the intervention into the preimplementation
period. To test the parallel trends assumption, we also per-
formed the DID analysis comparing outcome trends during
the preintervention period (i.e., comparing 2010 vs 2011) in all
comparison groups.
RESULTS
Patient and hospital characteristics of the 48,110 septic shock
patients stratified by pre- versus postimplementation and col-
laborative status are displayed in Table 1. Patients and hospital
characteristics stratified by high and low adherence hospitals
are displayed in Supplementary Table 1 (Supplemental Digi-
tal Content 2, http://links.lww.com/CCM/B931). The range
of hospital-level bundle adherence (potential range, 0–8 mea-
sures) was 2.1–6.7, with a median of 4.8 (interquartile range,
3.6–5.2). (Fig. 1) All hospitals with an average adherence of 4.8
or greater were deemed high adherence hospitals (n = 19), and
those below were deemed low adherence (n = 18).
Risk-adjusted in-hospital mortality rates and hospital
length of stay (d) pre- and postimplementation for each hos-
pital cohort are shown in Figures 2 and 3, respectively. Both
collaborative (33.8% vs 30.5%; p < 0.001) and noncollabo-
rative (32.9% vs 30.9%; p = 0.002) hospitals had significant
differences in pre- versus postimplementation. (Fig. 2) After
stratifying collaborative hospitals by bundle adherence level,
high adherence hospitals had significantly reduced in-hospi-
tal mortality between pre- and postperiods (35.0% vs 29.7%;
p < 0.001), whereas low adherence hospitals did not (32.2%
vs 31.7%; p = 0.643). Similarly, collaborative (12.8 vs 12.0 d;
p < 0.001) and noncollaborative (12.4 vs 11.8 d; p < 0.001) hos-
pitals had statistically significant reductions in hospital length
of stay. (Fig. 3) When stratified by adherence level, high adher-
ence hospitals had a significant reduction in length of stay
(13.4 vs 12.0 d; p < 0.001), whereas low adherence hospitals
did not (12.1 vs 11.8 d; p = 0.118).
Table 2 shows the results of the DID analysis for in-hospital
mortality and hospital length of stay. There was no signifi-
cant interaction between collaborative status and pre- versus
postimplementation in in-hospital mortality (DID odds ratio
[OR], 0.94; 95% CI, 0.87–1.01; p = 0.106) or hospital length of
stay (DID estimator, –0.3 d; 95% CI, –0.7 to 0.1 d; p = 0.174),
indicating no additional effect of the collaborative on improv-
ing patient outcomes compared to noncollaborative hospitals.
However, high bundle adherence hospitals had a significant
reduction in in-hospital mortality (DID OR, 0.85; 95% CI,
0.78–0.93; p = 0.001) and hospital length of stay (DID esti-
mator, –0.7 d; 95% CI, –1.1 to –0.2; p = 0.005) compared to
noncollaborative hospitals. Conversely, low bundle adherence
hospitals had no significant changes in in-hospital mortal-
ity (DID OR, 1.07; 95% CI, 0.97–1.19; p = 0.197) or hospital
length of stay (DID estimator, 0.2 d; 95% CI, –0.3 to 0.8 d;
p = 0.367) compared to noncollaborative hospitals.
Thompson et al

Table 1.Patient and Hospital Characteristics of Collaborative (n = 37) and Noncollaborative

(n = 50) Hospitals Before (2010–2011) and After (2012–2013) Full Implementation of the
Michigan Health & Hospital Association Keystone Sepsis Collaborative
Preimplementation (2010–2011) Postimplementation (2012–2013)
n = 22,197 n = 25,913

Collaborative Noncollaborative Collaborative Noncollaborative

Variable n = 10,893 (49.1%) n = 11,304 (50.9%) p n = 12,782 (49.3%) n = 13,131 (50.7%) p

Patient characteristics
  Male gender 50.3 51.4 0.110 49.9 51.2 0.032
 Age a
67.6 (15.6) 66.2 (15.9) < 0.001 67.5 (15.4) 66.4 (15.8) < 0.001
  Charlson comorbidity index a
2.8 (2.3) 2.8 (2.3) 0.109 2.8 (2.3) 2.7 (2.3) 0.149
  Insurance type < 0.001 < 0.001
  Medicare 70.9 66.8 70.1 69.2
  Medicaid 10.6 10.8 11.4 10.1
  Private 15.7 17.3 15.9 16.9
  Other 2.8 5.2 2.6 3.7
  Admission type < 0.001 < 0.001
  Emergency 82.6 69.3 87.0 73.2
  Urgent 11.4 23.2 9.1 22.4
  Elective 3.3 4.3 3.7 3.7
  Unknown 2.6 3.2 0.3 0.7
  Transferred patient 20.1 14.6 < 0.001 21.2 16.0 < 0.001
Hospital characteristics
 Rural 8.8 3.9 < 0.001 9.8 4.2 < 0.001
 Teaching 63.1 64.8 0.009 63.4 65.6 0.003
  Bed size < 0.001 < 0.001
  300+ beds 56.1 77.3 57.7 78.2
  100–300 beds 34.6 17.2 33.3 17.0
   < 100 beds 9.4 5.5 9.1 5.8
  Hospital Medicaid proportion a
19.4 (7.7) 17.4 (8.1) < 0.001 19.1 (7.7) 16.8 (7.6) < 0.001
Patient outcomes
  Hospital discharge status < 0.001 0.001
  Died 33.8 32.9 30.5 30.9
   Discharged to home 11.2 14.2 13.4 14.1
   Discharged to extra care 45.5 45.2 46.1 46.6
   Discharged to hospice 9.6 7.6 10.0 8.5
  Hospital length of stay (d), 10 (5–17) 9 (5–16) 0.008 9 (5–16) 8 (4–15) 0.147
median (IQR)
IQR = interquartile range.
Mean (sd).
a

n = 48,110 patients.

2126 www.ccmjournal.org December 2016 • Volume 44 • Number 12

Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
 Feature Articles

Supplemental Digital Content

4, http://links.lww.com/CCM/
B933).

DISCUSSION
We used a controlled, pre-post
design (DID analysis) to evalu-
ate a multi-ICU QI collabora-
tive employing a protocol-based
resuscitation bundle to improve
outcomes in septic shock
patients. Our study has several
important findings. First, we
illustrate recent trends in septic
shock outcomes from a diverse,
statewide population, which
saw a 2–3% absolute reduction
in in-hospital mortality and a
1-day reduction in length of
stay over a comparatively short
4-year period. Second, we found
Figure 1. Hospital-level adherence to the protocol-based resuscitation bundle for hospitals participation in the that participation in the Key-
Michigan Health & Hospital Association Keystone Sepsis collaborative (n = 37). IQR = interquartile range. stone Sepsis collaborative was
not associated with additional
When excluding patients who died, compared to noncol- improvements in in-hospital mortality or hospital length of
laborative hospitals, collaborative hospitals did have a significant stay. By using a concurrent control group of hospitals, we were
additional reduction in length of stay (DID estimator, –0.5 d; 95% able to separate out underlying secular trends in outcomes
CI, –1.0 to –0.03; p = 0.038). Only high adherence hospitals had from collaborative effects, thereby estimating the independent
an additional reduction in length of stay compared to noncol- impact of the collaborative. Most observational studies of pro-
laborative hospitals (DID estimator, –0.7 d; 95% CI, –1.4 to –0.3; tocol-based therapies use either a historical control group or
p = 0.003), whereas low adherence hospitals did not (DID esti- no control group at all and are unable to distinguish the inter-
mator, –0.1 d; 95% CI, –0.7 to 0.6; p = 0.801) (Supplementary vention effect from the underlying secular trend adequately
Table 2, Supplemental Digital Content 3, http://links.lww.com/ (30–34). Finally, we demonstrated substantial variation in
CCM/B932). After removing 2011 data from our analyses, our bundle adherence between hospitals and that hospitals with
results did not change substantially. When testing for parallel high adherence to the protocol received additional improve-
trends, we found no significant difference in trends during the ments in patient outcomes beyond the observed secular trends.
preintervention period between any of the comparison groups, These results may help explain why simply participation in the
suggesting that the assumption holds (Supplementary Table 3, QI collaborative was not sufficient to improve outcomes.
The simplest explanation for our findings in high adher-
ence hospitals is that the additional improvements in out-
comes beyond concurrent trends are due to the clinical benefit
of the protocol-based resuscitation bundle. Observational
studies have shown that protocol-based therapies do improve
patient outcomes, although results should be interpreted in
light of potential selection and confounding biases (6, 30–34).
Conversely, recent clinical trials have found that protocol-
based therapies provided no additional benefit in outcomes
compared to usual care for septic shock (7–9). However, the
reported overall mortality in these trials were 25% or less,
which is lower than the mortality rates reported in this study
and in more recent SSC data (29%) (6). Patients in these trials
may not represent the typical spectrum of septic shock patients
or be comparable with earlier studies of protocol-based thera-
pies (4, 35). Since we did not have information on patient-level
Figure 2. Pre-post implementation comparisons of risk-adjusted
in-hospital mortality, stratified by hospital collaborative participation, as bundle adherence, we can only speculate on the causal link
well as by adherence level in collaborative hospitals. between bundle adherence and improved patient outcomes.

Critical Care Medicine www.ccmjournal.org 2127

Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Thompson et al
Figure 3. Pre-post implementation comparisons of risk-adjusted hospital
length of stay (in days), stratified by hospital collaborative participation, as
well as by adherence level in collaborative hospitals.
However, there may be alternative explanations for why
high adherence hospitals gained additional improvements
in outcomes, beyond the purported clinical benefits of the
resuscitation bundle. The intent of the cultural improvement
component of the intervention (CUSP) was to create a col-
laborative and safe work environment, enhance leadership
support, and increase receptiveness to process improvement,
which are critical characteristics of successful QI projects
(36, 37). Hospitals with high adherence to the bundle may
reflect many of these characteristics. Exploring the relation-
ship between unit culture, bundle adherence, and patient
outcomes could prove especially valuable in understanding
our findings. Furthermore, because limited resources and
treatment complexity are common barriers to protocol-
based therapies, further research should explore the role of
characteristics of the ICUs, such as resources, staffing, and
training (18–20).
Successful implementation of QI initiatives rely on three
components: the clinical context of the initiative, the facili-
tation of the initiative in practice, and the quality of the
evidence employed by the initiative (37). The primary func-
tion of the MHA Keystone Center is to provide assistance
with the first two components through cultural improve-
ment and project management and coordination. Thus,
the only unique component in each collaborative is the
evidence-based practice being employed. In the Keystone
ICU program, a substantial reduction in the incidence of
central line-associated bloodstream infections (CLABSI)
was achieved (38). This was likely due in large part to clear,
high-quality evidence for prevention of CLABSI (39, 40).
Conversely, the Keystone Surgery program did not show
any improvement in surgical outcomes, which the authors
attributed to mixed evidence on the efficacy of surgical
checklists to improve outcomes (41). We believe that skep-
ticism surrounding the clinical efficacy of protocol-based
therapies for septic shock may have limited the success
of the Keystone Sepsis collaborative, as illustrated by the
2128 www.ccmjournal.org December 2016 • Volume 44 • Number 12
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
substantial hospital-level disparities in bundle adherence.
Guidelines requiring placement of central venous catheters
are perceived by many care providers to be invasive, time-
consuming, and providing little direct gain to the patient (5,
14). It is important to note that the SSC clinical guidelines
indicate that catheter-based measures have low levels of evi-
dence for benefit (grade C) (24). This study combined with
findings from previous MHA Keystone Center collabora-
tives underscore the importance of high-quality evidence
for interventions used in QI initiatives.
Finally, it is possible that our findings are simply the result
of regression to the mean. In-hospital mortality rates and
length of stay in high adherence hospitals were the highest of
all groups in the preimplementation phase and the lowest in
the postimplementation phase. When using the DID method,
it is important that expected improvements in outcomes are
not associated with baseline levels (29, 42). If high adher-
ence hospitals had relatively poorer adherence prior to the
intervention, we might expect those hospitals to have greater
improvements in outcomes. Since we did not have preinter-
vention bundle adherence data, we were unable to examine
how changes in bundle adherence affected outcome changes
over time. This analysis would confirm that our findings were
not simply a regression to the mean.
There are limitations to our study to consider. First, impor-
tant prognostic factors not available in the MIDB, such as
infection source, serum lactate, and Acute Physiology and
Chronic Health Evaluation II score, could lead to unmeasured
confounding (43–45). Second, recent evidence suggests that
improved recognition of severe sepsis and septic shock may
increase reported case incidence (1, 10). We show similar
trends in the number of septic shock cases in collaborative
and noncollaborative hospitals from pre- to postimplemen-
tation, suggesting nondifferential changes in recognition and
coding of septic shock between groups. Third, although the
MHA intervention applied to patients treated in the ICU, the
MIDB includes all discharged patients, which may include
non-ICU septic shock patients. Although rare, mortality in
non-ICU septic shock patients may have higher mortality
compared to ICU patients (46). Finally, the MIDB did not
provide data on other relevant outcomes, such as recurrent
organ failure, postdischarge ambulatory status, costs, and
postdischarge mortality.
CONCLUSIONS
This study found that participation in this ICU-based QI
collaborative was not sufficient to improve outcomes in
septic shock patients beyond concurrent trends. Only hos-
pitals with high bundle adherence had significantly greater
improvements in outcomes. Future work should explore the
mechanism behind the successes of high adherence hospi-
tals. Nevertheless, substantial work remains to improve out-
comes in septic shock patients. A universally accepted set of
treatment guidelines supported by high-quality evidence
is essential to ensure the success of future septic shock QI
initiatives.
 Feature Articles

Table 2. Results of Difference-In-Difference Analysis for In-Hospital Mortality and Hospital

Length of Stay (in Days) Showing Unadjusted and Adjusted Pre-Post Comparisons and
the Difference-In-Difference Estimator for the Independent Collaborative Effect
Pre-Post Comparisons
Difference-
+ Patient + Hospital In-Difference
Patient Cohort Unadjusted Characteristicsa Characteristicsb Estimatorc p

OR (95% CI) for septic shock mortality

  Collaborative (n = 37) 0.86 0.86 0.86 0.94 0.106
(0.81–0.91) (0.81–0.91) (0.81–0.91) (0.87–1.01)
  Noncollaborative (n = 50) 0.91 0.91 0.92 Reference —
(0.86–0.96) (0.87–0.97) (0.87–0.97)
  High adherence (n = 19) 0.79 0.78 0.78 0.85 0.001
(0.73–0.84) (0.73–0.84) (0.73–0.84) (0.78–0.93)
  Low adherence (n = 18) 0.97 0.98 0.98 1.07 0.197
(0.89–1.06) (0.90–1.07) (0.90–1.07) (0.97–1.19)
  Noncollaborative (n = 50) 0.91 0.91 0.92 Reference —
(0.86–0.96) (0.87–0.97) (0.87–0.97)
Absolute change in length of stay (95% CI) (d)
  Collaborative (n = 37) –0.9 –0.9 –0.9 –0.3 0.174
(–1.2 to –0.6) (–1.2 to –0.6) (–1.2 to –0.6) (–0.7 to 0.1)
  Noncollaborative (n = 50) –0.7 –0.6 –0.6 Reference —
(–1.0 to –0.4) (–0.9 to –0.3) (–0.9 to –0.3)
  High adherence (n = 19) –1.3 –1.3 –1.3 –0.7 0.005
(–1.7 to –1.0) (–1.7 to –0.9) (–1.6 to –0.9) (–1.1 to –0.2)
  Low adherence (n = 18) –0.3 –0.3 –0.4 0.2 0.367
(–0.7 to 0.2) (–0.8 to 0.1) (–0.8 to 0.1) (–0.3 to 0.8)
  Noncollaborative (n = 50) –0.7 –0.6 –0.6 Reference —
(–1.0 to –0.4) (–0.9 to –0.3) (–0.9 to –0.3)
OR = odds ratio.
Adjusted for age, gender, insurance status, Charlson comorbidity index, type of admission, and transfer status.
a

Adjusted for patient characteristics plus rural versus urban, teaching status, bed size, and hospital Medicaid proportion.
b

Difference-in-difference estimator is the comparison of pre-post differences between patient cohorts (collaborative vs noncollaborative, high/low adherence vs
c

noncollaborative) and represents the independent effect of the Michigan Health & Hospital Association Keystone Sepsis collaborative.

REFERENCES
1. Gaieski DF, Edwards JM, Kallan MJ, et al: Benchmarking the inci-
dence and mortality of severe sepsis in the United States. Crit Care
Med 2013; 41:1167–1174
2. Kaukonen KM, Bailey M, Suzuki S, et al: Mortality related to severe
sepsis and septic shock among critically ill patients in Australia and
New Zealand, 2000-2012. JAMA 2014; 311:1308–1316
3. Lagu T, Rothberg MB, Shieh MS, et al: Hospitalizations, costs, and
outcomes of severe sepsis in the United States 2003 to 2007. Crit
Care Med 2012; 40:754–761
4. Rivers E, Nguyen B, Havstad S, et al; Early Goal-Directed Therapy Col-
laborative Group: Early goal-directed therapy in the treatment of severe
sepsis and septic shock. N Engl J Med 2001; 345:1368–1377
5. Dellinger RP, Levy MM, Rhodes A, et al; Surviving Sepsis Campaign
Guidelines Committee including The Pediatric Subgroup: Surviving Sep-
sis Campaign: International guidelines for management of severe sepsis
and septic shock, 2012. Intensive Care Med 2013; 39:165–228
6. Levy MM, Rhodes A, Phillips GS, et al: Surviving Sepsis Campaign:
Association between performance metrics and outcomes in a 7.5-year
study. Crit Care Med 2015; 43:3–12
7. Peake SL, Delanay A, Bailey M, et al; The ARISE Investigators;
ANZICS Clinical Trials Group: Goal-directed resuscitation for patients
with early septic shock. N Engl J Med 2014; 371:1496–1506
8. Yealy DM, Kellum JA, Huang DT, et al; The ProcESS Investigators: A
randomized trial of protocol-based care for early septic shock. N Engl
J Med 2014; 370:1683–1693
9. Mouncey PR, Osborn TM, Power GS, et al; ProMISe Trial Investiga-
tors: Trial of early, goal-directed resuscitation for septic shock. N Engl
J Med 2015; 372:1301–1311
10. Rhee C, Gohil S, Klompas M: Regulatory mandates for sepsis care–
reasons for caution. N Engl J Med 2014; 370:1673–1676
11. Gao F, Melody T, Daniels DF, et al: The impact of compliance with
6-hour and 24-hour sepsis bundles on hospital mortality in patients
with severe sepsis: A prospective observational study. Crit Care
2005; 9:R764–R770
12. Casserly B, Baram M, Walsh P, et al: Implementing a collaborative
protocol in a sepsis intervention program: Lessons learned. Lung
2011; 189:11–19
13. Crowe CA, Mistry CD, Rzechula K, et al: Evaluation of a modified early
goal-directed therapy protocol. Am J Emerg Med 2010; 28:689–693
14. O’Neill R, Morales J, Jule M: Early goal-directed therapy (EGDT) for
severe sepsis/septic shock: Which components of treatment are
more difficult to implement in a community-based emergency depart-
ment? J Emerg Med 2012; 42:503–510
15. Jones AE, Shapiro NI, Roshon M: Implementing early goal-directed
therapy in the emergency setting: The challenges and experiences of

Critical Care Medicine www.ccmjournal.org 2129

Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Thompson et al
translating research innovations into clinical reality in academic and
community settings. Acad Emerg Med 2007; 14:1072–1078
16. Reade MC, Huang DT, Bell D, et al; British Association for Emergency
Medicine; UK Intensive Care Society; UK Society for Acute Medicine;
Australasian Resuscitation in Sepsis Evaluation Investigators; Proto-
colized Care for Early Septic Shock Investigators: Variability in man-
agement of early severe sepsis. Emerg Med J 2010; 27:110–115
17. Turi SK, Von Ah D: Implementation of early goal-directed therapy for
septic patients in the emergency department: A review of the litera-
ture. J Emerg Nurs 2013; 39:13–19
18. Carlbom DJ, Rubenfeld GD: Barriers to implementing protocol-
based sepsis resuscitation in the emergency department–results of
a national survey. Crit Care Med 2007; 35:2525–2532
19. Burney M, Underwood J, McEvoy S, et al: Early detection and treat-
ment of severe sepsis in the emergency department: Identifying bar-
riers to implementation of a protocol-based approach. J Emerg Nurs
2012; 38:512–517
20. Jones AE, Kline JA: Use of goal-directed therapy for severe sepsis
and septic shock in academic emergency departments. Crit Care
Med 2005; 33:1888–1889; author reply 1889
21. Pronovost PJ, Berenholtz SM, Goeschel CA, et al: Creating high reliabil-
ity in health care organizations. Health Serv Res 2006; 41:1599–1617
22. Johns Hopkins Medicine: Center for Innovation in Quality Patient Care:
The Comprehensive Unit-based Safety Program (CUSP). Available at:
http://www.hopkinsmedicine.org/innovation_quality_patient_care/areas_
expertise/improve_patient_safety/cusp/. Accessed June 30, 2014
23. Pronovost P, Weast B, Rosenstein B, et al: Implementing and validat-
ing a comprehensive unit-based safety program. J Patient Saf 2005;
1:33–40
24. Dellinger RP, Levy MM, Carlet JM, et al; International Surviving Sepsis
Campaign Guidelines Committee; American Association of Critical-Care
Nurses; American College of Chest Physicians; American College of
Emergency Physicians; Canadian Critical Care Society; European Soci-
ety of Clinical Microbiology and Infectious Diseases; European Society
of Intensive Care Medicine; European Respiratory Society; International
Sepsis Forum; Japanese Association for Acute Medicine; Japanese
Society of Intensive Care Medicine; Society of Critical Care Medicine;
Society of Hospital Medicine; Surgical Infection Society; World Federa-
tion of Societies of Intensive and Critical Care Medicine: Surviving Sep-
sis Campaign: International guidelines for management of severe sepsis
and septic shock: 2008. Crit Care Med 2008; 36:296–327
25. Data Koala: Product Information - Michigan Inpatient Database
(MIDB), 2014. Available at: http://www.datakoala.com/productinfo.
htm. Accessed June 17, 2015
26. Deyo RA, Cherkin DC, Ciol MA: Adapting a clinical comorbidity index
for use with ICD-9-CM administrative databases. J Clin Epidemiol
1992; 45:613–619
27. Quan H, Sundararajan V, Halfon P, et al: Coding algorithms for defin-
ing comorbidities in ICD-9-CM and ICD-10 administrative data. Med
Care 2005; 43:1130–1139
28. Dimick JB, Ryan AM: Methods for evaluating changes in health
care policy: The difference-in-differences approach. JAMA 2014;
312:2401–2402
29. Ryan AM, Burgess JF Jr, Dimick JB: Why we should not be indifferent
to specification choices for difference-in-differences. Health Serv Res
2015; 50:1211–1235
2130 www.ccmjournal.org December 2016 • Volume 44 • Number 12
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
30. Castellanos-Ortega A, Suberviola B, García-Astudillo LA, et al: Impact
of the Surviving Sepsis Campaign protocols on hospital length of stay
and mortality in septic shock patients: Results of a three-year follow-
up quasi-experimental study. Crit Care Med 2010; 38:1036–1043
31. Levy MM, Dellinger RP, Townsend SR, et al: The Surviving Sepsis
Campaign: Results of an international guideline-based performance
improvement program targeting severe sepsis. Intensive Care Med
2010; 36:222–231
32. Micek ST, Roubinian N, Heuring T, et al: Before-after study of a stan-
dardized hospital order set for the management of septic shock. Crit
Care Med 2006; 34:2707–2713
33. Ferrer R, Artigas A, Levy MM, et al; Edusepsis Study Group: Improve-
ment in process of care and outcome after a multicenter severe sep-
sis educational program in Spain. JAMA 2008; 299:2294–2303
34. Nguyen HB, Corbett SW, Steele R, et al: Implementation of a bundle
of quality indicators for the early management of severe sepsis and
septic shock is associated with decreased mortality. Crit Care Med
2007; 35:1105–1112
35. Seymour CW, Rosengart MR: Septic Shock: Advances in Diagnosis
and Treatment. JAMA 2015; 314:708–717
36. Kaplan HC, Brady PW, Dritz MC, et al: The influence of context on
quality improvement success in health care: A systematic review of
the literature. Milbank Q 2010; 88:500–559
37. Helfrich CD, Damschroder LJ, Hagedorn HJ, et al: A critical synthesis
of literature on the promoting action on research implementation in
health services (PARIHS) framework. Implement Sci 2010; 5:82
38. Pronovost P, Needham D, Berenholtz S, et al: An intervention to
decrease catheter-related bloodstream infections in the ICU. N Engl
J Med 2006; 355:2725–2732
39. Casey AL, Mermel LA, Nightingale P, et al: Antimicrobial central
venous catheters in adults: A systematic review and meta-analysis.
Lancet Infect Dis 2008; 8:763–776
4 0. Timsit JF, Schwebel C, Bouadma L, et al; Dressing Study
Group: Chlorhexidine-impregnated sponges and less frequent
dressing changes for prevention of catheter-related infections
in critically ill adults: A randomized controlled trial. JAMA 2009;
301:1231–1241
41. Reames BN, Krell RW, Campbell DA Jr, et al: A checklist-based inter-
vention to improve surgical outcomes in Michigan: Evaluation of the
Keystone Surgery program. JAMA Surg 2015; 150:208–215
42. Chay KY, McEwan PJ, Urquiola M: The central role of noise in evaluat-
ing interventions that use test scores to rank schools. Am Econ Rev
2005; 95:1237–1258
43. Leligdowicz A, Dodek PM, Norena M, et al; Co-operative Anti-
microbial Therapy of Septic Shock Database Research Group:
Association between source of infection and hospital mortality in
patients who have septic shock. Am J Respir Crit Care Med 2014;
189:1204–1213
44. Nguyen HB, Rivers EP, Knoblich BP, et al: Early lactate clearance is
associated with improved outcome in severe sepsis and septic shock.
Crit Care Med 2004; 32:1637–1642
45. Knaus WA, Draper EA, Wagner DP, et al: APACHE II: A severity of
disease classification system. Crit Care Med 1985; 13:818–829
46. Esteban A, Frutos-Vivar F, Ferguson ND, et al: Sepsis incidence and
outcome: Contrasting the intensive care unit with the hospital ward.
Crit Care Med 2007; 35:1284–1289