ANALYSIS PLAN: ASSESSMENT OF NON-RESPONSE BIAS IN AN EMPLOYEE SURVEY OF HEALTH AND WORKPLACE PRODUCTIVITY

Prepared by: COMPANY

For: CLIENT

October 12, 2006

Table of Contents
1 Research Objectives........................................................................................................1 2 Background.....................................................................................................................2 3 Data..................................................................................................................................5 4 Methods............................................................................................................................9 4.1 Descriptive comparisons and analysis of response propensity..................................9 4.2 Weighting.................................................................................................................10 4.3 Assessment of Bias and Measurement Error...........................................................11 5 Timeline.........................................................................................................................13

9/19/2011

Confidential

1 Research Objectives
The purpose of this work is to assess and adjust for sample non-response in webbased surveys of employees health and work productivity. The main objective of the survey is to estimate indirect costs to employers due to illness in the workplace. Because we want estimates of work loss to be unbiased, our assessment of the impact of adjusting for non-response will focus on survey variables absenteeism and presenteeism scores that estimate work productivity loss. The analysis will consist of three main phases of work: 1. Analyzing response propensity, which will address whether observable employee characteristics are associated with survey non-response; 2. Adjusting survey estimates for non-response bias; and 3. Assessing the impact of adjustment methods on the tradeoff between reduction in bias and decreased efficiency in estimates of absenteeism and presenteeism. Two concepts are important in obtaining an unbiased survey estimator. The first is to minimize nonresponse by planning during the survey development and implementation phases. Given that attempts have been made in the planning and implementation of the survey, post-survey adjustment of the estimator helps to further minimize bias originating from nonresponse. The availability of medical claims, prescription drug, and disability data from employers that have conducted surveys of health and work productivity offers a unique opportunity. Auxiliary information on the demographic and health characteristics of all employees who were eligible for the survey can be used to determine predictors of nonresponse. Moreover, this information can be used to adjust survey estimates for potential bias from nonresponse, leading to more accurate estimates of the impact of conditions such as depression on absenteeism and presenteeism. Note that validity issues surrounding concordance of condition reporting in the survey and in medical claims will not be addressed in this analysis, though the extent of concordance between self-reported information and claims-based indicators will be considered in the context of comparisons between respondents and non-respondents.

Page 1

9/19/2011

Confidential

Contents of this analysis plan include: Background (Section 2) Data (Section 3) Methods (Section 4) Timeline (Section 5)

2 Background
Reports of work absence and productivity are based on a modified series of questions from the World Health Organization Health and Work Performance Questionnaire (HPQ) and included in a web-based survey of employees in one firm.1 The web-based survey asked respondents to report whether they have a given chronic health condition (and whether they have been treated for that condition), along with self-assessments of work performance. Using methods developed by Kessler for the HPQ, responses to the survey can be used to determine indirect costs to employers due to illness in the workplace.2 Many of the survey items correspond with variables that can be created from available medical claims and disability data on the employee population which was surveyed, providing good auxiliary information to study correlates of non-response. We describe the survey items and claims-based data elements in Section 3 of the analysis plan.

2.1.1 Employee survey data

Data for the analyses are from a COMPANY database that is currently under development. The database is comprised of data from surveys of employees linked to medical and prescription drug claims on each employers covered lives. Topics covered by each employers primary data collection effort vary across employer. In general, the surveys collect information on medical conditions, treatment for medical conditions, and health-related work productivity. For this work, we are adjusting for potential biases in estimates from the health and productivity surveys that are due to survey non-response. It is important to adjust for potential bias due to non-response on an employer-by-employer basis because each data collection effort is implemented in one employer population, not across several different firms employee populations.

Kessler RC, Barber C, Beck A, et al. The World Health Organization Health and Work Performance Questionnaire (HPQ). JOEM 2003; 45(2): 156-174. 2 Kessler RC, Ames M, Hymel PA, et al. Using the World Health Organization Health and Work Performance Questionnaire (HPQ) to evaluate the indirect workplace costs of illness. JOEM 2004; 46(6 suppl): S23-S37.

Page 2

9/19/2011

Confidential

We will start by using one large firms medical and prescription drug claims, linked to employee responses to a web-based survey on employee health and work productivity. Employees of the firm, referred to as Database1, comprise the sampled population for the initial round of analyses.3 The sampling frame for the first company consists of all Database1 employees who were eligible for health benefits through the firm. For purposes of this analysis plan, the response set consists of 10,154 employees who completed the survey in Q4 2005 and were eligible for health benefits in 2005.4 The non-response set consists of employees who had health benefits in 2005 but did not complete the survey. This yields an approximate survey response rate of 37 percent. COMPANY is currently setting up the survey and claims data for Database2, another large employer. The employer survey for Database2 is similar to the structure of the Database1 survey. Because of the similar survey structures, the plan for analysis of Database1 data applies also to Database2 data, which COMPANY will analyze after developing preliminary models for Database1.

2.1.2 Using auxiliary data to assess survey non-response bias

Various auxiliary data are available from claims, pharmacy, and disability data. For each employer, these types of auxiliary data are available on the employees who were eligible to respond to the survey, giving information on both respondents and non-respondents. All employees can thus be assigned to categories on the basis of the auxiliary data and then these categories can be used as weighting strata.5 Some recent applications have used auxiliary data available from health care claims to adjust survey estimates for non-response.6,7 Possible stratification factors (or regressors, in the case of a response propensity model) that are available from auxiliary data include measures of: demographic characteristics health status prior to the survey health care use and costs prior to the survey whether the employee had a prior disability leave

Srndal and Lundstrm state optimal properties of an auxiliary vector:

COMPANY is also preparing data from a second employer (Custid56), which will be included in the analysis towards the end of the project. 4 Of a total of 10,923 persons who completed the survey, 10,154 employees were eligible for health benefits in 2005. 5 Fuller WA. Regression estimation for survey samples. Survey Methodology 2002; 28(1): 5-23. 6 McCall N, Khatutsky G, Smith K, Pope GC. Estimation of non-response bias in the Medicare FFS HOS. Health Care Financing Review 2004; 25(4): 27-41. 7 Kauttner J, Khatutsky G, Pope G et al. Impact of nonresponse on Medicare Current Beneficiary Survey estimates. Health Care Financing Review 2006; 27(4): 71-94.

Page 3

9/19/2011

Confidential

The auxiliary vector should explain the propensity for response to the survey; It should explain the main study variables (i.e., absenteeism and presenteeism); and It should closely identify the most important domains of the survey (e.g., health, disability, work productivity).8

When the first property exists, bias in the estimates is reduced for all survey variables. When the second property exists, bias is reduced only for the main survey variables but may not be reduced for estimates of other variables. However, when the auxiliary information explains both the propensity for response and the main survey variables, the variance in the survey estimates will also be reduced.9 Several statisticians and researchers have discussed approaches to minimizing nonresponse bias in surveys, including the incorporation of survey design features to maximize response, imputing missing data on individual survey items, and using multiple background variables to form non-response weights.10,11,12 Many methods of post-survey compensation for non-response fall under the category of calibration estimators, all of which rely on the use of an auxiliary vector of information that is known for the respondents but not for survey sample (here, the employees in one firm) or the target population (e.g., employees in all firms).13 Standard weighting class adjustment is one calibration-type method. Propensity weighting is another. Below, we describe each of these methods. Weighting class adjustment. In this approach, one divides the sample into a tractable number of subgroups based on a few background variables known for both respondents and non-respondents. Weights equal the inverse of the average response rate for each adjustment cell. One forms adjustment cells to minimize both bias and variance in the estimators, by choosing adjustment factors that form internally homogeneous groups having similar response rates.14 One problem with weighting class adjustment is that it relies on the assumption that a persons probability of response is equal to the response rate for the weighting class. Too few weighting classes can introduce variation of response rates within
8

Srndal C, Lundstrm S. Estimation in Surveys with Nonresponse. West Sussex, England: John Wiley & Sons, Ltd., 2005. 9 Srndal and Lundstrm (2005) 10 Kessler RC, Little RJA, Groves RM. Advances in strategies for minimizing and adjusting for survey nonresponse. Epidemiologic Reviews 1995; 17(1): 192-204. 11 Little RJA, Rubin DB. Statistical Analysis with Missing Data (2nd Edition). Hoboken, NJ: John Wiley & Sons, Inc., 2002. 12 Lohr SL. Nonresponse, In: Sampling: Design and Analysis. Pacific Grove, CA: Brooks/Cole Publishing Company, 1999; 255-287. 13 Srndal and Lundstrm (2005) 14 Little RJA. Survey nonresponse adjustments for estimates of means. International Statistical Review 1986; 54(2): 139-157.

Page 4

9/19/2011

Confidential

weighting classes and subsequent loss of precision in the survey estimates. Conversely, the use of too many background variables can lead to small weighting cell sizes and unstable weights. Because of these considerations we prefer not to use a weighting class adjustment approach. Propensity weighting. We recommend using a response propensity adjustment approach to weighting because it takes into account all available data and can reduce bias in the estimates without introducing too much variance in the estimates. Response propensity adjustment is an extension of the propensity score theory developed by Rosenbaum and Rubin.15 The response probability is estimated by fitting a regression model appropriate for binary dependent variables with the response outcome. At the most basic level, one weights survey observations by the inverse propensity for response, i.e., the estimated propensity score for each respondent. Our recommended approach is to weight by the inverse propensity for response for observations in all but the top and bottom tails in the distribution. For the top and bottom 5 percent, for example, we would use the within-group mean of the estimated response propensity to weight observations in the tails. Variations include sub-classification on the estimated propensity score to form smaller groups. Five classes are often sufficient to remove 90 percent of the bias.16

3 Data
The analytic data file will contain auxiliary information on all responding employees and nonrespondents, as well as the survey data for respondents only. Medical claims and enrollment data currently cover services incurred in calendar years 1999 through 2005 for people employed by the company in 2005. For purposes of this analysis, we will use the most recent 2 years of medical and disability history to construct variables reflecting each employees health and medical care experience in the time period leading up to Q4 2005, when the survey was conducted. Prior to determining the length of the time window for claimsbased variables, COMPANY will examine the proportion of employees who were continuously enrolled in 12 months and 6 months leading up to the survey. To minimize the number of employees that would be excluded from the analysis because of continuous enrollment criteria, COMPANY will consider the implications of different pre-period time intervals to construct variables. For purposes of this discussion, the analysis plan assumes use of a 6-month time frame.
15

Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika 1983; 70, 41-55. 16 Rosenbaum PR, Rubin DB. Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association 1984; 79(387): 516-524.

Page 5

9/19/2011 There are three types of data.

Confidential

First, there are data corresponding to parameters we wish to estimate for the total employee population but for which we have information for the survey sample only. The second type of data is auxiliary information available for the total population of employeesfor both respondents and non-respondents. This auxiliary information will be used to examine the inverse propensity for response and to develop a weighted estimation approach for obtaining population estimates of the parameters of interest. Auxiliary information will come from the eligibility, claims, and disability files. Variables available from auxiliary data include: demographic characteristics medical care utilization and cost measures claims-based measures of medical conditions disability leave

A third type of data includes other characteristics that are available for survey respondents but not for non-respondents. Examples are self-reported marital status, number of children, educational attainment, and occupation. Table 1 contains a description of key survey data elements.

Page 6

9/19/2011

Confidential

Table 1. Description of Key Survey Variables

Variable INTDATE CLAIMANT Description Date respondent completed web-based survey, in Q4 2005 COMPANY de-identified unique identifier for linkage with eligibility, claims and disability files QAGE Age, in years, at time of interview QSEX Gender Demographic and socioeconomic characteristics QMSTAT Marital status QKIDS Number of children OCCUPTN8 Description of job (8 categories) QEDUC Highest grade/level of education completed QINCOME Pretax annual income in 2005 U.S. dollars Health characteristics HEALTH Self rating of health (excellent, very good, good, fair, poor) TR_[condition]17 Do you have any of the following conditions? (No, yes-never treated, Yespreviously treated, Yes-currently treated) D7EXHOURS Typical number of hours expected to work in a week D7HOURS Total hours worked in the past 7 days Work loss from absenteeism and productivity ABS Absenteeism: A variable representing the ratio of hours actually worked to hours the employee was expected to work. (An absenteeism score of .9, for example, means the employee worked 10% less hours than s/he was expected to work.) PRES Presenteeism: A variable scaled as a relative measure of work performance compared to the average worker who does the same job as the respondent. (A presenteeism score of 1.1, for example, means the worker did 10% more work on the job than the average worker in the firm.) The variable PRES is an average of three presenteeism scores, each based on a different method.

Presenteeism scores Three presenteeism scores are included in the data. We will average the three scores to obtain one measure of presenteeism. The self-report method reflects respondent reports of their own overall work performance on a 0-10 scale compared to the work performance of the average worker on this job.

17

Conditions are allergy, anxiety disorder, arthritis, asthma, chronic bronchitis/emphysema, congestive heart failure, COPD, depression, diabetes, substance abuse, chronic fatigue, GERD, irritable bowel syndrome, migraine, obesity, cancer, chronic pain (excluding arthritis, migraine, headache), other emotional problems, osteoporosis, skin cancer, chronic sleeping problems, stomach or intestinal ulcer, urinary/bladder problem.

Page 7

9/19/2011

Confidential

The supervisor rating method is based on reports of a nationally representative sample of supervisors in the National Comorbidity Survey Replication (NCS-R). Supervisors were presented with hypothetical profiles of workers with various scores on the 0-10 work performance scale. The survey asked the supervisors how much they would be willing to pay each of these hypothetical workers. Lost productive value was then calculated by relating median supervisor responses to average wages within industry and occupation categories. The calibration method uses regression-based imputations of productivity based on surveys carried out in a number of occupations that had objective measures of work performance (e.g., telephone sales workers with objective data on sales volume). The self-report presenteeism reports were used as regressors in models predicting the objective reports; the coefficients from these models were then used to generate predicted values of objectively evaluated performance. Table 2 below compares respondents and non-respondents using selected characteristics: age, gender, and whether the employee is paid on an hourly basis (versus salaried). Table 2: Preliminary Comparison of Respondents and Non-Respondents
Respondents1 Number of Employees Mean Age Proportion Female
1 3

10,154 39.62 0.25 0.028

NonRespondents2 17,286 39.61 0.21 0.045

Pr > |t|

All Employees 27,440 39.62 0.23 0.038

0.92 <.001 <.001

Proportion Hourly Employee

We define respondents as employees who were eligible for health benefits in CY 2005 and who responded to the survey in Q4 2005. 2 Non-respondents are employees who were eligible for health benefits in CY 2005 and who did not respond to the survey in Q4 2005. 3 Two non-respondent observations were excluded from this comparison because of missing values on this variable.

This preliminary bivariate comparison indicates that female employees may have been more likely than male employees to have responded to the survey, and that employees paid on an hourly basis may have been less likely than salaried employees to have responded.

Page 8

9/19/2011

Confidential

4 Methods
4.1 Descriptive comparisons and analysis of response propensity
The planned work involves first a descriptive comparison of respondents and nonrespondents demographic, medical and disability characteristics to determine if respondents are systematically different from non-respondents in observable ways. We will conduct bivariate comparisons between respondents (n) and nonrespondents (N-n) according to characteristics which are observable for all employees (N) who were continuously enrolled for six months during the period April 1, 2005 through September 30, 2005. Comparisons between respondents and nonrespondents will include the following variables.
Employee and family characteristics Age Proportion female Proportion hourly Proportion with covered adult dependent Number of child dependents under age 18 Medical conditions in prior 6 months Charlson comorbidity index (CCI) With 1 diagnosis [of selected conditions]18,19 Number of diagnoses (unique 3-digit ICD-9 codes) Health characteristics of family members other than employee

Family member hospitalized in prior 6 months Family member with mental health diagnosis
Family member with CCI 1

Medical utilization in prior 6 months

With 1 inpatient stay Inpatient days With 1 ER visit With 1 physician visit Six-month health care costs, by category of service Hospital inpatient Emergency department Hospital outpatient
18

In descriptive comparisons, COMPANY will include conditions that have been shown in research to be strong predictors of absenteeism and presenteeism. For example, Kessler et al. (2001) study the effects of six significantly interacting conditions on 30-day work impairment. These conditions were arthritis, ulcer, major depression (MDD), panic, generalized anxiety disorder (GAD), and substance dependence. COMPANY will also include conditions of interest to the Client: MDD, fibromyalgia, GAD, low back pain, and diabetic peripheral neuropathy. 19 Kessler RC, Greenberg PE, Mickelson KD, et al. The effects of chronic medical conditions on work loss and work cutback. JOEM 2001; 43(3): 218-225.

Page 9

9/19/2011
Outpatient physician visit Outpatient, all other Total medical costs Prescription drug Total health care costs

Confidential

Six-month hospital costs for mental health and physical health care Mental health diagnosis on hospital inpatient, ER, or hospital outpatient claim No mental health diagnosis on hospital inpatient, ER, or hospital outpatient claim Six-month outpatient provider costs for mental health and physical health care Psychiatric diagnosis on non-hospital outpatient provider claim No psychiatric diagnosis on non-hospital outpatient provider claim Work loss in prior six months With 1 disability leave Disability days

Using maximum likelihood methods (e.g., logistic regression) we will estimate the propensity for response, using the eligible employee population and characteristics observed for both respondents and non-respondents. We will test several nested models, using likelihood ratio tests to assess how groups of predictors (e.g., demographic characteristics, health characteristics) and interactions among predictors perform in the models. Milestone: Tables of descriptive comparisons and regression results

4.2 Weighting
Using a response propensity adjustment approach to obtain weights, we will construct weights by fitting a logit model with the response outcome regressed on selected claims-based measures available for both respondents and nonrespondents. First, we will test several models to find the most parsimonious specification. One way of optimizing model fit is non-orthogonal partitioning, or cross-validation. Using this method, we will apply results from regression models estimated on one part of the sample (i.e., 75 percent of the observations) to the remaining observations. We will then examine how well the estimated model coefficients from the first partition predict actual responses in the second partition, and repeat the process with different sets of regressors until finding the optimal specification. This can be done with a forward stepwise regression. The optimal model maximizes the resulting adjusted R-squared across specifications.

Page 10

9/19/2011

Confidential

Secondly, from the best model specification we will then obtain the estimated propensity for response, pi :

pi =

exp xi 1 + exp x

( ) ( )
i

Finally, after pi is estimated for each unit in the study sample using the optimal set of predictors in the response propensity equation, we will use this information to develop calibrated estimates of survey outcomes.

One approach is to directly weight the estimates by the inverse propensity for response (1 p i ) . We will try this approach first. Sometimes this method can result in extreme values for the predicted probabilities of response, requiring additional trimming adjustments by imposing bounds on the adjustment factor. Alternatively, classes of weights can be formed by dividing the study sample into k groups based on the distribution of the estimated response probability pi for each k . The number of respondents in class k is multiplied by wk = 1 p k class k to get p , the weight factor for class k. This is a sub-classification on the propensity score, e.g., into quintiles, for example. Our preferred approach to weighting involves a combination of the above weighting methods. For most of the samplethose that have estimated propensity scores in the middle 80 to 90 percent of the distributionwe will weight observations by the inverse propensity for response. For extreme estimated propensity scores, i.e., those at the tails of the distribution, we will average the weights among observations in the tails. Milestone: Tables of weighted and unweighted estimates

4.3 Assessment of Bias and Measurement Error

After developing regression-based adjustments for nonresponse and applying the weighting system to key variables of interest, we will then investigate the extent that weighting adjustments improve estimates. Given an overall response rate of around 40 percent, it is likely that we will see reductions in bias with the application of weights derived from a propensity regression approach. Kessler and colleagues note that variance estimation can be problematic because the weighting scheme introduces a component of variability not otherwise present.20,21 Adjustment procedures such as weighting tend to increase standard errors of sample
20

Kessler RC, Little RJA, Groves RM. Advances in strategies for minimizing and adjusting for survey nonresponse. Epidemiologic Reviews 1995; 17(1): 192-204.

Page 11

9/19/2011

Confidential

statistics, making it important to consider the tradeoff between reductions in bias and decreased efficiency of the estimators. To assess this tradeoff, we will examine the impact of our choices of models and weight trim cut-points on the mean squared error (MSE)22 of survey estimates. We will decompose MSE to assess bias and variance in the estimated absenteeism and presenteeism scores. A table shell shown below provides an example of how we will present results of such an analysis (Table 3). To further assess how adjusting for response bias affects estimates, COMPANY will examine how a particular condition, depression, influences absenteeism and presenteeism. COMPANY will compare adjusted and unadjusted estimates of the impact of depression on absenteeism and presenteeism. This analysis will explore how the condition is related to both the probability of responding to the survey and to the outcomes of interest. In summarizing the results in the final report, COMPANY will discuss the implications of adjusting or not adjusting survey estimates for potential biases arising from nonresponse. Table 3: Effects of trimming on bias and efficiency of absenteeism estimates
Percent of trimming in each tail 0 1 2 3 4 5 6 7 8 9 10 [Bias (y)]2 Inefficiency = Var (y) MSE = [Bias (y)]2 + Var (y)

Milestone: Set of tables showing the effects of weighting on bias and efficiency of absenteeism and presenteeism estimates, including results of the depression analysis

21

Kessler RC, Berglund P, Chiu WT et al. The US Comorbidity Survey Replication (NCS-R): design and field procedures. International Journal of Methods in Psychiatric Research 13(2): 69-92.
22

MSE = bias 2 + ( SE )

Page 12

9/19/2011

Confidential

5 Timeline
Task Project Start-up, including Analysis Plan Response bias analysis: Database1 Adjustment for nonresponse: Database1 Assessment of Adjustment: Database1 Response bias analysis: Database2 Adjustment for nonresponse: Database2 Assessment of Adjustment: Database2 Final Report Executive Summary Aug '06 Sep '06 Oct '06 Nov '06 Dec '06 Jan '07

2a 3a 4a

2 b 3 b 4 b

5 6

Page 13