Analyzing Survey Data

Angelina Hill, Associate Director of Academic Assessment

2009 Academic Assessment Workshop
May 14th & 15th
UNLV

Prior to Analysis
What would you like to discover?

Perceived competence
Preferences, satisfaction
Group differences

Demographics

What are your predictions?

Prior to Analysis
Your goals drive the make-up of the survey

and how it should be analyzed.

Exploration can be informative, but with an

analysis plan.

Prior to Analysis
Survey design & layout

Stylistic considerations are important because they

increase response, validity, and reliability

Survey Design
Good questions reduce error

By increasing the respondents willingness to

answer

Increases reliability and validity.

Less error = better data

Reliability & Validity

Reliability Is the survey measuring

something consistently?

Typically measured using Chronbachs alpha

Validity Is the survey measuring what its

supposed to be measuring?

Typically measured using factor analysis

Construct Validity
Does your measure correlate with a theorized

concept of interest?

Correlate measure with values that are known

to be related to the construct.

Pilot
Piloting the survey can inform:

Question clarity
Question format
Variance in responses

Survey Analysis
Using data from

Paper Surveys
SurveyMonkey
SelectSurvey.Net

Survey Analysis
Paper surveys

Put data in spreadsheet format using excel or

SPSS

Columns represent variables

Rows represent respondents

Survey Analysis
Paper surveys

Create a data matrix

Variable name || Numeric Values || Numeric labels

Summarize open-ended questions separately

Response group || frequency

Survey Analysis
SurveyMonkey

Available under the analyze results tab

Frequencies & crosstabs

Download all responses for further analysis
Select Download responses from menu
Choose type of download select all responses
collected
Choose format select condensed columns and
numeric cells.

Survey Analysis
SelectSurvey.NET

Available under Analyze Results Overview

Frequencies
Download all responses for further analysis
Select Export Data from Analyze page
Export Format CSV (excel)
Data Format SPSS Format Condensed

Data Cleaning
Process of detecting, diagnosing, and editing

faulty data
Basic Issues:

lack or excess of data

outliers, including inconsistencies
unexpected analysis results and other types of
inferences and abstractions

Data Cleaning
Inspect the data

Frequency distributions
Summary statistics
Graphical exploration of distributions
Scatter plots, box plots, histograms

Data Cleansing
Out of range
Delete values and determine how to recode if possible
Missing Values
Refusals (question sensitivity)
Dont know responses (cant remember)
Not applicable
Data processing errors
Questionnaire programming errors
Design factors
Attrition

Missing Data
Missing completely at random (MCAR)

Cases with complete data are indistinguishable from

cases with incomplete data.

Missing at random (MAR)

Cases with incomplete data differ from cases with
complete data, but pattern of missingness is predicted
from variables other than the missing variable.
Nonignorable

The pattern of data missingness is non-random and it

is related to the missing variable.

Missing Data
Listwise or casewise data deletion: If a record has missing data

for any one variable used in a particular analysis, omit that entire
record from the analysis.
Default in most packages, including SPSS & SAS

Pairwise data deletion: For bivariate correlations or covariances,

compute statistics based upon the available pairwise data.

Useful with small samples or when many values are missing

Substitution techniques: Substitute a value based on available

cases to fill in missing data values on the remaining cases.

Mean Substitution, Regression methods, Hot deck
imputation, Expectation Maximization (EM) approach, Raw
maximum likelihood methods, Multiple imputation

Descriptive Statistics
Frequency distribution

Descriptive Statistics
Cross-tabs

Excel Pivot tables

Excel menu Data PivotTable and PivotChart

PivotTable menu Field setting summarize by
count show data as % of row or column

Data Analysis
Measurement scale determines how the data

should be analyzed:

Nominal, ordinal, interval, ratio

Move from categorical information, to also

knowing the order, to also knowing the exact
distance between ratings, to also knowing that
one measurement in twice as much as
another.

Data Analysis
Three instructors are evaluating preferences

among three methods (lecture, discussion,

activities)
1) Identify most, second, and least preferred.
2) Identify your favorite.
3) Rate each method on a 10-point scale,
where 1 indicates not at all preferred and 10
indicates strongly preferred.

Data Analysis
Nominal & ordinal variables are discrete

Can be qualitative or quantitative

Interval & ratio variables are continuous

Grades
Age

Data Analysis
Charts

Pie charts & bar charts

used for categorical
data
Histograms used for
continuous data
Line graphs typically
show trends over time

Data Analysis
Other descriptive statistics

Mean

Median

preferred, uses all of the data

ordinal data
open-ended scale
outliers

Mode

nominal data

Data Analysis
Other descriptive statistics

Interquartile range

Variability accompanying the median

Standard deviation

Variability accompanying the mean

Correlations
Are the variables related?

Determine variables that relate most to your

item of interest

Correlate Likert-scale questions with each other

Correlate interval/ratio demographic information
(e.g., age) to Likert-scale questions

Correlation
Which correlation coefficient to use?

Pearsons r

Used with interval and ratio data

Spearman & Kendalls tau-b

Used with ordinal data

Spearman used for linear relationship
Kendalls tau-b for any increasing or decreasing
relationship

Mean Differences
Are there meaningful differences between

groups?
class sections
instructors
on-line vs. off-line courses
major vs. non-major

Mean Differences
Which test to run?

Interval and ratio data

t-test when comparing 2 groups

Independent
Dependent (paired-samples in spss)

ANOVA when comparing > 2 groups

Independent (One Way ANOVA in spss)
Dependent (general linear model-repeated measure
in spss)

Presenting Results
Describe the purpose of the survey

List the factors that motivated you to conduct

this research in the first place.

Include the survey!

On assessment reports
When the survey is new/still being fine tuned

How it was administered

Presenting Results
Present the breakdown of results

Tables and graphs should complement text

Conclusions

Explain findings, especially facts that were

important or surprising

Recommendations

Describe an action plan based on concise

concluding statements

Presenting Results
Share results in formal venues
Familiarize yourself with key findings so that

you can mention results at every opportunity

Moving Forward
Continuously improve the survey

Delete, add, change questions

Evaluate method of administration

Compare results across semesters to look for

improvements
Compare with other assessment data for a
broader picture