Publishing Date: November 2003. © 2003. All rights reserved.

Copyright rests with

the author. No part of this article may be reproduced without written permission
from the author.

Web Evolution

Mail vs. On-line Research: Deciding Between Cost and

Quality

by F. Annie Pettit and Leanne Lam

The Internet is truly a viable market research tool that can

save time and money. Until recently, market research has
relied heavily on paper-based surveys. Consequently, we have
many years of experience in how paper survey results calibrate
with subsequent purchase behaviour. Now, it is imperative to
determine how paper-based survey results compare to on-line
survey results. Although on-line surveys are formatted quite
differently than mail surveys, the best test of the on-line
method is a comparison between a typical paper questionnaire,
and a typical on-line questionnaire. This paper will review two
case studies. The first is a comparison in incidence rates, while
the second is a comparison of efficacy, or test control,
research.

Case 1: Incidence Rates

In Spring 2002, survey responses from over 4 million
randomly selected American and 550,000 Canadian
households were collected using a four to six page survey of
branded purchase and use of household packaged goods,
direct response questions, and demographic information.

Three months after completing this survey, some responders

received a second survey consisting of some of the same
questions. Responders were randomly assigned to receive
either a mail-out or a web survey. Approximately 2500 on-line
and 6000 mail-out surveys were successfully completed and
matched back to the original data.
The initial survey was used to determine base rates, i.e., the
percentage of households that endorsed a particular item. The
next step was to determine the percentage of households that
endorsed an item on the second survey (three months post).
The match rate was calculated as the percentage of
households endorsing the item on the second survey, out of all
households endorsing the equivalent item on the first survey.
An index was calculated as the mail out match rate divided by
the on-line match rate.

Results:
Each analysis contains at least 200 responders in each group.
The average US index value was approximately 100, indicating
that mail and web survey results matched back to the original
paper-based survey data in a similar way (Selected variables
are shown).

Index>100 = Mail out has

US
higher Match Rate
Average Index 100
Arthritis Common 122
Arthritis - Rheumatoid 104
Asthma 112
Bladder leakage - Any 104
Bladder leakage - Light 96
Buy allergy remedy 5 or more times per
101
year
Cold Sores 97

On the other hand, the average Canadian index value was

approximately 109, indicating that the paper-based survey
matched back to the base survey better than did the on-line
survey (Selected variables are shown).

Index>100 = Mail out has higher Match

Canada
Rate
Average Index 109
Arthritis 105
Asthma 108
Back pain 105
Body aches 108
Dairy digestive difficulty 110
Migraine 101
Osteoporosis 120

Case 2: Efficacy Research Comparison

Efficacy research is used to determine the effect of sampling
and couponing programs on brand awareness and market
share. On-line and mail surveys are often used to do this, and
in this case, the offers were for a nutritional supplement and a
personal care paper product.

Test households received an offer (sample and/or coupon)

while control groups did not receive an offer. Test and control
groups received the same questionnaire weeks later that
included a number of standard items such as branded
purchase history, purchase intent, recall of the offer, and
demographics.

Table 3. Percentages for each comparison measure

On-line Mail
Nutritional Control Test Ratio Control Test Ratio
Supplement % % % %
Recall 23.6 45.4 192 30.9 46.9 152
receiving
an offer
Recall - 36.1 - 40.9
receiving a
sample
Recall - 73.5 - 72.7
brand
correctly
Purchase 6.3 10.1 160 9.2 13.5 146
P3M
Purchase 7.9 9.9 125 11.4 14.1 124
intent
Personal Control Test Ratio Control Test Ratio
Care Paper % % % %
Product
Recall 13.0 14.9 115 16.1 19.2 119
receiving
an offer
Recall - 25.8 - 27.6
brand
correctly
Purchase 7.1 9.9 139 11.4 9.6 84
P1M
Purchase 10.6 11.5 108 9.9 6.7 70
intent
* Numbers have been modified to protect client confidentiality. Overall trends, however,
remain intact.

Results:
Table 3 contains all of the comparison measures. On-line and
mail results can differ in two distinct ways. First, the data
could differ in terms of percentages. For instance, 45.4% of
the on-line Nutritional Supplement Test group recalled the
offer, whereas 46.9% of the off-line group did. It appears that
the Offline group recalled the offer similar to the on-line group.
Second, the data could differ in terms of relationships between
variables. In this case, taking false recall percentages into
consideration shows that the on-line group did in fact achieve
better results (46.9-30.9=16.0 vs. 45.4-23.6=21.8). As a
result, payout calculations would be very different depending
on whether on-line or off-line results were obtained.

The trends in the results for the on-line and off-line nutritional
supplement research were similar; test percentages were
larger than control percentages. This leads to the conclusion
that either on-line or off-line research would have generated
workable results.

For the personal care product, however, the conclusions that

would have been drawn based on only on-line or only off-line
are very different. For instance, the past one month purchase
ratio for the on-line research suggests that the offer was a
success, whereas the off-line research suggests that the offer
was not a success.

Table 4. Usage and Attitudes Comparison Measures

On-line Mail
Study 1. Control Test Control Test
Nutritional
Supplement
Female 79.3 % 76.6 % 79.6 % 80.9 %
responders
Average age 55.8 56.8 63.7 66.6
Would not try - 4.0 - 4.4
without offer
Learned - 3.7 - 4.0
something new
Would - 3.5 - 3.8
recommend to
someone
Coupon value - 3.9 - 4.4
was good
Study 2. Control Test Control Test
Personal Care
Product
Female 85.9 % 86.9 % 95.3 % 95.0 %
responders
Average age 36.6 38.8 46.5 46.3
Would - 3.1 - 3.9
recommend
Soothing on skin - 3.3 - 4.4
Works better - 2.8 - 3.5
than usual brand

Table 4 shows results that relate more to usage and attitude

types of data. We can see that for age, gender, and attitude
mean scores, the on-line numbers were consistently lower.
Although the values are not equal and may result in different
significance values, the conclusions that would be drawn are
the same. In effect, the on-line and off-line results provide
similar learnings.

Conclusion
On-line research is quickly evolving. However, we need to be
conscientious about how information is interpreted, as on-line
efficacy results do not necessarily match mail efficacy results.
It is very clear that base rates can easily be different, although
they may be directionally similar. Until more on-line historical
data is available, we must be mindful not to draw conclusions
that on-line research is necessarily comparable to off-line
research.

Annie Pettit is a Research Scientist at ICOM Information &

Communications Inc.. She can be reached at 416-297-7887 or
apettit@I-com.com.

Leanne Lam is a Project Coordinator and can be reached at

llam@I-com.com.

© 2003. All rights reserved. Copyright rests with the author. No part of this article
may be reproduced without written permission from the author.