Moderator Guide:

October 2016 Physician Compare Usability Testing

Session 2; 10/17 10am
URL: https://imp.medicare.gov/
https://imp.medicare.gov/physiciancompare

ONCE YOU GET STARTED: Im going to read through some things so bear
with me

Introduction
Hi, and thank you for agreeing to participate in this session today. My name
is <FIRST and LAST NAME>, and I work for Westat, a research company
based in Rockville, MD. Today Id like to get your reactions to a website that
allows users to search for different doctors and other clinicians who provide
medical services to people enrolled in Medicare.
Westat is conducting an evaluation of the Physician Compare website for the
Centers for Medicare and Medicaid Services (CMS), the Federal agency that
oversees the Medicare program. Physician Compare is currently available
through Medicare.gov, but today well be looking at a redesigned version of
the Physician Compare website. Were interested in evaluating potential new
pages for the website and identifying areas for improvement.
Your responses will be kept confidential. For our report, the comments that
you share will be grouped with comments from other people like you who are
providing feedback. Your name will not be associated with anything you say.
Todays session is completely voluntary. If youd like to stop at any time,
please tell me and well stop immediately. If there are any questions you
dont want to answer, just tell me and we can skip them.
Some team members working on this project and Medicare representatives
are logged into this session as observers. Wed like to record this session so
we can make sure we dont miss anything. Because well be recording,
please try to speak in a voice at least as loud as the one Im using now so
that we can make sure the tape is picking up our voices.

October 2016 Moderators Guide

Page 1

Consent
Interviewer Instructions: Show consent form on screen (Consent form will
be sent to respondent in advance of session). READ THE FOLLOWING ALOUD
TO PARTICIPANT:
Westat is a private company conducting this project on behalf of the
Centers for Medicare and Medicaid Services (or CMS).
We will be asking you to review a set of sample web pages to obtain
your feedback.
The session will last approximately 60 minutes.
Your participation is voluntary, and you have the right to stop at any
time or to refuse to answer any question.
The things you say may be put in a written summary of this discussion,
but your name will not be used anywhere.
We would like to record the interview in case we need to review a
portion of the session as we make recommendations for the website.
The recording will only be reviewed by those working on the website.
You will be sent a check for $60 for your participation.
Interviewer instructions: Answer any questions. Obtain consent to
proceed and to record.
Do we have your consent to record the session?
EXPLAIN FEATURE TO RECORD, TURN IT ON, AND THEN
OBTAIN CONSENT AGAIN SO THAT CONSENT IS RECORDED.
Okay, now that we are recording: Today is [DAY AND TIME]. Do we have your
permission to record, and do you agree to participate in this study?
Im going to ask you to do some tasks using the website. There are no right
or wrong answers. I am simply interested in how you go about completing
each task and how easy or difficult you find the site to use. Its important
to know that we are not testing you were testing possible new pages for
this website and identifying potential ways to improve them. If something
that we go through today is unclear or confusing, its important to share this
with me since its likely to be confusing to other people, too. I didnt design
the site, so you wont hurt my feelings if you criticize it. All of your positive
and negative feedback is welcome. Well save some time at the end of the
session to cover your general impressions.

October 2016 Moderators Guide

Page 2

One last thingthis website is NOT fully developed at this point. If you try to
get toor land onsomething that is not yet developed, Ill tell you.
Do you have any questions before we begin?
-

BRING TEST SITE UP ON SCREEN

PASS MOUSE CONTROL
ENSURE THAT PARTICIPANT THINKS FONT SIZE IS OKAY ON HIS/HER
SCREEN
MENTION THAT THERE MIGHT BE A BIT OF A DELAY WHEN CLICKING ON
BUTTONS OR LINKS DUE TO USE OF WEBEX.

Before we begin with the tasks, Im going to let you use, and familiarize
yourself with the website for about 5-10 minutes. Feel free to use the site as
you normally would if I wasnt here, click around, and explore the different
features. Ill be available if you get lost at any point. If you could, just think
aloud as you work.

Tasks
Now Im going to ask you to do some specific tasks. As we get started, here
are some things to keep in mind:

Ill read each task out loud to you then work on the task as if you were
doing it alone at home without me.
Feel free to talk aloud as you work.
When you find what youre looking for, let me know that youre done
with the task. Then, Ill tell you when you can move on to the next
task.

IF THE PARTICIPANT IS COMPLETELY CONFUSED ABOUT HOW TO RETURN TO

THE HOMEPAGE, THEN SET THE HOMEPAGE FOR THEM WITH THE BOOKMARK
OR LINK TO HOMEPAGE. ON SUBSEQUENT TASKS, GIVE ANOTHER CHANCE
OR TWO TO FIGURE OUT HOW TO GET TO HOMEPAGE WHEN THEY START
THE NEXT TASK.
IF PARTICIPANT CANNOT FIGURE THIS OUT ON OWN AFTER THE THIRD
TASK, THEN SAY AND SHOW:
Each time you are asked to start from the homepage, you can either select
the Physician Compare Home link at the top right of the page.
[TAKE MOUSE CONTROL AND POINT TO PHYSICIAN COMPARE HOME LINK].

October 2016 Moderators Guide

Page 3

October 2016 Moderators Guide

Page 4

As yourself (Self-directed Tasks)

1) You are interested in finding a new primary care doctor for
[yourself/the person you care for].
R looked up rheumatology and nothing appeared, was unsure to click
the blue box next to the name. Once instructed to do so, R got a list of
doctors, sounded impressed.
Would be interested to see if doctors on the site are accepting new
patients.

a. Starting on the home page, find two doctors who you consider to
be in a good location for [you/the person you care for].
i. Typed in the location bar Washington DC and searched
primary care. She then selected the blue box to go to a
page that showed internal medicine doctors
ii. Added William Harrison MD and Annick P Hebou to
compare, and was going to click compare
iii. R expects to see biographical information, where they went
to school, and qualifying information.
b. How would you compare these doctors?
i. R adds two physicians to the compare tab and clicked the
green compare button
2) Think of your own medical condition(s) or a health issue of interest to
you [or the person you care for].
a. Starting on the homepage, find a doctorother than your [or
his/her] own current doctorwho could give you a second
opinion about alternative treatments you might consider.
i. R searches by just typing in Washington DC and selects
another doctor to see his information; sees where the
doctor is located, educational background
ii. R wants to know how old the physician is; it is important
for her to know
iii. Expects to see information on how this doctor does
things when asked the clinical quality of care tab. R
October 2016 Moderators Guide

Page 5

clicked on this and sees the measures; thinks its a

percentage of this doctor stresses. Likes that this
information here is detailed.
iv. Does not tell the presenter that she would click show to
find out more information about one of the measures
v. R only clicks the back button to get to the homepage
instead of utilizing other options to do so

b. Click on Dr. [USE DOCTORS NAME]s name.

c. Take a minute to look over this page. What can you tell me about
this page and the information available?
i.
Pretend that (Standard Tasks)
1) Using [BALTIMORE, MD] as your location, type in [BACK] in the search
box and select search.
i. R says there are all different approaches if she has a
back problem, she would look for the specialty most
needed for her
b. Walk me through what you are seeing on this page.
i.
c. You decide you only want to look at [PHYSICAL THERAPISTS] and
[CHIROPRACTORS].
d. After looking at this list, how would you narrow your results to
only see doctors who accept Medicare Assignment?
i. She finds the medicare assignments tab and clicks to
narrow down her results.
2) Starting from the Home Page, pretend that you do not know how to
spell PREVENTIVE MEDICINE. Without using the main search box, how
would you find Preventive Medicine doctors in [ROCKVILLE, MD]?

October 2016 Moderators Guide

Page 6

a. R: If I didnt know how to spell it R identifies search by

specialty. Found it in list and clicked on it. Clicked on the letter P
and found Preventive Medicine
a. What can you tell me about where Dr [JULIANA ARUNA] works?
i. R: She is with the Cornerside Montgomery inc in Rockville.
b. Using the map on the right, find the location of this office.
i. R selected blue pin.
c. Click on [CORNERSTONE MONTGOMERY INC.] name.
d. Take a minute to look over this page. What can you tell me about
this group practice based on what youre seeing here?
i. R: (clicked on general information tab) Under general
information they are involved with HER, they reported
quality measures, and are committed to Million Hearts
initiative, gives her the feeling that they are in the cutting
edge they are not old fashioned.
ii. They are into these aggregating thingswhere your
information is not just about you and it goes into a system
and they start looking at populations and things like that.
iii. Clicks on affiliated health care professionals tab to find the
doctors who work there
e. Looking at this page, can you find [KRISTINA G. ABBOT]?
i. N/A
f.

Click on Dr. [KRISTINA G. ABBOT]s name.

g. Take a minute to look over this page. What can you tell me about
this page and the information available?
3) Pretend that youre experiencing pain in your foot, but you arent sure
what type of doctor to see.
a. If they already noticed Search by body part: Using the
search by body part button on the home page, find a doctor to
help with your foot pain.

October 2016 Moderators Guide

Page 7

i. She would type her city and then would click search by
body part. R chooses the woman option and clicks on feet
and clicks next.
ii. R says the images are weird and x-ray and opaque
looking says the highlight color should be more vibrant.
b. If they have NOT noticed Search by body part yet: Starting
from the home page, how would you go about finding a doctor
for your foot pain?

**If running short on time: Stop task portion of session

with about 25 minutes remaining to allow time for
questionnaire and then at least 15 minutes of
debriefing.**

Brief Quantitative Questionnaire

[Show short questionnaire to participant on screen and ask her/him
to complete it. Then save again when participant has finished. (Word
document pre-saved to correct project directory and folder with participants
number)]

October 2016 Moderators Guide

Page 8

Debriefing
(Cover points noted during task performance first, then summary reactions
to site pages and features.)

Task Performance

Were there any tasks that you werent sure how to do? Why was that?
o No, R says it was very straightforward and not difficult to do
[FOR ANY TASK THAT YOU NOTED A USABILITY ISSUE FOR THAT
PARTICIPANT DID NOT ALREADY COVER]:
o I noticed that on Task # __, you [FILL WITH WHAT YOU
OBSERVED]. Tell me about what you were thinking there

At any time, did you feel you needed some type of help in order to
figure out what to do next?
o [IF NEEDED]: Where was that? What were you doing at that time?
o What would you do to get help with the website if you were on a
website like this at home on your own?

Homepage

When you first looked at this page, what did you notice? Tell me about
that What made you notice [FILL BASED ON RESPONDENTS
ACTIVITY DURING TASKS]?
o R: It was trying to solicit more specific information from her
which is smart, a lot of these things just has generic boxes; it
drew R out, especially for someone who is not knowledgeable
about some of these things and it also gave you a way to look in
a couple different ways.

What are your thoughts about other areas of the homepage, the
[COVER ANY ELEMENTS NOT PREVIOUSLY MENTIONED]
o Header, More Medicare Compare Websites, Resources, Carousel,
etc.
They are different ways to find those solutions; its different
kinds of searches and drills down on more specificity; its
not just a doctor thing it offers other health care
professionals part of the equation. A lot of patients do not
know where to go when they are looking, so this is helpful.

October 2016 Moderators Guide

Page 9

 Main search box:

[POINT OUT SEARCH BOX]

How easy or difficult was it to use the search box to find what you were
looking for?
o It was not difficult to find or use the search box.
o Did you know that you could search for names, specialties, or
medical conditions?
Out of these options, how would you normally search for a
doctor on a website like this?
R: She would normally type in the specialty, she
knows what ones she is looking for; likes that it gives
her multiple ways to search.
o When you searched for [back] from the home page, you were
brought to this screen:

[BRING UP INTERIM RESULTS PAGE]

What did you think about this page?

R: Thinks it is super helpful because it breaks down
all the ways you could go; everyone diagnoses
themselves on the internet so this is helpful, because
it lists all the options; likes how doctors are
quantified under these specialties.

Browse by specialty (A-Z):

IF THEY HAVENT USED BROWSE BY SPECIALTY YET:

Did you see this button on the home page?

o What do you think youll find when you click on this?
[CLICK ON BROWSE BY SPECIALTY BUTTON]
o What do you think about this page?
R: Very straight forward. The alphabet at top which is
great. You have the geographical area which is great. And
you always give the numbers which is good.
o What do you think the numbers in parentheses mean?

Search by body part:

October 2016 Moderators Guide

Page 10

IF THEY HAVENT USED SEARCH BY BODY PART YET:

Did you see the Search By Body Part button on the home page?
o What do you think youll find when you click on this button?
R: You pretty much know what you are looking for but I
guess if you didnt this helps you. If you are having foot
pain, you click foot pain and it will break it down by
specialty and it will eventually get you to a list of doctors,
by virtue of using it as a research tool
o Would this type of search be helpful to you when looking for a
new doctor?

Results page

When you got to a Results page, did you see what you expected, or
not? Tell me more about that
o Were the results listed in the way you expected?

R: Yes. I appreciate that they had accepts Medicare

assignment.

R: It gives you geography, accepts Medicare. I can usually

tell if this one place has one podiatrist.. Yes, it would be
what I expect.

o PROBE: From the information provided on this page, can you tell
which doctors are part of a group practice?
[DID THEY NOTICE THAT GROUPS AND INDIVIDUALS WERE
LISTED TOGETHER?]
o What do you like about the way results are presented?
This site aggregates a lot of things in one place for me,
rather than.. Im thinking of such and so place, you have to
really drill down on that Dr. you may have to google some
information for yourself. This is all in one place and thats a
good thing.
o What, if anything, do you suggest changing about the display of
results?
R: As I said it does aggregate more information than I have
seen elsewhere and I think its a dramatic improvement
over some of the others. Thinks it is good for the most part.
October 2016 Moderators Guide

Page 11

 Performance scores available Icon:

IF THEY DONT MENTION THE PERFORMANCE SCORES AVAILABLE ICON
DURING THE TEST:
o What do you think this icon means?
What would you expect to see on these groups and
doctors profile pages?
o IF THEY CHOSE A DOCTOR WITH THE ICON DURING A TASK,
PROBE: I noticed you selected a doctor with this icon earlier; how
did this icon influence your decision?
Filter menu:

What are your thoughts on the Filter Menu at the top?

o Did you notice this here?
o What do you think of the types of choices you have to narrow
your results?
R: extremely helpful. Distances and accepting of Medicare.
Board certified and whether or not this person works alone
or in a group practice, thats good info. Easy to use.
R: All of the type of text is same. Maybe distinguish it by
possibly bolding it, and other color and font choices.
o PROBE: Are the filter options clear or unclear?

Map
[IF THEY MENTIONED THE MAP ALREADY]
o Did you think the map was helpful or not helpful for finding a
doctor?
R: it is what it is technology wise. It does bring up that
specific practice or doctor with phone number, locations;
its pretty standard.
[IF THEY DIDNT MENTION THE MAP]
o Did you notice the map on this page?
o Did you notice the pins on the map changed color when you
hovered over the doctors name?
R: As I was going along, that is what it would have popped
up. Maybe it was a more specific answer to the query I was
making. It became green for a reason.
October 2016 Moderators Guide

Page 12

R: Ohh so when I choose it, it will turn green.

Because sometimes they are on top of each other
and that is a really good thing.
o Do you think the map would be helpful or not helpful for finding a
doctor?
Results Page Search Box

[IF THEY USE/MENTION THE SEARCH BOX]

Did you notice the Search Box at the top of this page?
Do you think this would be helpful or not helpful when searching for a
doctor or a group practice?

Compare Feature

[IF NOT COVERED DURING THE TASK PORTION]

When you selected 2 doctors to compare at the beginning of the

session, what information did you expect to be able to compare?
o Would the information available to compare be helpful when
choosing a doctor?

Profile pages
Clinician Profile Pages

When you got a doctors profile page, did you see what you expected,
or not? Tell me more about that
o Was the information available what you expected to see, or not?

R: Yes

o What do you like about the way profile pages are presented?
o What, if anything, do you suggest changing about the display of
information on doctors profile pages?
R: Again, I would think a little more about the font or the
size being somewhat different. Maybe make the topics
more block or uppercase. Its just all the same, too much.
Needs to be distinguished more
R: The Accepting New Patients thing. Almost impossible b/c
it changes all of the time. Maybe you have something that
says when they are not accepting new patients.
October 2016 Moderators Guide

Page 13

R: How long it takes to get an appt. and I dont know if that

is your deal at all. You may have to call.

Group Practice Profile Pages

When you got a group practices profile page, did you see what you
expected, or not? Tell me more about that
o Was the information available what you expected to see, or not?
o What do you like about the way profile pages are presented?
o What, if anything, do you suggest changing about the display of
information on group practice profile pages?
You explain what anesthesiology is. It came from that other
page by specialties. If you didnt know what it is, you
would know what an explanation is.

Sticky Header

Did you notice that the contact information for the group practice
remained at the top of the screen when you scrolled down the page?
o Yes the top stays static. It was clear.
o Is this something that would be helpful, or not helpful?

R: Well its helpful to know you are not lost. If you get
bogged down with your results, you can retrace your steps
without completely closing yourself out which is a good
thing. If you are not good with technology, this helps. The
older you get, the more you need this technology thing so
dont resist.
R: Attention to patient medication cost!! That is a really
good item especially if you are older with Medicare. That is
a really good question to ask if you ask to create the page.
If someone considers asking a Dr this question, that is a
very important thing. A lot of people dont get care b/c
they cant afford it. That is a really good element to look
for. A really good thing.

Overall Reactions

Overall, what one or two things would you say you:

October 2016 Moderators Guide

Page 14

o Liked MOST about the website?

It was easy to use. Static situation and being able to

manipulate infor in it without having to start all over this
is really good. I think one of the big things is that if I didnt
know what podiatry is, I could look up feet. That is a really
bit hallmark of this site. A lot of people there is no way to
translate what they need or where to get help. You cut that
out right off the bat. That is how it is for people older ppl,
people with disabilities. You short circuited that. they may
not even know what to ask.

R: You aggregate so much more than other sites.

o Liked LEAST about the website?

R: The people are weird.

The graphics are less than what they could be.

In your opinion, what is most in need of change/improvement to make

this website easy for you to use?
o R: I would probably say the look of it. Giving the information in a
more impactful way graphically. Is it all going toether? Even if I
make it all bigger and bigger, Im sstill going against that.

What, if any, additional features do you think would improve the

experience of using this site? Why?
o R: When its not in draft form, will it move more smoothly? I
would make it as fluid as possible.

Is there anything we have not covered that youd like to comment on

before we conclude the session?
o R: I think it really covers a lot. It covers everything for Dr that is
not making it look like an ad. I would trust this information. Im
going at it from Medicare this is a dot gov I dont want to feel
promoted when I go to a government thing that I am paying for.
This is clinical. That is the difference. This is my medicare site
compared to if I go to go xyz hospital, they are pumping up their
deal.

October 2016 Moderators Guide

Page 15

o R: Pictures of individuals. I think people have a comfort level with

pictures. Even neighborhood helps picture the area. Pictures
of the practitioners. A lot of people with gender and nationality
that is a big deal to people. Maybe b/c it is a governmental site, I
dont know if it gets into any problems. But people do like to see
pictures of who this person is going to be before they get there.

Closing
Thank you for your time and sharing your opinions about the Physician
Compare website.

October 2016 Moderators Guide

Page 16