Running head: USABILITY TESTING OF A WEB AUTHORING TOOL

Usability Testing of a Web Authoring Tool

James Call, Kevin Fowler, Duren Thompson, Sharon Trahan
University of Tennessee

IT 595

USABILITY TESTING OF A WEB AUTHORING TOOL

Usability Testing of a Web Authoring Tool

It is very important for members in the field of Instructional Technology to share
resources with each other. As careers change, best practices evolve and members must have a
way to instantly access and disseminate the most up to date information. One of the most
common ways to do this is through blogging. In class we experimented with several different
blog and website design sites, but our group chose to test the usability of Word Press because it
seemed to be the most robust and we all wanted to experiment with it further. In this paper we
will discuss the description of the tool we tested, the testing methods that we used, and the
results of the test.
Part I - Description of the Web Authoring Tool
Our group chose Word Press because it is a versatile online web-authoring tool that is a
free or low cost source available to the general public. It began as a blogging system but has
evolved into an economical system for managing content as well. Its numerous functions,
features, and plug-ins extend its benefits to users of varying abilities. It provides great flexibility,
making it useful to all computer users from novice to moderate to expert.
Word Press is intended for current computer users, both male and female, over the age of
eighteen years old. In order to benefit from the uses of Word Press, the user must have at least an
average reading level and the ability to understand moderate to difficult instructions and
concepts. Possessing basic computer skills is another prerequisite for using Word Press but
more extensive experience with technology is required to reap the full benefits of this webauthoring tool.
The range of functions available in Word Press is enormous. Aside from blogging, Word
Press can be utilized to create websites for limitless purposes. Examples are content
management, resource repository, educational, photo management, and business websites. Its
functionality is immeasurable because of the use of plug-ins which are installed easily and allow
the user to use Word Press for almost any intended purpose.
A novice user can take advantage of the numerous templates available to create a
standard blog or website, referring to the help option available at the click of a button. An
everyday user can quickly create and customize unique blogs or websites that meet all of their
needs. There are also thousands of plug-ins available in Word Press that extend its capabilities to
allow more advanced users to take advantage of the tools full potential.
Part II - Low Cost Usability Testing Methods
For our usability test, we decided on a technology that allowed us to watch our
participants in real time called Adobe Connect and an application from Surveymonkey.com that
allowed us to survey our users afterward. It was important for us, as we believe it must be for
many usability testers, to watch our participants reactions while using the product so this is why
we chose to use Adobe Connect.
Adobe Connect is an online web conferencing product that features screen-sharing
capability. Although Blackboard Collaborate was free for our group to use as IT 595 students, we

USABILITY TESTING OF A WEB AUTHORING TOOL

unanimously agreed after testing Adobe Connect that its flawless screen-sharing capability
greatly outperformed Blackboard Collaborates choppy, subpar performance.
We believe that Adobe Connect is the perfect solution for over-the-web usability testing,
although unfortunately one must pay for the service. Thankfully for those in the education world,
a year license for Adobe Connect only costs $150.00 (after discounts of up to 50% off the
corporate licenses) or you can pay per use at a rate of $0.32 per user, per minute.
We also desired for users to submit a survey of the product and process anonymously. For
this we used the free features of Surveymonkey.com to gather and tabulate our results. The
results will be discussed below.
Part III - Usability Testing Results
The usability testing was conducted on three members of our class with varying degrees
of technology competence ranging from novice to moderate with all being comfortable with
computer technology. Each individual test took approximately twenty minutes. The day before
the test we provided our classmates with procedures to log into Adobe Connect, instructions for
the usability testing, and a link to the usability test page.
The first task we asked our test subjects to do after logging in was to create a new page
within the site. Two out of the three subjects found this process to be easy while the third
found it to be moderate in difficulty. During the test our group noticed that one of the subjects
had trouble locating the New Page button on Word Press. Another subject had issues seeing
the New Page icon and other icons that Word Press uses. The lightness of the icons caused
her to click on Posts first before clicking on Pages.
The next task was to add text to the page that the subjects created. This seemed to be an
easy task for all subjects and when they were polled all three described the task as easy. We
noted that one member added her title of the page in a different format than the other two,
resulting in difficulty locating her page later.
The third task was to insert a picture onto the page that the subjects created. Each person
had a different opinion of the difficulty of the task with one easy, moderate, and difficult
response each in the posttest survey. While observing each during the testing, we saw that this
task showed signs of difficulty for two of those tested. One subject had trouble locating the
insert picture icon and after publishing her page was not able to see the picture she inserted
(which we were not able to determine why because all steps were completed to insert an image).
Another almost chose the wrong icon to insert an image, but this was due to hesitation.
Finally we asked the people we tested to publish their pages and view them. We noted
difficulty during this portion of the testing, however two-thirds rated the task as easy and the
other chose moderate. The difficulty we observed was in viewing the pages that they created.
As stated earlier, one subject chose a strange title format which made finding the page more
difficult. There was also hesitation for all subjects while completing these tasks.
After the test we had the subjects complete a posttest survey and we compiled the results
for analysis. Word Press is useful a tool for blogging based on the results. Usefulness concerns
the degree to which the product enables a user to achieve his or her goals, and is an assessment

USABILITY TESTING OF A WEB AUTHORING TOOL

of the users willingness to use the product at all. (Rubin & Chisnell, 2008, p. 4-5) All three
subjects were able to complete the tasks fully and each either said they would use the product
again or suggested changes to the product, implying that they would use it again.
Word Press is also an efficient tool based on the time it took for the goals of each subject
to be accomplished accurately and completely. Although we noticed a large amount of hesitation
throughout the test, all three subjects completed the tasks in roughly the same amount of time
around twenty minutes.
Based on our results, Word Press is not very effective for new or novice users.
Effectiveness refers to the extent that a product behaves in the way that the user expects it to.
(Rubin & Chisnell, 2008, p. 4-5) The only task that 100% of users found to be easy was
adding text to a page. The other tasks ranged from easy to difficult with the most difficult
being inserting a picture or image and 66% of users were able to create the page, publish it, and
view it easily. The learnability of the tasks seemed to be equal for all subjects and one stated
that she would be comfortable with Word Press after a few more uses.
The satisfaction of the tool was mixed for our subjects. Aside from finding some tasks to
be difficult, each person had a suggestion to improve Word Press. Two had a suggestion to make
publishing and viewing a post easier and the other had a suggestion to modify the icons used on
Word Press which also has to do with the accessibility of the tool.
Accessibility involves making products usable by people who have disabilities. If the
icons used on Word Press are difficult to see then this is an issue for those with sight disabilities.
A suggestion to address this problem would be to enlarge the icons used and change the colors so
that they stand out more from their surroundings (Rubin & Chisnell, 2008, p. 5).
We learned a great deal from the results of the testing but we also learned from our own
procedures and decided there are many ways to improve them to receive better results from
future testing. First, we realized that if our test subjects did not enter the test with the
instructions that we provided, it made testing much more difficult. We would have to delay the
test for the subject to retrieve them or read them aloud during the test.
Second, we discovered that our instructions could have been more clear. For example,
we did not say what to call the title page our subjects created so one chose an odd title and it
caused confusion in viewing the page. In another example, we did not make it clear where to
click to participate in the posttest survey and one subject clicked in the wrong spot and this took
her to an advertisement page and not the survey.
Third, we understood after the testing was complete that many of the hesitations
discussed earlier might have been based on the subjects anxiety because they were being
watched by three extra testers. Hesitations were also probably due to their fear of failure during
testing. We could choose to only have one tester with one subject for future usability tests and
the other members of the testing team could watch the video later. We reassured each subject
before the test that there was no way to do poorly during the testing, but the fear of failure was
probably increased due to multiple people watching them.
Last, we discussed how being organized before a usability test makes the test run very
efficiently for the testers and the test subjects. The instructions that Duren and Kevin created

USABILITY TESTING OF A WEB AUTHORING TOOL

were very extensive and detailed. Our ability to test Adobe Connect before the test and assess its
usability for our testing was also integral to our process running smoothly. The availability of
the survey that Sharon created during the testing also made it very easy to take notes as each
subject completed the tasks assigned.
Finally, we feel that results we received prove Word Press has difficulty passing the
usability test. When a product is truly usable, the user can do what he or she wants to do the
way he or she expects to be able to do it, without hindrance, hesitation, or questions. (Rubin &
Chisnell, 2008, p. 4) We found the users could accomplish all tasks using Word Press, but there
were hindrances (not being able to see an uploaded image), hesitation due to various factors and
unfamiliarity, and questions when searching for individual icons. With small adjustments to our
testing and with updates to Word Press, it would not be difficult to define Word Press as a highly
usable web authoring tool.

USABILITY TESTING OF A WEB AUTHORING TOOL

References
Rubin, J., & Chisnell, D. (2008). Handbook of Usability Testing: How to Plan, Design, and
Conduct Effective Tests (2nd ed.). [rubin_chisnell_2008_chp1-2.pdf].

Appendix
Personal Interview Results

USABILITY TESTING OF A WEB AUTHORING TOOL

What do you consider your technology comfort level? (very comfortable, somewhat
comfortable, not comfortable)
#1 somewhat comfortable
#2 - somewhat comfortable
#3 very comfortable
What do you consider you technology experience level? (Expert, moderate, novice)
#1 novice to moderate
#2 moderate
#3 no answer (verbal was moderate to expert)
Survey Question Results
1. Please rate your experience with completing the following tasks in Word Press.
Easy

Moderat
e

Difficul
t

Creating a new page

66.7% (2)

33.3%
(1)

Adding text to a page

100.0%
(3)

Inserting an image or
picture
Publishing and viewing
your post

Rating
Averag
e

Respons
e
Count

0.0%
(0)

1.33

0.0% (0)

0.0%
(0)

1.00

33.3% (1)

33.3%
(1)

33.3%
(1)

2.00

66.7% (2)

33.3%
(1)

0.0%
(0)

1.33

Creating a new page responses:

I was able to locate the create new page option quickly. #1
The instructions really helped make this easier. #2
I find the menu options on word press to be confusing and the typeface is so light, it's not
noticeable enough. #3
Adding text to a page responses:
I was able to type in my text and it came up when I clicked view page. #1
Inserting an image or picture responses:
I had difficulty, because I thought the image had downloaded, but when I clicked on view
page the image was not there. #1
Again, the icons and labels are not readily viewable. #3

USABILITY TESTING OF A WEB AUTHORING TOOL

Publishing and viewing your post:

The publish button was easy to find, but I had trouble finding the view post button again
after I had left the page after viewing my post the first time. I sadly had to ask for
assistance. #1
2. What recommendations, if any, do you have that would improve your experience with
Word Press?
I need experience with it. The directions I followed were all something I had done before.
It took me a moment to remember where everything was. Your group described
everything perfectly. It was a painless process. #1
Make it clear when I have published something. #2
Better labeling for menu options, maybe dynamic buttons so when you scroll over one,
you can see your options for each heading. #3