Vous êtes sur la page 1sur 16

Usability Test Report for UC Berkeley Library Website

Date of Report: Location of Test: Prepared for: Prepared by: February 2012 UC Berkeley Library Web Advisory Group Julie Lefevre, Lisa Ngo, Lisa Weber, Matthew Prutsman, & Tim Dennis

Executive summary
As part of the Library website redesign process, a Web Advisory Group design subcommittee was formed in 2011 to conduct interviews with students and faculty about their search behaviors and library website usage. The goals of the interviews included determining the perceived content, organizational and navigation deficiencies of the library homepage, identifying what the homepage currently does well, gaining a better understanding of how students and faculty/GSIs actually use the website as well as how they would like to use the library website, and gaining a better understanding of our users research goals and how they achieve these goals. Portions of the interview were also designed to help develop user personas for the redesign and testing process, though these findings are not included in this Usability Test Report. A total of ten participants (users) were interviewed; however, due to a technical problem, only nine interviews were properly recorded and counted. Users were interviewed individually by two members of the design subcommittee. The user group consisted of one lecturer, two graduate students, and seven undergraduates varying from freshmen to seniors. Interviews consisted of an interview portion (~30 min., video recorded), a card sort portion (~15-20 min., video recorded, results analyzed separately outside of this Usability Test Report), and a task-based usability test (~15 min., screen and audio recorded). They were rewarded for their time with a $10 Free Speech Movement Caf gift card. Interview Website Usage We found that the library website is perceived as an important resource to students and faculty, with graduate students and faculty reporting more frequent usage (everyday or several times a week) than undergraduates (2-3 times a semester or twice a month). Users report that they most often use the site to access electronic resources, use OskiCat, find library hours, and to request/renew books. Novice users, especially the lower division undergraduates, frequently had difficulty with website navigation and report being overwhelmed by the number of options and links on the site. Website Usability Testing Users excelled at finding information that had direct links on the librarys home page. They had no trouble finding information on connecting from off-campus (100% success, 9 completed), copying/printing information (100% success, 9 completed), using Melvyl to find books not held by UC Berkeley (100% success, 5 completed task was not given to every user), find a known e-book (89% success, 8 completed), and library hours (89% success, 8 completed). For these, we have few recommendations to make for the redesign except to give the multi-search box a more prominent location on the homepage, and are suggesting some improvements to the e-books landing page based on user comments.

WAG Interview Sub-group -

Once users navigated away from the homepage, they had more difficulty locating the information they needed. Though the horizontal navigation bar at the top of secondary pages was designed to help users, it appeared that it was rarely utilized for navigation. When users were asked to go to the Interlibrary Loan page from the copying/printing page, only 5 participants (56% success) used the horizontal navigation bar successfully. As a result, we recommend that the redesign include creating & testing different treatments of the local or sectional navigation. Some tasks had fairly high rates of success but also a high mean time for completion, indicating that there was some trouble during task execution. For example, locating a known item in the Electronic Resources Finder (ERF) had an 89% success rate (8 completed) but the average user took 29 seconds to complete the task (compared to an average of 16 seconds or less for easier tasks). Users exhibited some confusion when landing on the ERF page and not understanding the different categories. As a result, we recommend possible solutions including putting the ERF search box on the ERF page, rethinking display of subject A-Z, type A-Z and General links, and further user testing (see Scenario 5 for more specific recommendations). Locating subject specialty library websites also took users longer than expected (67% success, 6 completed, 37 seconds average). Due to some ambiguity with the phrasing of this particular task, we recommend that this task be retested with a clearer scenario. The sites problems with navigation and site architecture become apparent when users had to go deeper than one or two links into the website. Finding subject librarian contact information was a failure (44% success, 4 completed) as was finding information on citation styles (44% success, 4 completed). When asked to find subject librarian contact information, the majority of users (7 out of 9) selected the Contact Us page first, suggesting that this problem can be easily solved by adding subject specialist information to the staff directory. Helping users find citation styles, however, speaks to a greater need for a closer examination of our site architecture.

Who we tested
Nine participants, having the following characteristics, evaluated the UC Berkeley Library Web homepage.

Audience Type Undergraduate Graduate Instructor TOTAL (participants) 6 2 1 9

Gender Women Men TOTAL (participants) 7 2 9

WAG Interview Sub-group -

What participants did

Participants met with two members of the WAG sub-group for a one-on-one interview, card sort exercise, and website testing. One member of the WAG sub-group acted as both the lead interviewer and study facilitator. The other member took notes and handled the recording devices. The whole session typically took between 1 and 1.5 hours to complete. The website usability testing component of the session took 15-20 minutes. Each participant was asked to complete 10 scenarios (see below) using a laptop provided by the Library Data Lab. We used a video camcorder to record both the interviews and card sorts. We also used a screen capture tool to capture both audio and screen activity of the website usability testing.

What data we collected

We video recorded the interview and took notes. We recorded the participants card sort terminology creation and groupings. We recorded both what the participants said and their screen activities during the website usability testing.

Major findings and recommendations

Issue: Users often could not find the multi-search box for OskiCat, Melvyl, and the Library Web. This is partly due to the placement of the multi-search box on the bottom of the homepage, below the fold for those with smaller screens. Interview subjects did not scroll down the homepage, so did not see the search box. Solution: Move the multi-search box higher on the Library Web homepage. Issue: The majority of respondents could not find subject librarian contact information via the homepage. Solution: Add subject responsibilities to librarian Staff Directory listings and simplify the Contact Us page (see scenario 8 for more detail). Issue: When respondents landed on the Electronic Resource Finder (ERF) entry page, they were often overwhelmed by the amount of information on the page. They were unable to quickly scan the page for the required information. Solution: Include the ERF search box on the ERF entry page. Issue: Respondents often had difficulty differentiating between what would be found on the Guides page and what would be found on the Tutorials page. Respondents did not seem to differentiate between Guides and Tutorials. Solution: Combine Guides and Tutorials into one page. Issue: Respondents rarely utilized the tabbed horizontal navigation bar at the top of secondary pages to navigate the site.

WAG Interview Sub-group -

Solution: Create and test different treatments of the local or sectional navigation (vertical along sidebar, highlighting where user is in the sectional navigation, etc.).

Detailed findings and recommendations Interview Summary - Website Usage

Summary of questions about website usage
1. How often do you use the Library website?
All respondents
# of Responses 3 2 2 2 Frequency Twice a month Everyday Several times a week 2-3 times a semester

By status
Status Freshman Sophomore Junior Senior Grad student Grad student Instructor Response Twice a month Twice a month 2-3 times a semester Everyday Everyday Several times a week Several times a week # of Responses 2 1 2 1 1 1 1

2. What do you typically do when you use the website? 4

WAG Interview Sub-group -

Task Electronic resources OskiCat Hours Renew books Melvyl ILL Subject guides Student jobs Course reserves Check loaned book status Staff contact News/events

# of Responses 7 6 6 4 3 2 1 1 1 1 1 1

3. If you have problems finding what you need on the Library website, what have you found was most difficult, confusing, or frustrating?
Problem OskiCat/catalogs confusing Too many options/buttons on website Melvyl navigation difficulty Confusing links for databases No list of available journals Website navigation difficulty Site search malfunctions Visitor information page confusing Expert searching functions difficult in OskiCat # of Responses 2 2 1 1 1 1 1 1 1

WAG Interview Sub-group -

Website terms confusing ERF UC-eLinks confusing Need more printing information Hard to find research articles

1 1 1 1 1

4. Do you have any specific suggestions for features or content that would improve your experience of the website?


make more sense if everything was in one website or there were more direct links to other libraries on the homepage more uniformity would be more convenient, user would be more likely to look at other library websites wants an iPad specific version of the website - the current mobile version made when viewed in an iPad is too simple. for the ERF: use hover-overs or informational pop-outs to provided help for new users for OskiCat: have availability status on item page


make clearer subject guides for students make students aware of resources beyond Google and the value of scholarly publications provide basic help on researching articles search have online tutorial about how to use the library available for freshmen provide subject breakdowns, like for humanities majors, sciences majors, etc., makes site more welcome for everyone

WAG Interview Sub-group -

Interview Summary - Website Usability Testing

NOTE: Tasks were designed based on perceived shortcomings of the library website as identified by library staff on staff surveys, and on common user tasks identified by library staff on staff surveys.

Scenario 1 - Copying information

You would like to make some photocopies of a book when you next visit the library, but you are unfamiliar with how copying works in the Library. Starting from the homepage, find out photocopying information. NOTE: This task was intended to be simple and get the user onto the Photocopying page for the second scenario. Success: The participant navigates from the homepage to the services/print.html page. Number of participants Percent successful Mean time Median time 9 100% 9 seconds 7 seconds

All participants completed the task with ease by finding the services/print.html page. Zero participants needed prompting or had significant difficult completing the task. 10 participants found the services/print.html via index.html services/print.html. One interesting finding from this test is that the terminology changed from photocopying to scanning about half way through the testing. However, we didnt change the language in our test instructions (from photocopying to scanning) and students still found the page with ease.

No changes needed.

WAG Interview Sub-group -

Scenario 2 Testing local horizontal navigation bar

Now, once youve located the photocopying in the Library information, you remember you need to find out how to submit a request for a book in a non-UCB library. Find the page with this information. You need a book that is not included in OskiCat. Success: The participant using the link to the ILL page (services/interlibrary_loan.html) in the local horizontal navigational bar.

Number of participants Percent successful Mean time Median time

9 56% 16 seconds 11 seconds

5 participants completed the task with ease by finding the ILL webpage via services/print.html navigational link. 2 participants didnt complete the task using the navigational link to ILL, but found the ILL link by going back to the homepage 1 participant didnt complete the task, but found ILL but through a longer path (4 clicks) 1 participant did not complete the task or ultimately find ILL

In the design phase, we recommend creating & testing different treatments of the local or sectional navigation (vertical along sidebar, highlighting where user is in the sectional navigation, etc.).

WAG Interview Sub-group -

Scenario 3 - Finding a book not held by the Library

You need a book that is not included in OskiCat. How would you use the homepage to help you get the book? Note: This question was only asked of 5 participants. Success: The participant navigates to Melvyl via any route or searches Melyvl directly from the homepage. Number of participants Percent successful Mean time Median time 5 100% 16 seconds 12 seconds

1 participant completed the task by using the Catalog search box on the homepage. 3 participants completed the task with ease by finding the Melvyl link in the Library Catalogs bucket on the homepage. 1 participant completed the task by navigating to Find Books & E-Books page (find/types/books) then Melvyl.

Provide easy access to Melvyl via the Library Web homepage. The search box on the homepage provides the most direct access to Melvyl and should be made more prominent (brought up above the fold) on the homepage.

WAG Interview Sub-group -

Scenario 4 Finding citation styles help (/Help/guides.html)

Youre not sure how to cite a website for a term paper. What links would you use on the homepage to help you answer this? Success: Participant navigates to the General Guides page (Help/guides.html) using the Guides link on the homepage. Number of participants Percent successful Mean time Median time 9 44% 53 seconds 44 seconds

1 participant used the most direct route to the target (homepage -> Help/guides.html). 2 participants found the Guides page via a link off the Research Help page (Help/research_help.html). 1 participant found the Guides page via Information for Undergraduate Students (services/for_users/ undergrad_students.html) -> Research Help (Help/research_help.html). 3 participants started their unsuccessful navigation for the Guides page from within the Find section of our site (Find Information, Electronic Resources, or Find Websites). 1 participant didnt find the Guides page trying to use the Information for Undergraduate Students page (services/for_users/undergrad_students.html). Participants used three different paths: 1. Find section, 2. Research help & 3. Information for Undergraduates.

1. Combine the Tutorials and Guides pages. 2. Put the Citation Management Tools link at the top of the page. 3. Links to tutorial videos and specific resource guides should be linked to at the point of need in the ERF, and on library and subject websites where the resources are linked to. 4. Since Guides include tutorials, A la Carte & other library resources, consider putting a link to the Guides page on the Find page. 5. Since Guides includes the entry point to A la Carte guides, we suggest WAG explore a better way to provide access to the subject & course guides (browse by subject, course, etc. linked off the homepage). Ideally, this listing should be built in A la Carte itself and not a static listing on the homepage.

WAG Interview Sub-group -


Scenario 5 Locating known item in Electronic Resources Finder

Your professor recommends that you start with the articles in Academic Search Complete. How would you find this database? Note: Academic Search Complete was not the best choice of database for this test. It is an example on the ERF homepage, making it easier for participants to find. Success: Participants finds the resource by navigating or searching the ERF. Number of participants Percent successful Mean time Median Time 9 89% 29 sec 1 Sec

4 participants completed the task with relative ease using the Academic Search Complete link in the Examples text on the ERF entry page (find/types/electronic_resources.html). 2 participants found the title via ERF General Article databases listing (one in 3 clicks & the other in 10). 2 participants started their search for the resource via the ERF Subjects A-Z listing. 1 participant didnt complete the task, but found Academic Search Complete in Melvyl. 1 participant used the ERF search to find the resource. Some participants found the links on the ERF homepage redundant (e.g., Electronic resources: subjects AZ, Electronic resources: types A-Z, etc.). Some participants found the link labels confusing e.g., subjects A-Z, types A-Z and General. Some participants wondered where the search box was, they didnt notice the link to the ERF locate page. Some participants found the ERF homepage confusing and difficult to navigate.

1. Include the ERF search box on the ERF homepage. 2. Include the ERF search as part of the homepage Search catalogs or website search box, either as part of the website search or its own index. 3. Rethink displaying the subject A-Z, type A-Z and General links. 4. Consider including recommended resources for a particular subject in the subject website. 5. Highlight new resources or featured resources on the ERF homepage. 6. Do some user testing to determine which link label makes more sense to users: Electronic resources or Articles. Both links in the Find bucket may be redundant. 7. Consider moving the ERF homepage (find/types/electronic_resources.html) into the ERF application itself.

WAG Interview Sub-group -


Scenario 6 Finding a known item e-book

You want to find the e-book Sergeant Presley: Our Untold Story of Elvis' Missing Years. How would you go about doing this from the Library Web homepage? Success: Navigate to the Find Books and E-books page and search Ebrary. Also acceptable for the participant to search OskiCat or Melvyl. Number of participants Percent successful Mean time Median time 9 89% 16 seconds 15 seconds

5 participants navigated to Find Books and E-books page and searched Ebrary to find the title. 2 participants clicked on the OskiCat link on the homepage & searched for the title. 1 participant clicked on the Melvyl link and searched for the title. 1 participant used the Search catalogs or websites search box on the home page to search Melvyl for the item. Some participants were surprised to see redundant links to OskiCat and Melvyl on the Find Books & Ebooks page Some participants wondered why the e-Book links were so far down on the page.

1. 2. 3. 4. 5. 6. Change the homepage link label to read E-books. On the Books and e-books page make it clearer that e-books can be found via OskiCat. Include links to e-books that cant be found via OskiCat. Include information about reading e-books on different devices, e.g. iPad and Kindle. Remove ebrary search box. All ebrary titles are in OskiCat. Remove text that explains what an e-book is.

WAG Interview Sub-group -


Scenario 7 Locating branch location & hours

Its finals week. You need to find a quiet place at one of the libraries to study for an exam tomorrow, and youve heard that the Environmental Design library actually enforces quiet study areas. How would you find out where the library is located and how late theyre open tonight? Success: Participant navigates from the homepage to the hours page. The most direct path (1 click) was using the Hours and maps link on the homepage. Number of participants Percent successful Mean time Median time 9 90% 16 seconds 16 seconds

7 participants completed the task with ease by finding the hours page via the link off the homepage. 1 participant navigated to the hours page via the About the Library index.html page. 1 participant did not complete the task using the homepage, but found ENVIs hours using berkeley.edu site search.

No change is needed.

WAG Interview Sub-group -


Scenario 8 Finding contact information for a librarian (You need | Your professor has asked you)
to contact the Political Sciences librarian for help (with library resources for a class you are teaching | on a term paper), how would you find the librarian? Success: If participant locates a link to the contact information for the political sciences librarian via two places: 1. Contacts(liaisons), Academic Departments and Programs (Help/liaisons.html) or 2. Libraries and collections A-Z (AboutLibrary/libraries_collections.html#p) . The most direct path possible was 2 clicks (homepage -> Contact us > Contacts(liaisons) or Libraries and collections A-Z). Number of participants Percent successful Mean time Median time 9 44% 79 seconds 79 seconds

1 participant completed the task using the most direct path to the target, but spent 30 seconds doing so. 3 participants completed the task, but averaged 5 clicks. 5 participants did not complete the task. 6 participants tried to use the Staff Directory, but couldnt find mention of Political Science via Search, Browse by Unit or Browse by Name. 4 participants tried to search the staff directory by some variation of political science librarian. No one used the Contacts (liaisons), Academic Departments and Programs link. The density of links on the Contact Us page made it overwhelming and difficult to read or scan.

1. In the Staff Directory add subject responsibilities to librarian/staff titles or entries. In effect merge subject information currently on the Liaison page into the Staff Directory. 2. Delete the Liaison page. 3. Either make the Contact Us link go directly to the augmented Staff Directory, or drastically reduce the number of links on the Contact Us page. 4. For consistency, make the contact information in A la Carte be sourced by the staff directory. 5. Redesign and test the Contact Us page altogether. Try combining like resources (all forms together -ideally into the same form or at least grouped together) and putting a search on the staff directory on the Contact Us page.

WAG Interview Sub-group -


Scenario 9 Finding a subject speciality library website

Youve heard about the Bioscience and Natural Resources Library having a replica of a T-Rex on display. Starting on the library homepage, show us how you would locate more information on the T-Rex. Success: The participant navigates to Libraries and Collections from the homepage and then to the Bioscience & Natural Sciences Library. Number of participants Percent successful Mean time Median time 9 67% 37 seconds 13 seconds

This test was problematic because we used finding information about an exhibit (T-Rex) as the motivation for finding the BIOS website. 4 participants completed the task with ease finding BIOS via the Libraries and Collections page. 1 participant got to BIOS after first trying the Visitor Information page and then clicking through to Libraries and Collections. 2 participants did not complete the task. 2 participants tried to complete the task through the events.berkeley.edu (this was because we asked to locate T-Rex exhibit). 1 participant said she would use the berkeley.edu site search

We recommend finding subject specialty libraries be retested with a clearer scenario.

WAG Interview Sub-group -


Scenario 10 Finding the connecting from off campus page

You need to connect to library e-resources from your apartment. Locate the instructions on configuring your laptop to enable this. Success: Optimally, the participant would use the link to the Connecting from off campus directly from the homepage, but we accepted getting there from Information for Undergraduates.

Number of participants Percent successful

9 100%

6 participants completed the task using the link on the homepage to get to the target. 2 participants found the Connecting from off campus link via Services for Students page (undergrad_students.html) 1 participant found the Connecting from off campus page via the FAQ.

No changes are needed.

WAG Interview Sub-group -