Memo

To: Dorothy Cavanaugh, NAVAIR Systems

From: MAPC Graduate Students Keith Sechrist, Xue Ding, and

Ciara Marshall

Date: February 7th, 2017

Re: Heuristic Evaluation of NAVAIR Career Guide Book Mobile Application

Cc: Dr. Tharon Howard

Introduction
The following memo contains a heuristic evaluation of the Dept. of NAVAIR Career Guide Book
mobile application, more specifically the version of the application available to iPhone users.
The goal of this evaluation is to draw attention to sources of confusion users may encounter
when using the application so that they can be remedied.

For our analysis, we will use the task list provided to help us evaluate the Career Guide Book
application. To guide you through our navigation, we will use the numerical hierarchy that you
already have established. If we are on the main page found in the navigation bar, we will refer to
it by their name: “Home”, “Competencies”, “My Bookmarks”, “My Notes”. Past this, the
navigation through the Competencies will use the numbers assigned to it. For example, in your
first task, you used the numbers “4.6.3” to tell us how to navigate. So starting with 4, we click on
“Research and Engineering,” then the 6 (middle number) refers to the second level of
navigation, so we will click on “Human Systems Dept,” on the third level, we will click on the -3
“In Service Engineering,” and the fourth level is where we reach the page where we reach the
definition, competency behavior, and recommended training information. Therefore, if we are
only referring to the second level of navigation, we will only give you the first two numbers. If we
are referring to the third level of navigation, we will use all three numbers. If we are referring to
general level of information, each level corresponds with the number: the first number is the first
level, the second number is the second level, the third number (dashed number) is the third
level and the fourth level is the actual information page.

To conduct this analysis, we used Nielson’s 10 Usability Heuristics for User Interface Design
(https://www.nngroup.com/articles/ten-usability-heuristics/) as the primary criteria. Below you
will find definitions for each heuristic category, and a score stating the severity of user
experience problems encountered for that category, and an explanation of each score. The
user-experience problem severity scale is as follows:
1: Cosmetic Problem - the user is able to continue, as the problem is only minimal.

2: Minor Usability Problem - the user is able to find alternative solution to the problem without
assistance.

3: Major Usability Problem - the user will most likely reach out for support or consult available
documentation to continue

4: Usability Catastrophe - the user will be unable to continue due to lost functionality.

Visibility of system status: 2 - Minor Usability Problem

The application should always keep users informed about what is going on, through appropriate
feedback within reasonable time.

When initially opening the app, the load screen has no indication of the how much longer the user
will have to wait for the app to finish loading and/ or preparing to launch. Also, when a competency is
selected, it sometimes takes 10-15 seconds to load. I would suggest incorporating a percentage bar
rather than a buffering wheel to let users know how much longer they can expect to wait, as this will
provide them with feedback to let them know the application is working and has not crashed.

The chronological numbering system of the competencies does help users keep track of which
section they are visiting, though it is unclear why the competencies homepage does not use perfect
chronological numbers. For example, the level 1 competency options read “1.0, 2.0, 4.0, 5.0, 7.0,
7.2, 10.0, 11.0”. Users may get the impression they are being supplied with incomplete information
because of skipped numbers in this sequence.

Match between System and Reality: 1 - Cosmetic Problem

The application should speak the users' language, with words, phrases and concepts familiar to the
user, rather than system-oriented terms. Follow real-world conventions, making information appear
in a natural and logical order.

Other than the occasional unexplained acronym, the language used in the application was jargon-
free and easily understandable. Content seemed to be well-organized by subject area, which will
help user navigate the application more easily. I would suggest adding a glossary to the competency
page so that users can find the the meaning of acronyms easily; if possible, it would be ideal for
acronyms and specific terms to be clickable links that would take the user directly to a glossary
definition.

User Control and Freedom: 3 - Major Usability Problem

Users often choose application functions by mistake and will need a clearly marked "emergency exit"
to leave the unwanted state without having to go through an extended dialogue. Support undo and
redo.

Every area of the competencies has a readily available “Back” button to return users to the previous
screen, however it may also be helpful to keep the navigation bar across the bottom as well, even
when users are within the competencies so that they can navigate more quickly and freely.

When trying to create an entry directly in the “My Notes” screen, it may be helpful for users to have a
dropdown menu of competency and sub-competency items to choose from. Otherwise, they have to
go back to the competency page they have in mind to confirm they are identifying it correctly, which
may take time and be an inconvenience.

In regards to this heuristic, our most pressing concern was that bookmarks or notes cannot be
accessed past the first level of competency options. Bookmarks and notes can be added once a
page with definitions has been reached, but this information cannot be accessed at the second or
third levels of navigation.

Consistency and standards: 2 - Minor Usability Problem

Users should not have to wonder whether different words, situations, or actions mean the same
thing. Follow platform conventions.

Level 1 Level 2-1 Level 2-2 Level 2-1-1

The first consistency issue occurs in the “Competencies” (Level 1) page. The table of content does
not go strictly with ordinal numbers. That is, after “7.0” is “7.2” and then “10.0.” In this case, the user
will be confused as to “Why does 7.2 not belong to 7.0?” and “Where are 8.0 and 9.0?”
The second consistency/standards problem can be observed by a comparison of the subpages
under “4.0” (Level 2-1) and “5.0” (Level 2-2). For one thing, the title areas (the blue part) look
completely different, as one has a “Back” button and the other has only the title in capital letters. For
another, while the Level 2-1 indicates that there are also subpages by clicking on each item (e.g. by
clicking on ‘4.0’ there comes Level 2-1-1, the Level 2-2 page just lists all the sub-items under “5.0.”
Error prevention: 3 - Major Usability Problem
Even better than good error messages is a careful design which prevents a problem from occurring
in the first place. Either eliminate error-prone conditions or check for them and present users with a
confirmation option before they commit to the action.This means that application should prevent should
prevent errors rather than explaining what the error is for its users.

Having the contact information available on the Homepage allows users to easily contact the owners
of the application for feedback and communication.

When you are on the Homepage and you travel to the “Contact Us” link, the first thing you see on
the next page is “Congratulations on taking advantage of the NAVAIR Mobile Career Guidebook.”
This can be very misleading to users because when on mobile websites or applications and you
follow a link and see “Congratulations” you immediately think you ran into an advertisement. This
could be seen as a deterrent from using the contact page to receive feedback from users. To
prevent this error, just change “Congratulations” to
“Thanks.”

Under the create a note icon, it is really great that the

notes tell you the location of where the note was taken.
When creating a note, there is a two-step process to
save the note and no warning for users if the note was
properly saved before they leave the page. The first
step is to click the “Done” button in the bottom right-
hand corner, and the next step to is to click the “Save,”
which is located right beside the “Cancel” button. To
prevent users from accidentally hitting “Cancel” rather
than save, it is important that these two buttons not be
so close together, especially on a phone application.
Lastly, there should be a warning that asks if they wish
to leave the page without saving if they have failed to
save or hit the “Cancel Button.” Currently, it just leaves
the page, so all of the information someone writes
could be lost without them knowing why.

Recognition rather than recall: 2 Minor Usability Problem

Minimize the user's memory load by making objects, actions, and options visible. The user should
not have to remember information from one part of the dialogue to another. Instructions for use of
the system should be visible or easily retrievable whenever appropriate.
While under notes, it is nice to see that they are organized by their “Competency Area” and their
“Focus Area”.

There are not specific instructions or detailed information that let the user know what each
Competency is about, nor is the detailed information of what each page is about at the second-level
or third-level. Without specific knowledge. instructions of locations, or terminology, it will be difficult
for users to navigate to specific page they desire or to find out if they are even in the right
competency. This can lead to users being frustrated with the application.

Once you are past the first-level, the Competencies, you cannot access your bookmarks or notes.
You can add bookmarks and notes once you reach the page that has the definitions, etc., but you
cannot access it at the second or third levels of navigation. Moreover, once you reach the level that
allows for you to write notes and bookmark, you cannot review or compare your information from
other pages without returning to the homepage.

When trying to complete the third task, it was confusing to navigate because it skips the second-
level and immediately goes to the third. The user will go under “5.Test and Evaluation” and they are
immediately met with numbers 5.0-1-25). Skipping this second level can be confusing because the
hierarchy of information does more than rank information by importance. The hierarchy allows you to
know what information you are looking at when you go to each level. My suggestion is under each
level, have a header that defines what information is at that level specifically. That way, users will not
become confused. The second level is also missing on 2.0
Contracts, 7.2 Information Technology, and 10.0 Comptroller.

Lastly, the subheadings of “Competency Behavior ” and

“Recommended Training” are blue and may be misconstrued as
hyperlinks when compared with the subheadings that are the
same size but only black. We suggest changing the color to black and
changing the subheadings in size and boldness.
Flexibility and efficiency of use: 2 - Minor Usability Problem
Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user
such that the system can cater to both inexperienced and experienced users.
Allow users to tailor frequent actions.

Overall, the application benefits both novice and experienced users by offering
the options of “bookmarks” and “notes.” However, it should be noticed that in
“My Bookmarks,” if the user has marked a number of bookmarks, they didn’t
show in an ordinal order, nor is there a “search” option or navigation. In this
regard, the user may find it inconvenient when they have a large number of
bookmarks and want to find an identical one quickly.

Aesthetic and minimalist design: 1 - Cosmetic Problem

Dialogues should not contain information which is irrelevant or rarely needed.
Every extra unit of information in a dialogue competes with the relevant units of
information and diminishes their relative visibility.

Overall, the information contained in this guidebook are all relevant and needed.
The only problem is that the some subpages such as the one shown here use all
capital letters which seem “unnecessary” and may hurt the ease of visibility.

Help users recognize, diagnose, and recover from errors: 3 - Major

Usability Problem
Error messages should be expressed in plain language (no codes), precisely indicate the problem,
and constructively suggest a solution.

Due to the simple design and practical functions of this application, the users may not run into too
many errors. However, a major usability problem here is that there is no error messages anywhere
in the application even when there is an error. During our evaluation of the application, there is a
time when we failed to get access to all the subpages under “Competencies” while no error
messages appeared.

Help and documentation: 2- Minor Usability Problem

Even though it is better if the system can be used without documentation, it may be necessary to
provide help and documentation. Any such information should be easy to search, focused on the
user's task, list concrete steps to be carried out, and not be too large.

Having the contact information on the front page is useful because it easily allows the users to
directly know where to find information about where to contact help.

The breakdown of “Recommended Training” into subcategories “Awareness,” “Basic,”

“Intermediate,” “ and “Advanced” allows for users to know exactly what kind of education they need
to meet the requirements of each level of “Competency
Behavior”

There is not any specific help documentation for this

application: the only contact point is as at the homepage,
which can only be accessed from the first level
(Competencies) page. There is not continuing navigation
bar that allows the user to reach the homepage at any
point in the document. There are only back buttons to
allow the user to return to the previous page until they
reach the first level in this application (see image). On the
homepage, it would be useful to have a How-To training
guide or a FAQ section that allowed the users know
some basic information about the purpose of the app,
how to navigate it, and where to find help.

There is not a clear definition between the “Competency Behavior” and “Recommended Training.”
Because both deal with knowledge. there needs to be a definition of the two headings that defines
the difference, or different names for the headings that let the user explicitly know the difference.

Summary:
While not comprehensive, the above heuristic-based analysis should provide insight as to how
NAVAIR Systems can improve the usability of its iPhone mobile application. Investing resources
to solve the described issues will improve the user experience of employees, which will likely
result in a better understanding of policies and procedures within the organization. We thank
you for the opportunity to become involved in this project.