961

16.3 Bringing Meaning to Tourism: The Use of

Augmented Reality to Navigate Foreign Language
Environments
Thomas Richard Schalow
University of Marketing and Distribution Sciences, Kobe
Abstract
Augmented reality is perhaps less well known than virtual reality, but in the near-
term it promises to transform our world, and the tourist industry, in more and more useful
ways than the artifcial digital environments that have become known to people through
movies such as Avatar. In the context of tourism, augmented reality technologies allow
real-time translation of signs and other printed, text-based information from the language
of the foreign country to the language of the traveler, without the need to actually change
the environment, or enhance the knowledge base of the traveler. Merely viewing the world
through something known as an augmented reality browser, most likely through a device
such as a smartphone, will magically transform the sign and its text, the world, and theen-
tire tourist experience.
Keywords
Augmented reality; foreign languages; translation; smartphones
1. Introduction: the problem of foreign languages for the tourist industry
The tremendous expansion in tourism in the past two decades, in line with
declining costs for travel, has allowed more people to travel who lack the advanced
linguistic skills necessary to optimally navigate foreign environments. This is par-
ticularly true when travelers have only basic levels of education, and especially so
when Asians travel to Europe and Europeans travel to Asia. Travelers from China,
for example, though capable of reading signs using what many in the West perceive
to be diffcult Chinese ideograms, could be entirely baffed by signs using common
Western languages such as French or English.
In the same manner, travelers from places such as Europe and America are now
commonly visiting tourist destinations in Asia and elsewhere, where their linguistic
skills are severely challenged by written languages that are quite different in form
from the languages they are familiar with. Although English has become a lingua
franca for much of the tourist industry, English and other Western European languag-
es are still not yet a part of the everyday environment for more than perhaps a quarter
of the worlds population. Consequently, attempts by travelers to achieve an authen-
tic experience in a radically foreign environment will inevitably be constrained by
the limited linguistic access afforded to non-natives. This diffculty is especially felt
when alphabets are non-Latin in their form, or in environments where pictographs
962
are used in place of alphabets. The linguistic cues found on signs and other textual
information sources in these foreign environments are hidden from the traveler as a
result of language differences, and threaten to make the tourism experience diffcult
or even potentially dangerous.
Technology has stepped in to help with these problems, offering translation
devices, electronic dictionaries, and the like, but problems still persist. Electronic
dictionaries, for example, are of little use when there is no alphabet to facilitate
navigation, as is the case for logographic/pictographic languages such as Chinese or
Japanese, or in the case where alphabets are non-Latin in form, such as for Thai or
Korean. What is needed is a tool to magically transform the environment into one
that is linguistically familiar to the traveler, without the need to actually alter the
environment from the perspective of the native. In this paper we will look at how
augmented reality (AR) is making the whole world accessible to any traveler, regard-
less of native language or foreign language experience.
For practical reasons, this paper will focus only on the problems faced by travel-
ers dealing with text or pictographic-based information sources. It is recognized that
a great deal of information is available only through verbal communication, and
this information will remain unavailable to the tourist as long as we lack real-time
universal language translation software. Nonetheless, AR applications focusing only
on text or pictographs will be of great utility to travelers. Since these AR applica-
tions are either available today or will be available in the near future, they will have
a more immediate impact on the travel experience than real-time verbal language
translation, which yet needs to overcome formidable technological challenges before
it is achieved.
2. An introduction to augmented reality
Augmented reality is perhaps less well known than virtual reality (VR), but in
the near term it promises to transform our world in more and more useful ways than
the artifcial digital and virtual reality environments that have become known to peo-
ple through movies such as Avatar. Augmented reality, a term frst used in the early
1990s by two engineers at Boeing named Caudell and Mizell, does not create a to-
tally new and artifcial reality. Instead, it merely enhances the reality we experience
in our everyday lives, overlaying descriptions, instructions, or explanations
on top of the environment we perceive through our normal senses, such as sight. This
is most often achieved by viewing the augmented world through the video display of
a smartphone. In the most technologically advanced forms of AR, the actual environ-
ment itself is not changed. Only our perception of that environment is altered, in a
way that makes it more understandable or richer in context.
In order to better understand AR technology, I would ask you to consider a bot-
tle of medicine sitting on a shelf in a Japanese pharmacy. A traveler visiting Japan
on a ten-day vacation has had some problems with the food and developed an upset
Thomas Richard
Schalow
Bringing
Meaning to
Tourism: The Use
of Augmented
Reality to
Navigate Foreign
Language
Environments
963
stomach in a remote area of the country. While not life threatening or requiring a visit
to a doctor, the visitor would appreciate some immediate relief and is now looking
at medicines in the small towns only pharmacy. The traveler speaks no Japanese
and the only available sales clerk speaks no English. The traveler has noticed some
packages in the pharmacy with pictures of what appear to be a human stomach, but
there are no products with familiar names to choose from, and the available Japa-
nese products have only Japanese characters printed on the packages. Fortunately,
the traveler is carrying a smartphone and has an Internet connection through the
telephone network.
Illustration 1: This is a Japanese medicine for an upset stomach
The traveler points a smartphone camera at the products on the pharmacy
shelves and AR technology begins to work. The AR technology frst establishes that
there is a barcode on the packages. It has found a marker to work with. The AR tech-
nology then also establishes the location of the pharmacy, as provided by a gravi-
metric device or GPS signal and the smartphones ability to make use of this signal.
Finally, the smartphone camera notices there are objects, the Japanese text, on the
package, and using OCR software is able to read the information. As a result, due
to the marker, the visitor knows the name of the product and any other informa-
tion encoded in the barcode. With smarter barcodes, even information such as the
intended use of the product could be provided. The visitor also learns, as a result of
information provided by querying a databases based on the location of the phar-
macy, that a trained pharmacist named Nakano Tomoko, who is able to speak some
English and thus might be able to help our traveler, works in the pharmacy from
noon to 5 p.m. everyday. Finally, the OCR software within the camera has translated
the objects, or Japanese text, on the package into Italian, the native language of the
traveler, and our traveler is now able to identify a Japanese versionof the medicine
being sought.
All of this AR magic is made possible as a result of something known as an
augmented reality browser, similar to the browsers used on personal computers, to
navigate the world through the lens of the smartphone camera.The video feed pro-
964
vided by the smartphone, combined with a network Internet connection to an online
database of information, transforms the environment seen by the traveler through the
browser into a world flled with relevant information, in the native language of the
traveler. Using the markers, location, and/or objects we have just mentioned, the AR
browser is able to generate the appropriate information to overlay on the environ-
ment the traveler sees through the smartphone camera.
Illustration 2: This is an augmented reality browser window seen through a smart-
phone camera (Source: Wikipedia)
3. Aumented reality technology with markers, location ad objects
Markers represent the least technologically advanced means for implementing
AR. A marker is a special type of object, such as a barcode, that is consciously added
to the environment in order to provide information. In most cases it requires that
the environment be altered by actual physical human intervention, since barcodes
and other common forms of markers are not usually embedded on anything other
than products. One might imagine, for example, a sign on which a barcode has been
pasted by a tourist organization, in cooperation with the local government.
The barcode or marker can be scanned by the camera video feed from a smart-
phone, and that code will be deciphered by software communicating with an online
database to overlay the appropriate textual information on the browser window to
the augmented reality world. The barcode or marker might, in our sign example, in-
dicate we are in a tsunami zone and need to exercise caution. Perhaps we also noted
the wave on the sign, but were not certain if this indicated this was an ideal location
for swimming or surfng, or if danger was indicated. The amount of information em-
Thomas Richard
Schalow
Bringing
Meaning to
Tourism: The Use
of Augmented
Reality to
Navigate Foreign
Language
Environments
965
bedded in a barcode can be large, so we might also learn the sign was put in place last
year, and therefore the information is fairly current and probably worth considering.
Illustration 3: This is a Japanese sign indicating an earthquake and tsunami zone
Augmented reality applications based on location, as provided by gravimetric
or GPS technology, do not require the physical environment be changed in order to
overlay information on the environment. These applications are more fexible than
marker-based AR, but require greater technological sophistication from the AR de-
vice than marker-based systems. They of course require that the AR device, such as
the smartphone, be equipped with a means to compute relative location on the basis
of GPS signals. At present this is one of the limiting factors in the adoption of AR,
as commercial GPS devices do not yet have the accuracy to locate objects as small
as a sign with pinpoint accuracy. Although the military is able to blow up ships,
buildings, and other objects with missiles coordinated by GPS signals, commercial
users can presently expect nothing more than to be able to identify buildings on the
basis of location, and even in these cases errors of up to hundreds of meters is not
uncommon. Therefore, the type of information that can be reliably overlaid on an en-
vironment using location-based AR is still limited by errors in GPS signal accuracy.
Nonetheless, once a location is established, AR technology can begin to provide the
user with a wide variety of information about the area. That information can come
from databases used by Twitter, Facebook, or other online sources of social informa-
tion.
The fnal form of AR we are looking at involves object recognition, and the
subset of OCR-enabled translation. Although we do not need to know the location
of the object to provide this type of information, the processing requirements for
translating information using object recognition place a substantial strain on todays
smartphones, and robust network connections are also a necessity in most cases. In
966
our sign example, no barcode or marker would be required if we could rely on our
smartphone and its OCR software to translate the Japanese characters on the sign.
There would be no need to physically alter the environment in order for our visi-
tor to have the same information, as a result of the OCR-based translation, native
Japanese visitors to the area would have. Fortunately, a great deal of effort has gone
into developing OCR and object recognition software, and a number of smartphone
applications have already been introduced.
4. Smartphone translation applications and limited augmented reality
The potential benefts of augmented reality are so great because browsers will
be able to combine marker, location, and object recognition technologies to construct
an entire world of information unavailable to tourists who cannot read the language
used in the country where they are traveling. It will even be able to augment the
information available to native citizens and language users, overlaying additional
contextual information taken from online databases.
Today tourists already have access to more limited applications that use just
a small spectrum of AR technology. Instead of viewing the world through an aug-
mented reality browser, for example, we are able to use our smartphone cameras and
software such as Word Lens, Camera Translator, CamDictionary, Photo Translator,
or similar applications to provide limited OCR-based translation of signs. Even these
more limited applications work remarkably well, with a great number of language
choices, and can be of tremendous assistance to travelers. It is now possible to travel
in Japan, China, or other exotic environments and be able to read at least some of the
signs without a working knowledge of the foreign language. Lesser used languages,
such as Lao or Cambodian, at present lack the population base to make the develop-
ment of OCR translation software commercially viable, but it is only a matter of time
before we will also have applications to make reading signs in Vientiane or Phnom
Penh a trivial matter.
Some countries, of course, already translate some of their signs into English,
particularly in large cities, and this translation greatly facilitates ease of travel. Trav-
el to more remote and less densely populated locations, however, still presents dif-
fculties, as governments lack the economic resources to provide translations for
each and every sign that is put up. Moreover, while English translations are certainly
welcome, they are not a solution to the navigation problems of every foreign visi-
tor. As we pointed out at the beginning of this paper, due to declining transportation
costs and other factors, more people who lack basic foreign language skills are now
traveling abroad. English translations might be useful to more educated tourists, but
are of limited use in their raw form to these travelers.
Thomas Richard
Schalow
Bringing
Meaning to
Tourism: The Use
of Augmented
Reality to
Navigate Foreign
Language
Environments
967
Illustration 4: This is a Cambodian road sign with an English translation
Nonetheless, there is still a case to be made for providing English or some
other Western language translations on signs whenever possible, and particularly in
countries where text is not alphabet-based, or not Latin in its form. English itself is
an object that can easily be decoded by OCR software available today. The Italian
visitor to Cambodia, for example, might not be able to read a sign in the native Cam-
bodian/Khmer alphabet, but if that sign included an English translation an educated
traveler would probably be able to understand the information. With the assistance
of an OCR translation application, the English itself could even be translated into
its Italian equivalent to make the information available to anyone able to read Ital-
ian. Eventually, when Khmer-Italian translation software is available the English
language translations would become unnecessary, but until that time they provide a
method to make the information accessible to the widest possible group of travelers.
5. Future challenges for augmented reality - hardware issues
Although it is most likely that augmented reality will initially become a part of
the travelers world through a device such as the smartphone, Arth and Schmalstieg
(2011) believe that smartphones are not yet truly ready to deliver full augmented
reality to the traveling masses. One basic problem smartphones have is they are
not meant to provide continuous camera operations for long periods of times due
to limitations in battery technology. I do not believe this is a serious problem, how-
ever, as I expect augmented reality will be used only when needed, rather than on a
continuous basis. We will continue to view the world through our own eyes, unaided
by AR technologies,to gather most of the information we need, and only access an
augmented reality version of the world when confronted by needs such as language
translation.
Another serious issue relates to network speeds, and network availability. Bar-
codes and markers can be decoded locally, without the need for an Internet con-
nection, if the set of phrases or languages required is limited. For true AR imple-
mentations, however, a broadband Internet connection to enable access to databases
where larger volumes of information are stored will be required. For more sophisti-
968
cated AR environments a variety of databases will be queried, to provide the maxi-
mum amount of available information. In fact, the amount of information available
through AR browsers, even today, has the potential to overwhelm the user, and many
slower networks, with essentially irrelevant information. AR technology will require
sophisticated fltering algorithms to discard information we do not need, while at the
same time gathering as much relevant information as possible. The flter will also
be required to authenticate information, guaranteeing it is accurate, unbiased, and
current.
Unfortunately, due to the fact that augmented reality development for smart-
phones is at the mercy of hardware and network service providers, who make their
marketing and development decisions based on factors that may not take the needs of
augmented reality developers into account, hardware constraints on augmented real-
ity will not be easily overcome. We will certainly eventually begin to see independ-
ent augmented reality devices brought to the market, perhaps in the form of goggles,
for example, that are developed specifcally with the power and network needs of
augmented reality applications in mind.
6. Augmented reality: software and social issues
The most diffcult issues confronting AR and other forms of pervasive comput-
ing will present themselves as software issues and, as Kang and Cuff (2005) have
observed, social issues. This is in part due to the fact that individuals will create
most of the information needed to construct a useful augmented reality environment,
probably using one of the major AR platforms available today, such as Layar, junaio,
or Wikitude. When I, for example, post a remark on a social information network, in
my native English language, about the sign I have seen on some store near my home
in Japan, that remark becomes part of the database of information available to all
subsequent travelers to the area. If this remark was posted to Twitter, for example,
that remark now becomes part of the Twitter database. If I have noted that this par-
ticular store is open for business from 9 a.m. to 5 p.m., that information could now
also become available to any augmented reality environment that is created using the
Twitter database.
In this example I have not relied on OCR translation software to interpret the
sign, but merely on my own linguistic ability to produce a translation of the informa-
tion posted on the sign in Japanese. However, my linguistic ability may be incom-
plete, or even so defcient as to produce errors. Perhaps the sign actually indicated
that the store was open during these times, except on Monday, when it is closed for
business. As a result of my own incomplete knowledge, or misunderstanding, I have
introduced a piece of data that is only partially correct. Users of the information I
have provided will view an augmented reality world that is created with at least some
information that is not entirely true.
Thomas Richard
Schalow
Bringing
Meaning to
Tourism: The Use
of Augmented
Reality to
Navigate Foreign
Language
Environments
969
This error may have been introduced as a result of carelessness on my part. Per-
haps I failed to read the entire sign due to a lack of time. Perhaps this is the frst and
only error I have made in a post to the Twitter database. Although my translations are
normally reliable, in this particular instance the translation was defcient.However,
there is also the possibility that I consistently post incorrect information. Perhaps my
linguistic skills are so poor that I should not be subjecting others to what amounts
to little more than slightly educated guesses about my environment. Who, though,
knows that the information I provide should not be trusted?
It is also possible that I am purposefully posting incorrect information. Perhaps
I have a grudge against the owner of this storeand am consciously seeking to destroy
the stores business by providing incorrect information. Again, who is to know what
my motives are when I post the information, and whether or not that information can
be trusted?
These issues are so important because augmented reality is a constructed real-
ity, constructed by information that is subject to concerns about credibility, timeli-
ness, and relevance. To the extent that any of these are compromised, the environ-
ment itself becomes compromised, and perhaps seriously so. These same issues are
present on the World Wide Web, of course, and have not been resolved there. It is as
possible to fnd incorrect, biased information as it is to fnd accurate, relevant infor-
mation on web pages.The most glaring biases or inaccuracies that could potentially
become a part of AR environments will be eliminated by flter and algorithmsbased
on large number and data sets. If there are 99 positive reviews for a hotel, for exam-
ple, from a source such as Tripadvisor, and 1 negative review, I can probably assume
the 1 negative response was an aberration, and it could be fltered out.
Augmented reality browsers will also certainly, as Yovchevaet. al. (2012) note,
flter the information they provide to the traveler on the basis of some sort of con-
textual awareness software. This would help to overcome the information overload
that threatens to make augmented reality impractical. Viewing too much information
crowded onto a small smartphone screen would hide truly important information. As
Google and other search engines have discovered, the way in which all this fltering
is performed will determine how useful the information will be.
Filtering, however, also represents a form of control over expression, and over
the value assigned to information. Tourist organizations should be proactive in their
embrace of AR technology, working with developers for the major AR platforms to
guarantee information is accurate, unbiased, and current.
There are also important privacy concerns that are related to augmented reality
environments, and these need to be considered by anyone involved with this new
technology. In one sense we are all now consumers as well as products in a new
augmented reality. This idea goes beyond the term prosumer, used by Alvin and
Heidi Toffer (2007) and coined to express our dual function as producer and con-
sumer. As human social products, our actions, our location, our very existence, be-
970
comes the source for data consumed by others. To what degree do I have the right to
deny others knowledge about my location, about who I am, and when I surrender this
right by checking in with a service such as FourSquare, do I lose all control over
those rights? These are some of the concerns I hope the tourist industry will address
as it works with AR developers to set standards and establish methods by which
user-created information can be evaluated for credibility, timeliness, and relevance.
7. Conclusions
A successful tourism experience presents the exotic in a form that is under-
standable and in some way familiar to the traveler. It does not overwhelm the cogni-
tive or sensory inputs of the traveler. The widespread adoption of augmented reality
browsers, and development of the software and information needed to enable them
in any foreign environment, will make the linguistic diffculties and associated cog-
nitive overload of past travel a forgotten inconvenience. Merely viewing the world
through one of these augmented reality browsers, most likely through a device such
as a smartphone, will transform the world and the tourist experience for the traveler.
Thomas Richard
Schalow
Bringing
Meaning to
Tourism: The Use
of Augmented
Reality to
Navigate Foreign
Language
Environments
971
References
Arth, Clemens and Dieter Schmalstieg. Challenges of Large-Scale Augmented Reality on
Smartphones.ISMAR 2011 Workshop: Enabling Large-Scale Outdoor Mixed Reality
and Augmented Reality. 2011. Internet access.
http://www.navteq.com/outdoor_mar2011/challenges.pdf
Cunha, Carlos, Raul Morais, MaximinoBessa, and Manuel Cabral Reis. Contextualized
Ubiquity: A new opportunity of rendering information and services. Journal of Theo-
retical and Applied Electronic Commerce Research. Vol. 5, 3, 2010, pp. 55-64. ISSN
0718-1876.
Curran, Kevin, Denis McFadden, and Ryan Devlin. The Role of Augmented Reality within
Ambient Intelligence.International Journal of Ambient Computing and Intelligence.
Vol. 3, 2, 2011, pp. 16-33. ISSN 1941-6237.
Fragoso, Victor, Steffen Gauglitz, Jim Kleban, Matthew Turk and Shane Zamora. Trans-
latAR: A Mobile Augmented Reality Translator. IEEE Workshop on Applications of
Computer Vision. 2011. Internet access.
http://cs.ucsb.edu/~vfragoso/publications/workshops/131.pdf
Furht, Borko, ed. Handbook of Augmented Reality. New York: Springer Science and Media,
2011. ISBN 978-1461400639.
Julier, Simon, YohanBaillot, Dennis Brown and Marco Lanzagorta. Information Filtering for
Mobile Augmented Reality.IEEE computer graphics and applications. Vol. 22, 5, 2002,
pp. 12-15. ISSN 0272-1716.
Kang, Jerry and Dana Cuff. Pervasive Computing: Embedding the Public Sphere. Washing-
ton & Lee Law Review. Vol. 62, 1, 2005, pp. 93-147. ISSN 0043-0463.
Kent, James. The Augmented Reality Handbook - Everything you need to know about Aug-
mented Reality.2011. ISBN 978-1743040942.
Madden, Lester. Professional Augmented Reality Browsers for Smartphones: Programming
for juniao, Layar and Wikitude. West Sussex, UK: John Wiley & Sons, 2011. ISBN
978-1119992813.
Toffer, Alvin and Heidi Toffer. Revolutionary Wealth: How It Will Be Created and How It
Will Change Our Lives. New York: Doubleday, 2007. ISBN 978-0385522076.
Yovcheva, Zornitza, DimitrioBuhalis, and Christos Gatzidis. Overview of Smartphone Aug-
mented Reality Applications for Tourism.e-Review of Tourism Research. Vol. 10, 2,
2012, pp. 63-66. ISSN 1941-5842.