Ideogram Based Sentiment Analysis in Japanese Text

Tyler Thornblade

1. INTRODUCTION Network Sentiment Dictionary corpora. They then

expand the dictionary using two thesauri,
In recent years a growing body of literature has tong2yi4ci2ci2lin2 (Mei et al. 1982) and the Academia
applied techniques for sentiment analysis to languages Sinica Bilingual Ontological Wordnet (Huang et al.
other than English. While many of these works apply 2008).
similar techniques across differing languages, a
To develop character sentiment scores, they mine
technique notable for its specificity to ideographic
this dictionary seeking counts of the number of positive
languages has arisen: rather than just considering
and negative words in which each character appears.
sentiment at the word, sentence, and document level
fpci and fnci denote the frequencies of a character ci in
sentiment is also analyzed at the level of the constituent
the positive and negative words, respectively; n and m
characters of a word.
denote total number of unique characters in positive
This technique may be of unique value to and negative words, respectively (Ku et al. 2006).
ideographic languages for two reasons. First, all
multi-character words consist of characters that
themselves have distinct meanings (or sets of
meanings) (Henshall, 1988). Thus, information on
individual characters can inform judgments made on
composite words. Second, the concept of a word is not
well defined in some ideographic languages and many
errors may be introduced by the “word segmentation”
step in processing (Zagibalov & Carroll, 2008).
Moving to a character-based approach can avoid the The score for a character is:
need for segmentation.
Two recent papers apply this technique to the
Chinese language. In this work, we apply the same
approach to Japanese. We theorize that techniques for The score for a word is simply the average of the
assigning sentiment scores to individual ideograms in characters in the word:
order to develop sentiment scores for words will not be
as effective in Japanese as they are in Chinese without
some significant modifications.

2. RELATED WORKS
2.1. Ku, Liang, and Chen
Ku, Liang and Chen (2006) introduce a character-
based technique using a simple approach. They first
construct a sentiment dictionary at the word level. In
their paper, the base dictionary is constructed from the
General Inquirer (Stone et al. 1966) and the Chinese
Thus words with strong positive sentiment will have
values much greater than zero, and negative words will
likewise have values much less than zero. Words with
weak sentiment scores will be found near zero. They
use this technique to expand their small seed dictionary
to be able to handle novel words that are composed of
known characters.
 2.2. Huang, Pan and Sun
Huang, Pan and Sun (2007) use a very similar
approach. As their base dictionary they use the
NTCIR emotion dictionary (Kando 2008) and also
utilize tong2yi4ci2ci2lin2 as their thesaurus. Their
approach is otherwise identical and they cite Ku et al.
for this technique.
 2.2. Results
Both works in this section report reasonable system
results for the sentiment analysis task at the sentence
or document level, yet neither paper reports the results
at the word or phrase level. Therefore, it is hard to
establish a baseline for the performance of this phase.
3. HYPOTHESIS
The technique of Ku et al. at the word level can be
considered analogous to the “bag of words” approach
to sentential understanding. Much as “bag of words”
ignores syntactic information encoded in the sentence,
this simple approach ignores information contained in
the composition of multi-character words. We theorize
that system performance in Japanese will be limited by
this lack of compositional knowledge.
3.1. Linguistic foundation
The Japanese language uses three different writing
systems: Chinese characters and two phonetic scripts
(Jorden and Noda, 1998). A significant portion of the
Japanese vocabulary consists of loanwords from
Chinese that are written with the same or similar
characters. It is compounds of this nature that seem
most amenable to the technique of Ku et al.
Additionally, there are words which combine
Chinese characters with script (referred to as
okurigana) and words which are written entirely in
script (referred to as kana). These latter words seem
less relevant, because unlike Chinese characters the
individual script characters do not have independent
meanings. Since Chinese does not have any equivalent
script, this is a class of words on which performance in
Japanese is likely to suffer compared to that of
Chinese.
In Japanese there are five canonical ways in which
characters can be combined to make two-character
compounds (Yamamoto, 2008). These are:
1. Both characters have the same meaning.
2. The characters have opposite meanings.
3. The top character modifies the bottom
character.
4. The bottom character is the target, direct
object, or complement of the top character.
5. The top character negates (“flips”) the meaning
of the bottom character.
The first two classes would not seem to present a
problem for the “bag of words” approach as the
sentiment values will either reinforce or cancel each
other out, respectively. However, the remaining three
classes may present a problem as each of these classes
can result in a word whose meaning is not simply the
sum of its parts.
3.2. Experiment
In order to evaluate the efficacy of this technique in
Japanese, we formulate a simple experiment as a proof
of concept. Using the Japanese sentiment dictionary of
Kaji and Kitsuregawa (2007) we apply the approach of
Ku et al.
The Kaji and Kitsuregawa data is not intended to be
a word-based dictionary and contains a significant
number of bigrams and trigrams. Since these
multiword entries are amenable to syntactic analysis,
they are not the target of this experiment and the first
step is to clean the data and remove them. This
reduces the number of usable entries from 10,000 to
2386.
Next, we extract the unique characters and generate
sentiment scores for every Chinese character in the
corpus using the same approach as Ku et al. We do
not generate sentiment scores for script characters for
the reason cited in section 3.1. This results in a total
of 954 scored characters.
Finally, we generate sentiment scores for each word
in the corpus using the character scores. We evaluate
the results by comparing the sign of the score from the
sentiment dictionary to the sign of the score by our
method, ignoring intensity.
It should be stressed that this is considered at best a
proof of concept in applying this technique to Japanese
and this is not intended to be a rigorous experiment.
By training and testing on the same corpus we
effectively simulate perfect knowledge of the sentiment
scores of the words. Thus, the results should be
interpreted more as an upper bound on the
performance of the approach than as results for
comparison to other sentiment analysis systems.

4. RESULTS

4.1. Overall results

Overall results for this approach are surprisingly

high, with the precision scores being fairly impressive
for both negative and positive sentiment. Although we
cannot draw conclusions about whether this level of
performance is achievable in a real-world system, it
does show rather emphatically that despite its
simplicity the technique does not suffer from structural
limitations that would prevent it from being effective
given a reliable source for word-level sentiment data.

4.2. Detailed results for character sentiment  
analysis
In order to show that the system is indeed
characterizing character sentiment in a manner similar
to that of a human, we present the top 10 most positive
and negative characters along with their most common
English meaning or set of meanings. We believe the
results are intuitive.
5. ERROR ANALYSIS
In this section we examine the output of the system
in an attempt to identify systematic errors. Due to time
constraints, it was not possible to analyze all errors.
Instead, we attempt to choose a representative subset.
Of the 475 errors that occurred, 100 were selected for
analysis: 50 from the false positives and 50 from the
false negatives. This list was further pruned to
eliminate bigrams, words that contained script
characters as part of the compound, and words that
consisted of only a single Chinese character. The
motivation for this pruning was that we are interested
most in errors due to lack of compositional knowledge,
and the above classes of words cannot by their nature
demonstrate such errors.
 After pruning we arrive at 15 false positives and 31
false negatives for analysis. For each word, we
examine it to see if a compositional explanation exists
for the error. E.g. for composition class 5, where the
first character negates the second, we look for cases
where the sentiment of the second character is opposite
in sign to the true sentiment of the word.
5.1 Errors due to missing compositional   develop any kind of sentiment score for a word
knowledge
In this section, references are made to the
compositional classes introduced in section 3.1.
Of the false positives, 33% were found to be
explained by a lack of compositional knowledge.
There were two classes of such errors. The first was
composition class 5, or compounds where the first
character negates (or flips) the meaning of the second
character. An example is the word “ 無 難 ” (safety)
which is formed of the characters “not” and “difficulty,
hardship”. Although both characters have negative
sentiment scores, the overall score should be the
reversed value of the second character. These errors
represented 6.7% of the total.
The second is composition class 3, specifically
compounds where the first character is emphasizing a
positive character. An example is “ 重宝”(priceless)
consisting of “heavy” and “treasure”. The first
character is associated with a negative sentiment score,
yet in this compound it serves to enhance a positive
character and thus the score should be positive. These
errors represented 27% of the total.
Of the false negatives there were again two classes
of errors, and 54.8% of the errors can be explained by
them. The first class is again composition class 5
representing 3.2% of the errors.
The second class is not one of the five
compositional classes introduced in section 3.1. It is a
novel group making up 51.6% of the errors and
consisting of words that include the character “ 的 ” .
This character has no meaning in itself for the words in
this class; it simply changes the part of speech of the
word. Therefore, it should not influence the word’s
sentiment score, but since it has a large negative value
it results in many false negatives.
If the results of this sample were representative
across the corpus, it would indicate that roughly 27%
of all errors are due to compositional knowledge.
5.2 Script characters
In section 3.1 we introduced the fact that there are
two major classes of words containing script
characters: those entirely composed of script, and those
combining both script and Chinese characters.
For the former, 150/475 or 31.6% of positive
sentiment errors were due to the fact that we could not
consisting entirely of script. This class made up
201/536 or 37.5% of the negative sentiment errors.
It is difficult to gauge the impact of script
characters on the latter class, although it is likely
nonzero. As a result, we can conclude that at least
34.7%, and possibly more, of all errors were due to
script characters.
5.3 Problems with source data
The simple methods used to prune bigrams and
trigrams from the source data were not completely
effective. Examination of the output shows that
approximately 4-5% of the entries in the training
corpus are bigrams, which could introduce noise into
the data.
Finally, although for the purpose of this paper the
sentiment dictionary of Kaji & Kituregawa was
considered the gold standard, we must point out that
there are several items in the error analysis that call
into questions their sentiment scores.
Some specific examples are 無用 and 不用, both of
which mean “useless” yet quizzically received high
positive sentiment scores and showed up as false
negatives for positive sentiment in these results.
5.4 Detailed ErrorAnalysis
The first figure below shows the full list of multi-
character false positives. The column headings are
respectively the word itself, the score as calculated by
Kaji & Kitsuregawa, the score for the word via the
method of this paper, and finally the scores for the
individual characters in the word(Kaji & Kitsuregawa,
2007).
 The second figure below repeats this analysis for
the false negatives. The detailed analysis was not done
for all words because their sentiment value was
dominated by the single character “的” which has a
very high negative value of -24.
6. CONCLUSIONS AND FUTURE WORK
6.1 Conclusions
We hypothesized in section 3 that the technique of
Ku et al. would be less effective on Japanese due to
characteristics of the language. After performing the
experiment, the results are not exactly what we had
supposed – yet in many ways are more interesting than
what we had expected.
First, the overall results were quite good. F1 scores
above 0.7, while perhaps not state of the art, are
certainly not poor. Although this was unexpected, it
has the benefit that it is easier to draw conclusions
from good results than from bad and we can at least
conclude that this general method does appear to hold
promise.
Second, despite the surprise of the overall results,
the hypothesis that there would be structural issues
with Japanese was shown to be accurate. As much as
60% of the errors can be explained between the
compositional rules and the system’s inability to
process script characters.
6.1 Future work
The overall results encourage further work in
adapting this technique to Japanese. The first step
should be to apply the technique with more rigor. A
simple way to do this would be to do an annotation
study of novel content and apply the character
sentiment data developed in this work to it. From this,
meaningful conclusions could be drawn about the
efficacy of the approach. A more complex experiment
would be to develop an independent Japanese sentiment
dictionary and then perform the experiment; the
sentiment dictionary of Kaji & Kitsuregawa a large
amount of data and it would be ideal to start from a
more robust corpus.
Next, it would be interesting to experiment with
compositional features. Preliminary results seem to
indicate that approximately 27% of the errors could be
addressed with compositional knowledge and
appropriate logic. A potential barrier to this is that we
are not aware of any current lexical resources that
could be used to implement it.
Finally, a large percentage of the errors
(approximately 35%) are due to script characters.
Performance cannot be improved on these words using
the techniques of this paper since the script characters
are not sentiment bearing. A hybrid approach could,
however, be applied that combines character and word-
based sentiment analysis to improve performance.
REFERENCES
[1] Rui-hong Huang and Le Sun and Long-xi Pan. "SCAS
in Opinion Analysis Pilot Task: Experiments with
sentimental dictionary based classifier and CRF
model" Proceedings of the Sixth NTCIR Workshop
Meeting on Evaluation of Information Access
Technologies: Information Retrieval, Question
Answering and Cross-Lingual Information Access,
2007.
[2] L. W. Ku, Y. T. Liang, and H. H. Chen, "Opinion
extraction, summarization and tracking in news and
blog corpora," Proceedings of AAAI-2006 Spring
Symposium on Computational Approaches to
Analyzing Weblogs, 2006.
[3] Kaji, N. & Kitsuregawa, M. "Building Lexicon for
Sentiment Analysis from Massive Collection of HTML
Documents" Proceedings of the 2007 Joint Conference
on Empirical Methods in Natural Language
Processing and Computational Natural Language
Learning (EMNLP-CoNLL), 2007.
Processing. Cambridge Studies in Natural Language
Processing. Cambridge: Cambridge University Press.
[10] STONE, Philip J., SMITH, Marshall S., OGILIVIE,
Daniel M., DUMPHY, Dexter C. The General
Inquirer: A Computer Approach to Content Analysis.
MIT Press. 1966.
[11] Kando, Noriko. "Evaluation of Information Access
Technologies: Information Retrieval, Question
Answering, and and Cross-Lingual Information
Access." The 6th NTCIR Workshop (2006/2007). 01
December 2008. <http://research.nii.ac.jp/ntcir/ntcir-
ws6/ws-en.html>.
[12] Jordan, Eleanor Harz and Noda, Mari. Japanese: The
Written Language. Boston: Chen & Tsui Company.
1998.

[4] Kim, S. and Hovy, E. 2004. "Determining the

sentiment of opinions" Proceedings of the 20th
international Conference on Computational
Linguistics (Geneva, Switzerland, August 23 - 27,
2004). International Conference On Computational
Linguistics. Association for Computational
Linguistics, Morristown, NJ, 1367.

[5] Yamamoto, Natsuhiko. "漢字検定２級体験記　熟語

の構成." 山本夏彦の本　つかぬことを言う. 20
October 2008.
<http://www.fujitana.net/kanken/2kyuutaikenki/2jukug
onokousei.html>.

[6] Henshall, Kenneth. A Guide to Remembering

Japanese Characters. Tokyo: Tuttle Publishing, 1988.

[7] Zagibalov, T. and J. Carroll "Automatic seed word

selection for unsupervised sentiment classification of
Chinese text" Proceedings of The 22nd International
Conference on Computational Linguistics (COLING),
Manchester, UK, 2008.

[8] Mei, J., Zhu, Y. Gao, Y. and Yin, H..

tong2yi4ci2ci2lin2.Shanghai Dictionary Press. 1982.

[9] Huang, Chu-Ren, Ru-Yng Chang, and Shiang-bin Li.

2008. Sinica BOW: A bilingual ontological wordnet.
To appear in: Chu-Ren Huang et al. Eds. Ontologies
and Lexical Resources for Natural Language