Vous êtes sur la page 1sur 6

Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009

CORSE-FINE OPINION MINING


RUIFENG XU, CHUNYU KIT
Department of Chinese, Translation and Linguistics, City University of Hong Kong, Kowloon, Hong Kong
E-MAIL: ruifeng.xu@cityu.edu.hk, ctckit@cityu.edu.hk

Abstract:
Most existing opinion mining systems recognize
opinionated sentences and determine their polarity as one-step
classification procedure. This paper proposes a different
multi-pass coarse-fine opinion mining framework. In this
framework, a base classifier firstly coarsely estimates the
opinion of sentences. The obtained sentence-, paragraph- and
document-level opinions are incorporated in an improved
classifier as features to re-estimate the opinion of sentences.
The updated opinions are feed back to the classifier for
further refining the sentence opinion until the classifier
outputs converge. Three base classifiers are incorporated in
this coarse-fine opinion mining framework, respectively. Their
performances are evaluated on NTCIR-6 and NTCIR-7
opinion analysis dataset. The achieved performance
improvements show that the proposed coarse-fine strategy is
effective to improve the developed opinion mining classifiers.

Keywords:
Opinion mining; Opinion analysis; Coarse-fine opinion
mining; Classifier

1.

Introduction

Aiming at identifying and analyzing the opinions in


text, opinion mining becomes an increasingly interesting
research topic in information extraction and knowledge
discovery areas. The discovered opinions are useful to
many applications. For example, the opinions on products
are helpful to customer purchase decision and manufactory
quality improvement [1], while the opinions on specific
policies from different sources are helpful to improve
government management. Besides, as a fundamental natural
language processing (NLP) technology, the opinion analysis
technique helps to promote research in information
extraction and knowledge discovery such as automatic
summarization [2] and question answer system [3].
Many researches on opinion analysis have been
reported in the recent decades [4,5,6,7,8]. Most of them
focused on the subjective words extraction [4] and opinion
classification at the document [9] or sentence level [10].
However, identifying only opinionated documents or
sentences may not be sufficient [7]. Deep opinion analysis

including the opinion holder identification, opinion target


analysis therefore becomes a hot research topic [11].
The existing opinion analysis techniques are generally
camped into three approaches. The fist approach is based on
sentiment knowledge and mainly utilizes the knowledge on
sentiment words and some opinion-related heuristic rules
for opinion analysis [12,13]. The second approach is based
on machine learning which incorporates sentiment words,
word bi-grams, syntactic features and opinion-related
linguistic features into a machine learning based classifier.
The popular used classifiers including Nave Bayes (NB)
[14], Maximum Entropy (ME) [15] and Support Vector
Machine (SVM) [16]. The third approach combines
sentiment knowledge, machine learning and a general
linguistic framework for opinion analysis such as semantic
role labeling in the FrameNet [17].
Most existing opinion mining systems recognize
opinionated sentences and determine their polarity as
one-step classification procedure. These systems have two
shortcomings. The first one is that only the inner-sentence
and neighboring-sentence opinion-related features are
adopted while the paragraph- and document-level opinion
features are ignored. The second one is that the whole
opinion mining is separate opinionated sentence recognition
procedures while the analyzed opinion information does not
help the recognition of opinions in the rest sentences. To
overcome these shortcomings, this paper proposes a new
framework, which analyzes opinions under a coarse-fine
analysis framework. This framework is a multi-pass
analysis procedure. A base classifier is firstly applied to
estimate the opinion of each sentence in the document. The
analyzed sentence opinions generate the paragraph and
document opinion. The obtained paragraph- and
document-level opinions are then incorporated in a
improved classifier as new features to re-analyze the
sentence opinions. The refined opinions of sentences,
paragraph and document are feed back to the classifier to
further refine the opinion analysis. Such circles terminate
until the analysis outputs converge. Three base classifiers
and one improved classifier are developed in this study.
They are incorporated in the proposed coarse-fine opinion

978-1-4244-3703-0/09/$25.00 2009 IEEE


3469

Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
mining framework, respectively. Their performances are
evaluated on NTCIR-6 and NTCIR-7 opinion mining
dataset. The experimental results show that by adopting the
proposed framework, the performances for these classifiers
are all improved, especially the precisions. The achieved
promising results support the idea of a multi-pass
coarse-fine opinion mining framework.
The rest of this paper is organized as follows. Section
2 briefly reviews the existing works on opinion mining.
Section 3 gives some basic concepts and terminologies in
this study. Section 4 presents the coarse-fine opinion
mining framework. Section 5 presents the implementation
of three base classifiers and improved classifier. Section 6
gives the evaluation results and finally, Section 7 concludes
this paper.
2.

Literature Review

Early works in opinion analysis focus on the


identification and polarity determination of sentiment
words. Hatzivassiloglou and Mckeown predicted semantic
orientations of sentiment adjectives by analyzing adjective
pairs occurring in the corpus [4]. Turney and Littman
adopted a bootstrapping strategy to determine polarity of
each new sentiment word using a set of seed sentiment
words [18]. Kamps utilized the lexical relations maintained
in WordNet to estimate the relation between new sentiment
word and seed sentiment words for polarity determination
[19].
Recently, various opinion analysis techniques are
proposed to identify document- and sentence-level opinions
in different applications domains such as news articles [13],
product reviews [1], movie reviews[20] and web blogs [21].
These techniques can be generally categorized into three
approaches: (1) the approach based on sentimental
knowledge; (2) the approach based on machine learning;
and (3) the approach combining sentimental knowledge,
machine learning and a general linguistic framework. The
first approach mainly utilizes the linguistic knowledge on
sentiment words and some opinion-related heuristic rules as
the clues for opinion analysis. The sentiment lexicon acts a
core role in this approach. For the giving text, this approach
identifies sentiment words and uses the products of the
polarities of these sentiment words as the features to
recognize the opinionated sentences and determine their
polarities. Furthermore, some opinion-related heuristic rules
are applied to improve opinion analysis. The typical
systems following this approach include [7] on English and
[13] on Chinese, respectively. The second approach
normally uses sentiment words and some opinion-related
linguistic features as the observing features. The classifier

based on machine learning algorithms, which is trained


through annotated data, is used to classify input documents
or sentences into opinionated ones or non-opinionated ones.
The sentiment words, word bi-grams, word n-grams,
syntactic patterns, punctuations and topic-relevant features
are always adopted as observing features. The supervised
and unsupervised learning techniques were used to develop
a classifier to recognized opinion sentences. The adopted
classifiers include Nave Bayes (NB) [15], Maximum
Entropy (ME) [5] and Support Vector Machine (SVM) [16].
The last approach combinines sentiment knowledge,
machine learning and a general linguistic framework for
opinion analysis. Kim and Hovy proposed a method based
on semantic role labeling [17]. It collected sentiment words
and opinion-related frames from the FrameNet corpus, and
a maximum entropy based model is applied to label the
semantic roles of opinion related frames in a sentence.
Finally, the opinion holders and topic are determined by
mapping semantic roles to them.
3.

Basic Concepts

A typical opinion expression consists of five key


components, (1) Opinion Holder which is the governor of
an opinion. It normally refers to a person, a state or an
organization as well as the corresponding pronouns; (2)
Opinion Operator which is the verb indicates an opinion
event, The typical opinion operators including
(warning) (emphasize) and (point out). Some
operators bring opinions by themselves such as
(praise) is always for positive; (3) Opinion Object which is
the target of the opinion; (4) Opinion Word/Sentiment Word
which reflects the opinion polarity, i.e. positive, neutral or
negative. They are further classified into: a. Context-free
Opinion Word (CFOW) whose polarity is constant
irrespective of context, e.g. (perfect) is absolute
positive and (bad) is negative; b. Context-dependent
Opinion Word (CDOW) whose polarity is determined by
their context. For example, is positive when it is
used in the context of talk shows (meaning burlesque); but
it is negative when it is used in the context of politics
(meaning absurd); and c. Object-dependent Opinion Word
(ODOW) is the neutral word carrying different polarities
when associated with different opinion objects. For
example, (high) expresses positive sense when
collocating with (performance) but brings negative
sense when collocating with (debt). For practical
reasons, this kind of words are processed as CDOWs; and
(5) Opinion Indicator, which is the word indicating the
orientation of an opinion or the orientation tendency of

3470

Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
multiple opinions. They include negation conjunctions,
continual conjunctions, adverbs and adverbial phrases
directly indicate the polarity of the opinionated sentence.
4.

Most existing opinion mining techniques regard


opinionated
sentence
identification
as
one-step
classification problem. The linguistic features and
statistical-based features in the observing sentence are
utilized in the classifier to determine the opinion and
polarity of the sentences. These techniques ignore the
influence of opinions of the paragraph, document and the
neighboring sentences to the opinion of current sentence.
The observation on NTCIR-6 corpus and NTCIR-7 training
corpus shows that a sentence in a strong polarity document
has higher probability to be the same polarity while a
sentence in a factual document tends to be factual too.
Naturally, the paragraph-, document-level and neighboring
sentence-level opinions should be considered in the
sentence opinion analysis. Meanwhile, humans normally
analyze the opinion trend of a document coarsely in the first
step and then remove the ambiguities in sentence opinion
based on the opinion of document and neighboring
sentences. It motivates the design of a coarse-fine opinion
mining framework. This framework has multi-pass
coarse-fine analysis which is described below.
Input: Document D consists of sentences S0, S1, Si Sn
Step 1. Use the base classifier, Cbase, to analyze the
opinion of each sentence in D. The output is the polarity
value, Pol(Si).
Step 2. Estimate the polarity of D and each paragraph
P.

1 n
Plo( Si )
n i =1

1 j
Pol ( P) = Plo( Si )
j i =1

The Implementation of Base Classifiers and


Improved Classifiers

To test the effectiveness of the proposed coarse-fine


opinion mining framework, three base classifiers and a
improved classifier are implemented.

Coarse-Fine Opinion Mining Framework

Pol ( D) =

5.

(1)

(2)
Step 3. Use the improved classifier, Cim, to estimate
the opinion of each sentence, Pol(Si)*. Cim incorporates
paragraph-, document- and neighboring sentence-level
opinions as new features.
Step 4. Update the document and paragraph opinion
using Pol(Si)*.
Step 5. If the sentence and document opinion mining
output converge, terminate. Otherwise, go to Step 3.

5.1.

Base Classifier One: Opinion Lexicon-based


Classifier

This base classifier is similar to NTU system [13]. It


uses opinion holders, negations and sentiment words as the
clues for finding opinionated sentence. The polarity of
opinionated sentence is determined by summarizing the
polarity of known opinion words while the polarity will be
reversed if a neighboring negation is detected. If the sum
greater than 0, the sentence tends to positive. Otherwise,
negative.
5.2.

Base Classifier Two: Machine Learning based


Classifier

The second classifier is based on machine learning.


The occurrence of opinion holders, opinion operators,
sentiment words, negations and degree adverbs are
regarded as discriminative features. A classifier based on
support vector machine is adopted.
5.3.

Base Classifier Three: Machine Learning based


Multi-features Classifier

Three levels of features are adopted in this base


classifier including the punctuation-level, word-level and
collocation level. Table 1 gives the description of these
features. The more information can be found in [21].
A linear multinomial SVM classifier is implemented
for opinionated sentence recognition. While describing the
sentences as a set of values of opinion-related features,
these sentences are transferred to points in a
multi-dimension space. For a binary classification problem,
the SVM classifier attempts to find out a hyperplane in the
feature space which separates the positive training
examples from the negatives ones. This classification
hyperplane has the maximal margin to the training
examples. The three groups of features are adopted as the
features for the classifier.

3471

Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
Table 1. Features adopted in base classifier three for opinion
mining
Punctuation level features
The presence of direct quote punctuation and
Word-level and entity-level features
The summary of the probabilities of known opinion
operators
The presence of known opinion word
The presence of known strong opinion word
The sentence polarity score.
The presence of a named entity
The presence of pronoun
The presence of known opinion indicators
The presence of known degree adverbs
Collocation-level features
The presence of collocations between named entities and
opinion operators
The presence of collocations between pronouns and opinion
operators
The presence of collocations between nouns or named
entities and opinion words
The presence of collocations between nouns or named
entities and strong opinion words
The presence of collocations between pronouns and opinion
words
The presence of collocations between pronouns and strong
opinion words
The presence of collocations between degree adverbs and
opinion words
The presence of collocations between degree adverbs and
strong opinion words
The presence of collocations between degree adverbs and
opinion operators

5.4.

distance of two sentences are estimated. These conditional


probabilities are used as features.
Furthermore, the paragraph opinion and document
opinion are estimated by summarizing the sentence
opinions in the paragraph and document, respectively.
A Support Vector Machine based classifier, which
incorporates the output features from each base classifier
and features listed in Table 2, is trained through
semi-supervised learning on NTCIR-6 corpus and more
webpage relevant to the documents. The training algorithm
is described in [21]. The trained classifier analyzes each
input sentence and determines its polarity as the output.
Here, the SVM with linear kernel is adopted to perform
opinionated
sentence
identification
and
polarity
determination.
Table 2. Additional features adopted in complex classifiers for
opinion mining

Sentence level features


P(Pol(si)=positive |Pol(si-1)), values [0,1]
P(Pol(si)=neutral|Pol(si-1)), values [0,1]
P(Pol(si)=negative |Pol(si-1)), values [0,1]
P(Pol(si)=non-opinionated |Pol(si-1)), values [0,1]
P(Pol(si)=positive |Pol(si-2)), values [0,1]
P(Pol(si)=neutral|Pol(si-2)), values [0,1]
P(Pol(si)=negative |Pol(si-2)), values [0,1]
P(Pol(si)=non-opinionated |Pol(si-2)), values [0,1]
P(Pol(si)=positive |Pol(si+1)), values [0,1]
P(Pol(si)=neutral|Pol(si+1)), values [0,1]
P(Pol(si)=negative |Pol(si+1)), values [0,1]
P(Pol(si)=non-opinionated |Pol(si+1)), values [0,1]
P(Pol(si)=positive |Pol(si+2)), values [0,1]
P(Pol(si)=neutral|Pol(si+2)), values [0,1]
P(Pol(si)=negative |Pol(si+2)), values [0,1]
P(Pol(si)=non-opinionated |Pol(si+2)), values [0,1]
Paragraph level features
Pol(P)
Document level features
Pol(D)

Improved Classifier

Using the base classifiers to analyze the opinion of


sentences and document, the coarse analysis results are
obtained. Now, we incorporate the document-level,
paragraph-level and neighboring sentence-level features in
the improved classifier.
For the i-th sentence in the document, labeled as si, we
assume its polarity, labeled as Pol(si), is positive, (its values
including positive, neutral, negative and non-opinionated)
and the polarity of its previous sentences si-1, labeled as
Pol(si-1), is positive. The conditional probability,
P( Pol ( si ) = positive | Pol ( si1 ) = positive )
=

6.
6.1.

P ( Pol ( si ) = positive Pol ( si 1 ) = positive )


P ( Pol ( si ) = positive )

(3)
can be estimated. The conditional probabilities of other
polarity co-occurrence combinations between si and si-1 are
calculated in the same way. Similarly, the conditional
probabilities corresponding to the co-occurrences with

3472

Evaluations
Dataset
In this study, we adopted three set of opinion data.
1. The NTCIR-6 Opinion Training corpus consists of
429 news documents. In its 7,980 sentences, 4,716
ones are annotated as opinionated, including 2,735
positive ones, 986 negative ones and 995 neutral
ones.
2. The NTCIR-7 Opinion Training corpus Simplified

Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
Chinese side has 19 documents in two topics. The
Traditional Chinese side consists of 58 documents
from three topics.
3. The NTCIR-6 Opinion Testing dataset with the
gold answers is adopted for evaluation. It consists
of 27,841 sentences in 2,250 documents
corresponding to 28 topics. In the 17,269
opinionated sentences, 6,894 ones are positive,
5,889 ones are neutral and 4,486 ones are
negative.
The first two sets are used as training corpus and the
third set are used as testing corpus.
6.2.

Classifier 1 Classifier 2 Classifier 3


Strict
P
0.119
0.095
0.253
Evaluation R
0.649
0.461
0.659
F
0.201
0.157
0.365
Lenient
P
0.346
0.284
0.518
Evaluation R
0.441
0.403
0.342
F
0.388
0.333
0.411
By adopting multi-pass coarse-fine opinion mining,
the achieved performances on polarity determination are
given in Table 6, respectively.
Table 6. The performance on polarity determination by
multi-pass analysis

Evaluation Results

Firstly, the performances achieved on opinionated


sentence identification are evaluated. The achieved
performances by three base classifiers are listed in Table 3,
respectively. In the following tables, P is for precision, R is
for recall and F is for F-value.
Table 3. The performance
identification by base classifiers

Table 5. The performance on polarity determination by base


classifiers

on

opinionated

sentence

Classifier 1 Classifier 2 Classifier 3


Strict
P
0.237
0.453
0.469
Evaluation R
0.920
0.384
0.675
F
0.377
0.416
0.553
Lenient
P
0.654
0.741
0.821
Evaluation R
0.852
0.532
0.718
F
0.740
0.619
0.772
The multi-pass coarse-fine opinion mining terminated
after 4.2 passes in average. The achieved performances of
opinionated sentence identification by three classifiers are
given in Table 4, respectively.
Table 4. The performance on
identification by multi-pass analysis

opinionated

sentence

Classifier 1 Classifier 2 Classifier 3


Strict
P
0.256
0.462
0.512
Evaluation
R
0.916
0.380
0.661
F
0.400
0.417
0.577
Lenient
P
0.676
0.789
0.866
Evaluation
R
0.841
0.521
0.709
F
0.749
0.628
0.779
It is observed by adopting multi-pass coarse-fine
opinion mining, all the performance on opinionated
sentence identification are improved. Especially, their
precisions are obviously enhanced, even more than 5%.
Secondly, the performances on polarity determination
of opinionated sentences are evaluated. The achieved
performances by three base classifiers are listed in Table 5,
respectively.

Classifier 1 Classifier 2 Classifier 3


P
0.210
0.215
0.412
R
0.646
0.452
0.643
F
0.316
0.291
0.502
Lenient
P
0.467
0.408
0.674
Evaluation
R
0.436
0.399
0.341
F
0.451
0.403
0.452
It is observed that the achieved precisions are
improved for even 0.120 while the recalls are slightly
decreased which leads to an obvious improvement on F1.
The experimental results show that the proposed coarse-fine
multi-pass analysis framework effective to improve the
overall performance of opinion mining classifiers,
especially for precision performance. Meanwhile, it is
observed that the coarse-fine opinion mining framework
carries high improvement on polarity determination over
opinionated sentence identification.
Strict
Evaluation

7.

Conclusions

This paper presents a coast-fine opinion mining


framework which is different from the most existing
systems. The multi-pass coarse-fine analysis utilizes the
document opinion, paragraph opinion and contextual
sentence opinions to incrementally refining the sentence
opinion analysis output by the base classifier. The
evaluations of three classifiers on NTCIR-6 and NTCIR-7
opinion dataset show all these three classifiers are improved,
especially the precision. These results verify effectiveness
of the coarse-fine opinion mining framework.
Acknowledgements
This paper is supported by a post-doctoral research
fellowship funded by City University of Hong Kong, Hong
Kong.

3473

Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
References
[1] M.Q. Hu and B. Liu, Mining and Summarizing
Customer Reviews, Proceedings of ACM SIGKDD
2004, pp.168-177, 2004
[2] M.Q. Hu and B. Liu, Opinion Extraction and
Summarization on the Web, Proceedings of NCAI
2006, Boston, 2006
[3] H. Yu and V. Hatzivassiloglou, Towards Answering
Opinion Question: Separating Facts from Opinions
and Identifying the Polarity of Opinion Sentences,
Proceedings of EMNLP 2003
[4] V. Hatzivassiloglou and K. McKeown K., Predicting
the Semantic Orientation of Adjectives. Proceedings
of ACL1997, pp.174-181, Madrid, 1997
[5] B. Pang, L.L. Lee and S. Vaithyanathan, Thumbs up?
Sentiment Classification Using Machine Learning
Techniques. Proceedings of EMNLP 2002, pp.79-86,
Philadelphia, 2002
[6] A. Gordon et al. Recognizing Expressions of
Commonsense Psychology in English Text.
Proceedings of ACL 2003, Japan, 208215, 2003
[7] J. WIEBE et al. Learning Subjective Language,
Computational Linguistics, Vol. 30, No. 3, pp.
277308, 2004
[8] Y.Q. Xia, R.F. Xu, K.F. Wong, F. Zheng, The
Unified Collocation Framework for Opinion Mining, ,
Proceedings of IEEE ICMLC 2007, pp.844-850, Hong
Kong, 2007
[9] H. Yu and V. Hatzivassiloglou, Towards Answering
Opinion Question: Separating Facts from Opinions
and Identifying the Polarity of Opinion Sentences,
Proceedings of EMNLP 2003, Japan
[10] E. Riloff, J. Wiebe and T. Wilson, Learning
Subjective Nouns Using Extraction Pattern
Bootstrapping. Proceedings of CONLL 2003,
pp.25-32, 2003
[11] Y. Seki, D.K.Evans, L.W. Ku, H.H. Chen, N. Kando
and C.Y. Lin, Overview of Opinion Analysis Pilot
Task at NTCIR-6, Proceedings of NTCIR-6,
pp.456-463, Japan, 2007
[12] J. Wiebe, T. Wilson, and C. Cardie, Annotating
Expressions of Opinions and Emotions in Language,
Computer Intelligence, Vol. 39, No. 2, pp. 165210,
2005
[13] L. W. Ku, T. H. Wu, L. Y. Lee, H. H. Chen,
Construction of an Evaluation Corpus for Opinion
Extraction, Proceedings of NTCIR-5, pp.513-520,
Japan, 2005
[14] W.H. Lin, T. Wilson, J. Wiebe and A. Hauptmann,
Which Side are You on? Identifying Perspectives at

[15]

[16]
[17]

[18]

[19]
[20]

[21]

3474

the Document and Sentence Level, Proceedings of


CONLL2006, pp.109-116, 2006
B. Pang and L.L. Lee, A Sentiment Education:
Sentiment Analysis Using Subjectivity Summarization
based on Minimum Cuts. Proceedings of ACL 2004,
pp. 271-278, Spain, 2004
E. Riloff, S. Patwardhan and J. Wiebe, Feature
Subsumption for Opinion Analysis, Proceedings of
EMNLP 2006
S.M. Kim and E. Hovy, Extracting Opinions
Expressed in Online News Media Text with Opinion
Holders
and
Topics,
Proceedings
of
COLING-ACL2006
P. Turney, M. Littman, Measuring Praise and
Criticism: Inference of Semantic Orientation from
Association. ACM Trans. Information Systems, Vol.
4, No. 21, pp. 315-346, 2003
J. Kamps, M. Marx, et al. Using WordNet to Measure
Semantic Orientation of Adjectives. Proceedings of
4th LREC, Lisbon, Portugal, 2004.
P. Chaovalit, L. Zhou, Movie Review Mining: a
Comparison between Supervised and Unsupervised
Classification Approaches, Proceedings of HICSS
2005, Hawaii, 2005
R.F, Xu, K.F. Wong, Q. Lu and Y.Q. Xia, Learning
Knowledge from Relevant Webpage for Opinion
Analysis, Proceedings of IEEE/WIC/ACM WI-IAT
2008