Classification of tweets in real-time and detecting live
trends
A Sumaiya1
P. G. Scholar, Computer Science and Engineering
B S Abdur Rahman Crescent University
Chennai, India
Sumaiya.mustafa93@gmail.com
Abstract With the ever growing interest to tweet and re- tweet,
summarizing the central concept from a whole lot of them is
getting more and more critical. Its criticality can be attributed to
arriving at a result via virtual debate through technologies. Many
of the technological predictions have been proven both right and
wrong. While the previous works had focussed much on the
classification of tweets into their social focus roughly, this project
focuses on detecting the live trends as well. By doing so, a fair
amount of correctness in summarization is lost. This is why this
project deploys the ranking technique at a non-streaming basis
which implements TCV rank summarization algorithm for two
main operations which are namely rolling up and drilling down
of tweets. Rolling up summarizes the central theme and retrieves
the trend on any one particular point of time whereas drilling
down retrieves all the tweeted domains on which trends were
alive between any two points of time. On a broad view of the
project, tweets are clustered to centrality using k-means
algorithm and are ranked by their frequency. Sumblr has been in
the usage to handle the streaming tweets online so far. But
Twitter API(Appication Programming Interface) is made use of
in our project to handle and retrieve tweets. It helps in making
business decisions. In other words, it tries to become a
technological judge.
Keywords Sumblr, API, Roll up, Drill down
I. INTRODUCTION
Twitter is one of those powerful media available in
the contemporary world to voice opinion on any particular
issue, event, commemorative or news. It works very simple in
all the ways right from its login to tweeting. It is mandatory
that the user of Twitter must hold an account with it. Anybody
who grants permission to follow them can be followed on
Twitter. One also has the privilege of seeing, following, re-
tweeting on the tweets of one's follower. It is also heavily used
all around the globe and many predictions are being made
through Twitter as well.
Various occurences in everyday life through TV, internet
and numerous other sources induce and inspire Twitter users to
R ShriRam2
Professor, Department of CSE
B S Abdur Rahman Crescent University
Chennai, India
shrionsong@gmail.com
tweet on those. Having deeply and closely watched the Twitter
micro blogs for months, its typology has been putforth into
four broad classification as follows. Social ,Political and
economic events which were broadcasted on TV are used as a
forum are discussed in Twitter and virtual or unorganised
opinion polls also arise spontaneously which give rise to
predictions. Sometimes news are released in Twitter even
before by the other media .This has led to the evolution of a
typology of tweets known to be as news. News which are mere
reports of incidents from all streams and walks of life become
very sensational topics with discussions and tweets on Twitter.
An event-graph based method technique is used to extract
information which customizes summaries depending on the
topic since the perfect length of summaries vary from topic to
topic. The page ranking algorithm is only drawn from the
previous works to fragment event graph and identify the
intricate orientations of what to be summarized. This made our
primary results considerably worthy and crisp by human
judges. These also provide short survey of datasets to evaluate
the design from the previous work for automatic
summarization tasks [1]. Over the event-graph based
technique, an event summarization algorithm is chosen as it
tells a detailed problem statement and methodology. . Tweets
that are repetitive and related are summarized using highly
sophisticated techniques. Tweets on sports is one such. They
are viewed as structured problems and solved using Hidden
Markov models for representing the hidden states of it [4]. To
rule out disambiguities in the word sense revolving around
lexical knowledge and the outcome of top-syntactic learning a
wordNet-based algorithm for word sense disambiguation was
designed. Instead of relying on large chunks of manually
acquired knowledge or surveyed information from large
organisations, using syntactic information in the WordNet is
much wiser [6].
Phrase Reinforcement (PR) algorithm to enhance the
summarization of tweets is done to handle chunks of streaming
data [7]. Tracking of social streams and detecting events are
channelized through an algorithm for a clear module for a
perfect maintenance of glossary [8]. An automatic
summarization of Twitter topics through survey level research
by a collection of streams by statistical analysic of collection of
streams is done for a precise estimation [9]. Examinations were
done on to check if the comments of a web document by the
users could be used to enhance the precision of the web
document summarization system which is channeled as novel
based tweet summarization [10]. Multiple tweets in real-time
using popular real-time tweets could be done for an all round
monitoring of channels in tweets [2]. An algorithm for multiple
post summaries is designed to handle rough lengthy words of
micro blogs [3]. Automatic tweet summarization by a novel
speech act summarization approach is also done [5].
II. FLOW DIAGRAM
The flow diagram representation of the real-time
classification of tweets architecture is shown in Fig 1.
A. Input tweets
The input tweets are streaming and ceaseless. In order to
process them with a precise technique excluding the outliers
they are put into the clusters that either group them and merge
into their family or allow them form the clusters of their own
under distinct cases.
B. Tweet Stream Clustering
The ceaseless running in of streams are put in this block of
the system. The processing of dynamic streams is not an easy
task. Therefore as and when they come in a pre-processing is
performed on them by k-means clustering. A cluster is formed
with a centroid of approximate matching of key words. When
any new tweet or a stream of tweets do not fit with the existing
clusters centroids, they tend to form a new cluster. This goes
on and on from time to time.
C. High Level Summarization
The clustered tweets are actively scanned through for the
frequency of every distinct word discarding the verbal words.
The highest frequency of words is thus obtained. Followed by
that, the words following the hashtags are counted and the
centrality of word cluster is calculated here for putting them
into high level summary.
INPUT TWEETS
TWEET STREAM CLUSTERING
HIGH LEVEL SUMMARY
TIMELINE GENERATION
Fig. 1. Flow diagram of Hybrid Architecture
D. Timeline Generation
The approximately calculated centralities are put forward
into this block for timeline generation. It involves all the active
tasks of summing up the results and producing a functional
output timeline that shows the product of the process.
III. ARCHITECTURE OF THE SUMMARIZATION SYSTEM
The block diagram representation of the model of
Summarization is shown Fig 2. . It helps in knowing that the
system and its inter-related courses have all added up to it
working in an effective and comprehensive manner. All the
sub-blocks of the system put together give the overall
architecture of the system. When the sub-blocks have to be
neatly integrated problems also arise in it. Tweet clusters are
managed by a centralized controller. The controller will
otherwise not be able to neatly under draw what was told in an
effective manner.
A. Enhanced Sumblr model
The new working models are sketched in a way that
it doesnt resemble Sumblr to it in anyway. The uninformed
blocks of users are also putting up leaps and bounds onto it.
The major drawbacks may be a cause of concern in this case.
They would not enjoy any support to it in the entertained
areas. Thereby the re-identified works have to be clearly
envisaged. The one and only way to putting up of resources
will not just modify anything to it. The remembered parts of it
will just not leave the system in a stranded way. It would
further encourage and put forth a clearly visible part to it. The
meaningful ideas are all got and repeated. The rejuvenated and
made blocks are tested for integration and neatly sketched.
When sketched and tested from all parts of it, it is just not
enough or known to be understandable. The repeatedly
checked and verified parts of it are known for testing. This
will further just put an off to the controlled pattern. The re-did
works should all be tested and exemplified. The important
parts of it becomes highly narrative and henceforth they are
readily available.
B. CluStream
CluStream is one of the most classic stream clustering
methods. It consists of an online micro-clustering component
and an offline macro-clustering component. A variety of
services on the Web such as news filtering, text crawling, and
topic detecting etc. have posed requirements for text stream
clustering.
It is to generate duration-based clustering results for text
and categorical data streams. However, this algorithm relies
on an online phase to generate a large number of micro-
clusters and an offline phase to re-cluster them. In contrast,
our tweet stream clustering algorithm is an online procedure
without extra offline clustering and in the context of
tweet summarization, we adapt the online clustering phase by
incorporating the new structure TCV, and restricting the
number of clusters to guarantee efficiency and the quality of
TCV. At the start of the stream, we collect a small number of
tweets and use a k-means clustering algorithm to create the
initial clusters. The corresponding TCVs are initialized. Next,
the stream clustering process starts to incrementally update the
TCVs whenever a new tweet arrives.
Input Tweet Stream Clustering
tweet
stream Tweet Pyramidal
Cluster Time Frame
Vector
High Level Summarization
Online Historical
Summaries Summaries
Timeline Generation
Topic Evolution Detection
Fig 2. Summarization system of the overall tweets
IV. METHODOLOGY
1) Tweet Generation
The tweets posted are not retrieved and posted as such. They
are put into filters and drained finally. This will help to
improve the readability of the tweets. Some blocked users
cannot enjoy the liberty of tweeting or re-tweeting on a
follower. The tweet generation module is the fundamental or
the primary step in the developed systems post. This will
have to be viewed as separate entities. Otherwise, the
developed system will not be suitable for all such cases. The
basic normalized four steps are rightly given and evaluated to
repeat the systems flow.
. very accurate. This was because most of the tweets follow a
2) Drill Down
Tweet cluster vector is an important part of this
algorithm. It clusters the tweets accordingly and posts onto.
The clustering algorithm like K-means are made use of. The
duration is asked to be chosen by the user and various levels
are also chosen. The user who wants to know a brief summary
of tweets on what had happened so far will definitely bring
down the ideal and perfect way to understand it all. This is
purely a handling of streaming data which poses a threat to
loss always. Therefore, it has to be neatly and straightly put
forward for a good work. Thereby, any user of the system
would have an insight to what has been so far. This would
definitely bring a rough idea to what we have done so far. The
various schematic representations are to be introduced in this.
Therefore, the high level summarization which is obtained in
it are clearly put forward. The durations selected can be
customized pertaining to the needs of the user. Therefore, the
needed idea hence forth has to be clearly and preferably
understood in an elaborate manner. This will not cause any
change to the underlying system. The system which we have
designed has contributed a lot to the clear vision of it.
Thereby, the major rest of the ideas are not going to be too far
slow. The majority of the solutions that further encourage are
kindly given and greeted with exploitation. It has to be a part
under various techniques of resource scattered. The major
pitfalls and drawbacks has to be further classified into clean
and enormous ones. Therefore, the clean idea behind will have
to be followed and reproduced at various levels of
encouragement. The clarity has to be checked upward and
downward for a clear and rough sketch.
3) Roll Up
Rolling up of tweets for high level summarization is did. It
need not necessarily be between any two days, minutes or
seconds. It could be therefore put forward in a comprehensive
manner. The algorithm which is made use of in this system is
the TCV rank summarization which was discussed about in
the earlier chapters. Tweet cluster vector is an important part
of this algorithm. It clusters the tweets accordingly and posts
onto. The clustering algorithm like K-means are made use
of.This will not cause any change to the underlying system.
The system which we have designed has contributed a lot to
the clear vision of it. Thereby, the major rest of the ideas are
not going to be too far slow. The majority of the solutions that
further encourage are kindly given and greeted with
exploitation. It has to be a part under various techniques of
resource scattered. The major pitfalls and drawbacks has to be
further classified into clean and enormous ones. Therefore, the
clean idea behind will have to be followed and reproduced at
various levels of encouragement. The clarity has to be checked
upward and downward for a clear and rough sketch.
V. EXPERIMENTAL RESULTS
The results are the summary from a streaming surge of
twitter data. The drilling down and rolling up of tweets were
bull run or otherwise a bear run process. It therefore is true
from it that any peak state has to fall down irrespective of the
degree of peak it attained. The upcoming results of it might
not prove accurate in terms of summarization. Summarization
here was done through a passing in and passing out of an array
of texts that are followed by a sequential hashtag cult. This
helped in identifying the unique results to it. In summary
Twitter diverges from well-known traits of social networks: its
distribution of followers is not power-law, the degree of
separation is shorter than expected, and most links are not
reciprocated. But if we look at reciprocated relationships, then
they exhibit some level of homophily.
[5] Sudhanshu Shiwarkar, Indraneel Deshmukh Nikhil Patil and Akash
Thanke, "Automatic twitter summarization", Volume 5 Issue 10, International
Journal of Advanced Research in Computer Science and Software
Engineering, pages 60-64 ,October 2015.
[6] Xiaobin Li, Stan Szkowicz and Stan Matwin, A WordNet-based
Algorithm for Word Sense Disambiguation , Proceedings of the 14th
International Joint Conference on Artificial Intelligence, pages 75-82, October
1999.

[7] Joel Judd and JugalKalita, "Better Twitter summaries", 15th International
joint Conference on Social media mining, pages 445-449,March 2015.

[8] Hassan Sayyadi, Matthew Hurst and Alexey Maykov, Event detection
and tracking in social streams, Proceedings of the Third International
ICWSM Conference, pages 311-314,2009.

[9] Beaux Sharifi,Mark Anthony Hutton and JugalKalita, "Summarizing

microblogs automatically" ,The 2010 Annual Conference of the North
American Chapter of the ACL, pages 685688, Los Angeles, California, June
2010.
[10] EviYulianti, SharinHaspi and Mark Sanderson, "Tweet biased
summarization" ,Journal for the association of Information Science and
Technology, pages 114-119 , 6 December 2014.

Fig 3. Output of the system

VI. CONCLUSION
Having pleased with clustering and summarizing of tweets
at the preliminary level, this project faces some crests and
troughs through the way, which has skewed a bit from the
initial set path. Initially, the attempt was to cluster the hash
tags with string similarity leaving the semantic similarity for
future. But such a clustering yielded results which were not
meaningful. This therefore foresees that there are no hash tag
usagesthey are in use with evidently a variety of different
semantic usage but not structurally. There were some
misconceptions like predictability of any user's future tweets
based on the past hash tags. Rather, the realization was that a
variety of users chose to tweets with a relatively short group of
hash tags that left us with the idea that prediction is less
meaningful. Such hindrances directed into the current project
plan which has left a clear mark for the future work to start on
from this.

REFERENCES
[1] WeiXu, Ralph Grishman ,Adam Meyers and Allan Ritter, A Preliminary
Study of Tweet Summarization using Information Extraction,Proceedings of
the Workshop on Language in Social Media (LASM 2013), pages 2029, June
13 2013
[2] Mohammed Asif Hossain Khan,DanushkaBollegala, Guangwen Liu and
Kaoru Sezaki , Multi-tweet Summarization of Real-Time Events,IEEE
Conference on Social Computing , pages 128-133,8-14 September 2013.
[3] David Inouye and JugalK.Kalita,"Comparing Twitter Summarization
Algorithms for Multiple Post Summaries" , Privacy, Security, Risk and Trust
(PASSAT) and 2011 IEEE Third Inernational Conference on Social
Computing (SocialCom), pages 298-306, 9-11 October 2011..
[4] DeepanChakrabarti and KunalPunera ,"Event summarization using
tweets", Proceedings of the Fifth International AAAI Conference on Weblogs
and Social Media , pages 66-73, 2011