Académique Documents
Professionnel Documents
Culture Documents
Mikio L. Braun mikiobraun twimpact UG (haftungsbeschrnkt) with Matthias Jugel thinkberg Apache Hadoop Get Together, Berlin April 28, 2012 http://blog.mikiobraun.de http://twimpact.com
2012 TWIMPACT
Trend analysis (trending topics) Sentiment analysis Impact analysis (Klout, Kred, etc.) More general studies (diameter of network, distribution patterns, etc.) Event treams (Twitter stream) Graph data (user relationships, retweet networks) Text data (sentiment analysis, word clouds) URLs
Types of data
2012 TWIMPACT
Examples
Twitter firehose/sprinkler Click-through data bit.ly URL resolution requests up to a few thousand events per second events are small up to a few kilobytes
Some numbers:
2012 TWIMPACT
What's in a Tweet?
Tweet Hashtag Link User Mention Keywords Retweeting User Retweeted User Retweeted Tweet
Timestamp
2012 TWIMPACT
Trending by retweet activity Robust matching of tweets even if shortened, edited (slightly) Compute trends for links, hashtags, URLs Aggregate TWIMPACT score for users
2012 TWIMPACT
2012 TWIMPACT
History of approaches
Version 3
2012 TWIMPACT
Hardly real-time. Also, data bases will become slower and slower...
2012 TWIMPACT
NoSQL: Cassandra
Structure: Families Tables Rows Key Value pairs Easy clustering (peer-to-peer configuration) Flexible consistency, read-repair, hinted handoff, etc. No locking, (in 0.6.x:) no support for indices, counters complete rewrite Operations profile (about 50:50 read/write)
2012 TWIMPACT
Cassandra: Multithreading
Seconds
Apache Hadoop Get Together, April 18, 2012, Berlin
Cassandra: Configuration
Flush
Memtables, indexes, etc.
Compaction
2012 TWIMPACT
Cassandra: Configuration
Compaction
Big GC
2012 TWIMPACT
NoSQL/Cassandra - Summary
Works quite well, faster than PostgreSQL (from 200 to 600 tps) Lack of locking/indices require a lot of manual management Configuration messy 4 node cluster vs. single node:
Single node consistently 1.5 3 times faster!
Twitter's (backtype's) Storm https://github.com/nathanmarz/storm Yahoo's S4 http://incubator.apache.org/s4/ Esper http://esper.codehaus.org/ Streambase http://www.streambase.com
2012 TWIMPACT
Maximum throughput hard to estimate Not everything can be parallelized Scalable storage system still necessary How to deal with failure/congestion? Persistent messaging middleware not what you might want.
2012 TWIMPACT
http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html
Parse
Augment Content
Custom Filters
Delivery
936 CPU cores Analyzes 250 million tweets per day Peak throughput: 120,000 t/s monitoring & accounting
C++, PHP, Java/Scala, Ruby MySQL on SSDs, HBase (30 nodes, 400TB), memcached, Redis for some queues 0MQ, Kafka (LinkedIn)
Keep resource needs constant Control maximum processing rates Disks too slow, keep data in RAM
2012 TWIMPACT
Stream mining
fixed number of slots asd erq qwe 13r13t erqew erq qer fgsa gwth 5z3 wet 42 37 25 20 17 13 10 7 4 qer 5 21
Focus on relevant data, discard the rest Provably approximates true counts Keep data in memory
Space Saving algorithm (Metwally, Agrawal, Abbadi, Efficient Computation of Frequent and Top-k Elements in Data Streams, International Conference on Database Theory, 2005.)
2012 TWIMPACT
TWIMPACT
Stream mining to keep hot set of few hundred thousand most active retweets in memory Secondary indices, bipartite graphs, object stores Write snapshots to disk for later analysis Up to several thousand tweets per second in single threaded operation.
2012 TWIMPACT
2011 in Retweets
2012 TWIMPACT
2011 in Retweets
2012 TWIMPACT
Snapshots
Trends
2012 TWIMPACT
2012 TWIMPACT
2012 TWIMPACT
2012 TWIMPACT
Summary
Many interesting challenges in social media. Many different data types, including streams. MapReduce doesn't really fit stream processing You can't just scale into real-time Principles of Stream Processing
Bounded hot set of data in memory Mine stream, discard irrelevant data
Real world applications often include a mixture of multithreading, stream processing, map reduce and single thread stages.
2012 TWIMPACT