REAL TIME UI WITH

APACHE KAFKA
STREAMING ANALYTICS
OF FAST DATA AND
SERVER PUSH

Lucas Jellema (CTO AMIS & Oracle Developer Champion)

12th May 2017, Oracle Friday Cloud Update, Utrecht, The Netherlands
 F5
F5 Application
F5
CTRLF5
Server
CTRLF5
CTRLF5
 F5
F5 Application
F5
CTRLF5
Server
CTRLF5
CTRLF5
 F5
F5 Application
F5
CTRLF5
Server
CTRLF5
CTRLF5
FAST DATA AND ACTIVE UI

• Handle influx
• Publish findings instantaneously
• Update UI & notify end user immediately
• Analyze in real time

• Decoupled components
• No data loss when a component is temporarily down
• Scalable with volume of events and of number of clients
THE CASE AT HAND

Tweets on #oow17
#javaone #oraclecode

Show live
tweet feed forAllow users to
conferences like tweets –
and show live Show live
Client list of liked tweet
tweets aggregates
Client per
conference Show a live
list of top 3
Client liked tweets
Client per
conference
THE CODE
https://github.com/lucasjellema/real-time-ui-with-kafka-streams
THE DEO
http://tinyurl.com/kafkaui
EVENTS Consumers

Producers

Robust, Scalable,
History Retention
Containerized/Cloud-
enabled
REQUIREMENTS FOR EVENT CAPABILITY

• Provide decoupling between publisher and consumer

• Generally accessible for all consumers
• Using standardized protocols and formats for communications and event payload
(http, JSON)
• Scalable (handle high loads)
• Available (allow speedy event publication)
• Reliable (do not lose events, at least once delivery)
• Event Ordering (deliver events in the order of publication)
• Manageable at scale
• Retain Event History
• For consumers that are late to the game
• To construct state from all historic events: Event Sourcing
INTRODUCING APACHE KAFKA

• ..- 2010 – creation at Linkedin

• Message Bus | Event Broker
• High volume, low latency, highly reliable, cross technology
• Scalable, distributed, strict message ordering, ….
• 2011/2012 – open source under the Apache Incubator/ Top Project
• Kafka is used by many large corporations:
• Walmart, Cisco, Netflix, PayPal, LinkedIn, eBay, Spotify, Uber, Sift Science
• And embraced by many software vendors & cloud providers
• Client libraries available for NodeJS, Java, C++, Python, Ruby, PHP
and many more
KAFKA TERMINOLOGY
Message

• Topic Key
Value
• partition Time

• Message Producer Consumer

• == ByteArray
Broker
• Broker Topic
• replicated
• Producer
• Consumer
• Working together
in Consumer Groups
 Consumers
tcp
Producers
tcp
Broker
Topic
THE CASE AT HAND – STEP ONE

Tweets on #oow17
#javaone #oraclecode

Tweets
Topic

Show live
tweet feed for
conferences
Client

Client

Client
Client
 THE CASE AT HAND – STEP ONE AND A HALF

Tweets on #oow17
#javaone
#oraclecode

Tweets
Topic

Show live
tweet feed for
conferences
Client

Client

Client
Client
CONFIGURE IFTTT RECIPE
CONFIGURE IFTTT RECIPE
CONFIGURE IFTTT RECIPE
HIGH CODE APPROACH
HIGH CODE (2) – PRODUCE MESSAGE TO KAFKA
THE CASE AT HAND – STEP ONE AND TWO

Tweets on #oow17
#javaone #oraclecode

Tweets
Topic

Show live
tweet feed for
conferences
Client

Client

Client
Client
KAFKA CONSUMER IN NODE
GET EVENTS PUSHED INTO APPLICATION
THE CASE AT HAND
SERVER SENT EVENTS FOR PUSH BACK
Tweets on #oow17
#javaone #oraclecode

Tweets
Topic

Show live
tweet feed for
conferences
Client

Client
Server Sent
Event
Client
Client
SERVER SENT EVENT – SERVER SIDE
SERVER SENT EVENT – CLIENT SIDE
LIVE TWEET STREAM

Server Sent
Event
THE CASE AT HAND
TWEET LIKES – CLIENT TO SERVER TO ALL CLIENTS
Tweets on #oow17
#javaone #oraclecode

Tweets
Topic
Show Allow
live users to
tweet feed fortweets –
like
conferences
and show live
list of liked
Client
tweets

Client

Client SSE
Client
THE CASE AT HAND
WEB SOCKETS – FOR BI DIRECTIONAL PUSH
Tweets on #oow17
#javaone #oraclecode

Tweets
Topic
Show Allow
live users to
tweet feed fortweets –
like
conferences
and show live
list of liked
Client
tweets

Client WebSockets

Client SSE
Client
WEBSOCKETS – SERVER SIDE
WEBSOCKETS – CLIENT SIDE
TWEET LIKES BROADCASTING

WebSockets

WebSockets
THE CASE AT HAND
STREAMING ANALYSIS OF TWEET EVENTS
Tweets on #oow17
#javaone #oraclecode

Tweets
Allow
Show users to
live Topic
tweetlike
feed tweets – live
forShow
and show live
conferences tweet
aggregates
list of liked
tweets per
Client conference

Client WebSockets

Client SSE
Client
THE CASE AT HAND - STREAMING ANALYSIS OF TWEETS

Tweets on #oow17
#javaone #oraclecode

Tweets tweetAnalytics
Allow
Show users to
live Topic Topic
tweetlike
feed tweets – live
forShow
and show live
conferences tweet
aggregates
list of liked Streamingµ
tweets per Tweets
Client conference Aggregation

Client WebSockets

Client SSE
Client
KAFKA STREAMS
• Real Time Event [Stream] Processing integrated into Kafka
• Aggregations & Top-N
• Time Windows
• Continuous Queries
• Latest State (event sourcing)
• Turn Stream (of changes) into Table
(of most recent or current state)
• Part of the state can be quite old
• A Kafka Streams client will have state
in memory
• Always to be recreated from topic partition
log files
• Note: Kafka Streams is relatively new
• Only support for Java clients
KAFKA STREAMS
Topic
Filter

Aggregate

Topic
Join

Map (Xform)

Topic
Publish
 EXAMPLE OF KAFKA STREAMS
Topic Set Conference as key
groupBy

Sum/Avg/Top3 by key
TweetMessage (==conference)
Conference Aggregate
Text
and possibly per time window
Author
Hastag Topic
Latest Conference
Join
Details

Round aggregate to
nearest 100 Map (Xform)

As JSON
Publish
Topic: CountTweetsPerConference
KAFKA STREAMS –
RUNNING COUNT TWEETS PER CONFERENCE
STREAMING TWEET ANALYTICS
PUSHED TO CLIENTS

Server Sent
Event
THE CASE AT HAND - STREAMING ANALYSIS
OF TWEET LIKES
Tweets on #oow17
#javaone #oraclecode

Allow
Show users to
live Tweets tweetAnalytics
tweetlike
feed – live
forShow
tweets Topic Topic
and show live
conferences tweet
Show a live
aggregates
list of liked list of top 3
tweets per liked tweets Streamingµ
conferenceper Tweets
Client Aggregation
conference

Client WebSockets

Client SSE
Client
THE CASE AT HAND - STREAMING ANALYSIS
OF TWEET LIKES
Tweets on #oow17
#javaone #oraclecode tweetLike
Topic

Allow
Show users to
live Tweets tweetAnalytics Top3TweetLikes
Topic Topic
tweetlike
feed tweets – live
forShow PerConference
and show live
conferences tweet
Show a live
aggregates
list of liked list of top 3
tweets per liked tweets Streamingµ
conferenceper Tweets
Client Aggregation
conference

Client Likes µ
WebSockets Aggregation

Client SSE
Client
WEBSOCKETS – SERVER SIDE
RUNNING TOP 3 OF
BEST LIKED TWEETS PER CONFERENCE

Server Sent
Event
END TO END FLOW CLOUD ENABLED
Cache
API
EventHub CS
µ Tweets Likes
Tweet Likes
API
Count Top3
µ

UI µ
Client
Chrome
Tweets µ Likes µ
Client Aggregation Aggregation
Firefox
CONCLUSION

• Fast data – Active UI

• Decoupled processing
• And distributed across clouds and on premises
• Kafka
• Events & Data Store
• Streaming Analysis
• Modern browser – push capability
• SSE, WebSocket, HTTP/2, WebWorker Notifications
• Active UI with fresh data without burden on application server and
back end system
• “Step away from that F5 key”
• Blog: technology.amis.nl

• Email: lucas.jellema@amis.nl

• : lucasjellema

• : lucas-jellema

• : www.amis.nl, info@amis.nl
+31 306016000
Edisonbaan 15,
Nieuwegein