01/06/2016

IntroductiontoLogstash

IntroductiontoLogstash
TableofContents
WhatisLogstash?
Downloading/InstallingLogstash
RunningLogstash
Agent
Web
Logstasharchitecture
Simplestconfiguration
Changingthewaydataisrepresented
Readinginputfromfilesondisk
OutputingtoanembeddedElasticsearch
OutputingtoaseparateElasticsearch
Addingafilterintothemix
The"grok"filter
Combiningeverythingtogether
TheUIforLogstash

WhatisLogstash?
Agenericconceptforreceivingdata,transformingit,andoutputtingit.

Downloading/InstallingLogstash
Downloadingisaseasyasgettingitfromhttp://logstash.net,butI'llbeusingversion1.4(whichiscurrentlybetaas
ofthiswriting)
Toinstall,untarthepackagesomewhere,orusethe.deb/.rpmrepositoriesforyourrespectiveoperatingsystem.

RunningLogstash
Thereare2mainmodesofrunningLogstash.Notethatbothcanberunatonce.

Agent
Runningasanagentcollectsinformation,forwardingittothebackend(inourcase,Elasticsearch)

Web
RunsthewebUI(knownasKibana)bundledinLogstash

Logstasharchitecture
Logstashisacollectionof:
Inputs
Codecs
Filters
Outputs

Simplestconfiguration
Startingwiththesimplestinput,standardin:
input{
stdin{}
}

Andthesimplestoutput,standardout(nofiltersfornow):
output{
stdout{}
}

Torunthis,youcando:
bin/logstashagentflogstashsimple.conf

Changingthewaydataisrepresented
Let'schangethecodec(datarepresentation)toprintmoreinformation:
input{
stdin{}
}

http://writequit.org/articles/logstashintro.html

1/4

01/06/2016

IntroductiontoLogstash
output{
stdout{
codec=>rubydebug
}
}

Readinginputfromfilesondisk
Thistime,insteadofreadinginfromstdin,readfromafile:
input{
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
}

OutputingtoanembeddedElasticsearch
Logstashcanoutputtomanymoreplacesthanjuststdout,itcomeswithelasticsearchasanoutputoptionthatcanrun
embedded:
input{
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
elasticsearch{
embedded=>true
}
}

Addafewlogstothefile:
echo"thisisalogmessageaboutfoo">>example.log
echo"thisisalogmessageaboutbar">>example.log
echo"thisisalogmessageaboutbaz">>example.log

Logstashcreatesanindex,noticethatitcreateditforthedaythiswasrun.Logstashwillcreatedailyindicesby
default:
curl'localhost:9200/_cat/health?v'
echo""
curl'localhost:9200/_cat/shards?v'

epochtimestampclusterstatusnode.totalnode.datashardsprireloinitunassign
139504637202:52:52elasticsearchyellow2155005

indexshardprirepstatedocsstoreipnode
logstash2014.03.172pSTARTED099b172.22.255.231MultipleMan
logstash2014.03.172rUNASSIGNED
logstash2014.03.170pSTARTED099b172.22.255.231MultipleMan
logstash2014.03.170rUNASSIGNED
logstash2014.03.173pSTARTED24.2kb172.22.255.231MultipleMan
logstash2014.03.173rUNASSIGNED
logstash2014.03.171pSTARTED13.9kb172.22.255.231MultipleMan
logstash2014.03.171rUNASSIGNED
logstash2014.03.174pSTARTED099b172.22.255.231MultipleMan
logstash2014.03.174rUNASSIGNED

Andyoucansearchforlogmessages(here'sanexamplequery)
{
"query":{
"simple_query_string":{
"query":"foo|bar",
"fields":["message"]
}
},
"size":3
}

Andgetbacktheresults:
HTTP/1.1200OK
ContentType:application/json;charset=UTF8
ContentLength:1248

http://writequit.org/articles/logstashintro.html

2/4

01/06/2016

IntroductiontoLogstash

{
"took":76,
"timed_out":false,
"_shards":{
"total":20,
"successful":20,
"failed":0
},
"hits":{
"total":4,
"max_score":0.35355338,
"hits":[{
"_index":"logstash2014.03.17",
"_type":"apache",
"_id":"q8EqCk2RjWwB70rxz7bw",
"_score":0.35355338,"_source":{"message":"thisisalogmessageaboutfoo","@version":"1","@timestamp":"20140317T08:52:
},{
"_index":"logstash2014.03.17",
"_type":"apache",
"_id":"e0KXf2eCQjmm302UB6n60g",
"_score":0.35355338,"_source":{"message":"thisisalogmessageaboutbar","@version":"1","@timestamp":"20140317T08:52:
},{
"_index":"logstash2014.03.13",
"_type":"apache",
"_id":"DXwFHMvTTsauxjr9lJ5Xcg",
"_score":0.25427115,"_source":{"message":"thisisalogmessageaboutbar","@version":"1","@timestamp":"20140313T08:45:
}]
}
}

OutputingtoaseparateElasticsearch
Embeddedisgreatfordevelopment,butoutputtingtoadifferentElasticsearchserverisbetterforproduction:
input{
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
elasticsearch{
host=>"localhost"
port=>9300
node_name=>"logstashagent007"
workers=>2
}
}

Addingafilterintothemix
Filtersallowyoutomodifyoutput
Themostusefulisgrok,butlet'sstartwithmutate.Sothestandardinput/outputconfigurationfirst:
input{
stdin{}
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
}

Addingthemutatefiltertoaddafieldaswellaslowercasethe"message"field
filter{
mutate{
add_field=>["myhost","Hellofrom%{host}!"]
lowercase=>["message"]
}
}

The"grok"filter
Logstash'sarguablymostusefulfilter.~120differentpatternsthatcanbecomibned.
https://github.com/elasticsearch/logstash/tree/1.4.x/patterns
https://grokdebug.herokuapp.com/
Again,standardboilerplate:
input{

http://writequit.org/articles/logstashintro.html

3/4

01/06/2016

IntroductiontoLogstash
stdin{}
file{
type=>"apache"
path=>"/Users/hinmanm/introtologstash/example.log"
}
}
output{
stdout{
codec=>rubydebug
}
}

Andthenagrokfiltermeanttomatchthetext"name:John"intheinputs:
filter{
grok{
match=>["message","name:%{WORD:custom_name}"]
}
mutate{
lowercase=>["custom_name"]
}
}

Combiningeverythingtogether
Readfromafile(thistimeanElasticsearchlogfile),usetheeslogtypewhenputtingthelogmessageinto
Elasticsearch.OutputwillbewrittentoaseparateElasticsearchclusteratlocalhostonport9300:
input{
file{
type=>"eslog"
path=>"/Users/hinmanm/introtologstash/es/logs/elasticsearch.log"
}
}
output{
stdout{
codec=>rubydebug
}
elasticsearch{
host=>"localhost"
port=>9300
}
}

ThisexamplefilterwillmatchElasticsearch'slogformat,extracttheusefulpiecesofthelog(time,level,package,
node_name,andlogmessage).
Themutatefilterwillthen:
lowercasetheloglevel(INFO=>info)
stripthewhitespaceforthepackage("indices.recovery"=>"indices.recovery")
Additionally,themultilinefilterwillmatchlinesthatlooklikeaJavaException,andcollapsethemintoasingle
messagefromthepreviousline.
filter{
grok{
match=>["message",
"^\[%{TIMESTAMP_ISO8601:time}\]\[%{LOGLEVEL:level}.*\]\[%{DATA:package}\]\[%{DATA:node_name}\]%{DATA:logmsg}$"
}

mutate{
lowercase=>["level"]
strip=>["package"]
}

multiline{
pattern=>"(org\.elasticsearch\.Exception.+|(at.+))"
what=>"previous"
}
}

TheUIforLogstash
LogstashbundlesKibana,whichcanbeusedforvisualizingdata,andisaseasyasrunning:
bin/logstashweb

orbothatoncewith:
bin/logstashagentflogstash.confweb

Author:LeeHinman
Created:20140319Wed01:07
Emacs24.3.50.1(Orgmode8.2.5h)
Validate

http://writequit.org/articles/logstashintro.html

4/4