HackingTomcat

SecretsRevealed

TalkSponsored
By

Actual
Sponsor

WhoamIblabla

fhanik@apache.org
TomcatCommitter/ASFmember
CodesignedtheCometimplementation
ImplementedNIOconnectorin6
Responsibleforsessionreplicationand
clustering
BeeninvolvedwithASFsince2001

WhoareWe

TopLevelProjecttomcat.apache.org
24committersonfile
Active/coregroupismuchsmaller
Findusondev@tomcat.apache.org
Workonwhatinterestsyouandwork
withoutguidance
Ourpeopleskillsareimproving;)

WelcometoTomcat

Whatwewillcover
HistoryofTomcat
TheBasics
ConfigurationandContainerArchitecture

TheAdvanced

SwappableComponents
TomcatConnectors
ServletContainer
JSPCompiler
Developing/DebuggingTomcat

WhatWeWillNotCover

TooBasicstuffThisisatechnicalpresentation
Configurationdetails
HowtheactualJSP.jspto.javacompilerworks
ForkedTomcatcodebases,howtheydifferand
whytheyhappened
OlderversionsofTomcat,wewillworkwith
Tomcat6,nolookingback

HistoryofTomcat
StartedoutasareferenceimplemenationbySun
Microsystem
DonatedtoASFTomcat3(roughly1999)
Tomcat4Newspecs&Firstrewrite
CodenameCatalina
Tomcat5.0Newspecs
Tomcat5.52ndRewritePerformance
Tomcat6.0Newspecs,NewCoolFeatures

Basics
server.xml

Mainconfigurationfile
Buildsserveronthefly
Parsedusingcommonsdigester
Tomcathashardcodedrulesetsfortheparsing
Everyelement/componentisswappable

Basics

<ElementName
className=the implementation
attribute=call setAttribute/>
Example:
<Server
className=o.a.c.core.StandardServer
port="8005
shutdown="SHUTDOWN">

Basics
Entireserver.xmlparsedbasedonrules
Lookfortheserules:
Catalina.java
org/apache/catalina/startup/

Evenweb.xmlisparsedusingthedigester

Basics
Catalina.javacreateStartDigester
Digester digester = new Digester();
digester.setValidating(false);
digester.setClassLoader(
StandardServer.class.getClassLoader());
digester.addObjectCreate("Server",
"org.apache.catalina.core.StandardServer,
"className");
digester.addSetProperties("Server");

Basics
Theexception<Connector>
<Connector
className=ignored
protocol=nested object
ConnectorCreateRule.javabegin
digester.push(
new Connector(
attributes.getValue("protocol")));
protocol>nestedclassName

Basics

Tomcat The Server

Services

Engine (Catalina)
Hosts
AJP Connector

Realm

Valves

Context

8009
SSL Connector

Valves

JSPs

8443
HTTP Connector
8080

Valves

Servlets

Basics
Service/Engine/Host/Context
AllareContainers
AllimplementLifecycleListeners

LifecycleEvents
Howobjectsgetinitialized,startedandstopped

Objectrelationshipsareestablishedduring
creation(digesterparsing)

Basics
LastBasicsIpromise
conf/web.xml

Defaultweb.xml
MergedwithwebappsWEBINF/web.xml
DefaultServletstaticcontent
JSPServletJSPfiles

conf/context.xml
Mergedwithapps<Context>definition

Advanced

Connectorstheentrypoint
ServletEngineandContainerDesign
JasperTheJSPengine
Valvesinterceptorpattern
DevelopingandDebugging
Howtojoinifyouareinterested

PerformanceTip
TomcatproducesverylittleGC
Mostobjectsarepooled
EventhoughmakersofVMsay,neverpool
objects
PreventsCPU/GCjigsawpattern
ResettingfieldsisfasterthanGColdobject,
createnewobjectandinitialize
NoGClockupsurprises

Connectors
HTTPConnectorprotocol=

o.a.coyote.http11.Http11Protocol
o.a.coyote.http11.Http11AprProtocol
o.a.coyote.http11.Http11NioProtocol
HTTP/1.1aliasedtoHttp11andHttp11Apr

AJPConnector
org.apache.jk.server.JkCoyoteHandler
org.apache.coyote.ajp.AjpAprProtocol
AJP/1.3aliasedtothetwoabove

Connectors
Connector

Protocol
<Handler>

Processor
Adapter

EndPoint
Engine
There are some pretty ugly interdependencies here.
While re-factoring would resolve that, time has been
spent improving performance.

Connectors
Connector

RequestProcess
Protocol

Processor
Adapter

EndPoint
Engine
Once the request
Processor
is1.
parsed
New
sets
HTTP
up input/output
Request buffers
The CoyoteAdapter
HTTP Parsing
(bridge between
logic
is logic
inconnector
here
and
engine)
All java.io/java.nio/apr
socket
is in the
EndPoint
Passes the request
Parsestorequest,
the servlet
if request
engineis available

PerformanceTip
MessageBytes.java
AllHTTPParsingdoesntdealwithstrings
Everychunkofdatagetsparsedintoa
MessageBytesobject
ThisobjectrepresentsastringintheHTTP
header
Avoidstringcomparisonroutines
Doesntcontaincopyofbyte[],butapointertothe
originaldatawithanoffsetandlength

PerformanceTip
UseHttp11Protocol

KeepAliveisturnedoff
Kernelacceptfilterisinplace

UseHttp11AprProtocol

TakeadvantageofSEND_FILE
NativeSSLhandling
CometSupport

UseHttp11NioProtocol

TakeadvantageofSEND_FILE
LargenumberofsocketsusingKeepAlive
APRisnotavailableorJNIisnotpreferred
CometSupport

Advanced
CoyoteAdapter.java
CreatesRequest/Responseobjects
MapsRequest/Responseto
AHostobject(StandardHost)
AContextobject(StandardContext)
AServlet(StandardWrapper)

ParsesSessionCookie
URL
Cookie

GrabsEnginesvalveandpassestherequestintothe
servletengine

PerformanceTip
DefaultServlet.java
Handlesallstaticcontent
Getsdeployedintoeverywebappthrough
conf/web.xml
IfSEND_FILEsupportisenableditwilluseit
ItsaREGULARSERVLET!!
Filesandtheirattributesarecached
Youcanreplacewithyourownimplementation

Advanced
ServletInvokationChain
o.a.c.servlets.DefaultServlet.doGet
javax.servlet.http.HttpServlet.service
o.a.c.core.ApplicationFilterChain.doFilter
o.a.c.core.StandardWrapperValve.invoke
o.a.c.core.StandardContextValve.invoke
o.a.c.core.StandardHostValve.invoke
o.a.c.valves.ErrorReportValve.invoke
o.a.c.core.StandardEngineValve.invoke
o.a.c.connector.CoyoteAdapter.service
o.a.coyote.http11.Http11NioProcessor.process
o.a.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.proces
s
o.a.tomcat.util.net.NioEndpoint$SocketProcessor.run
java.util.concurrent.ThreadPoolExecutor$Worker.runTask
8.
StandardHostValve
11.
3.
6.
7.
9.
SocketProcessor
StandardEngineValve
ErrorReportValve
StandardContextValve
ApplicationFilterChain
the thread
1.
10.
12.
4.
2.Everything
5.
HttpProcessor
NIO
CoyoteAdapter
The
StandardWrapperValve
Connector
Servlet
starts
defaults
with
java.util.concurrent.ThreadPoolExecutor$Worker.run
Sets
context
class
Represents
Simple
First
Catches
Invokes
Runnable
(spec)
Throwable
Servlet
theservlet
engine
toloader
(spec)
invoke
Invokes
Execution
parses
to valve
Creates
ThreadPoolExecutor
(spec)
HTTP
ofin
Request
the
request
java.lang.Thread.run

FilterChain,
Handler.process
container
Reports
ServletRequestListeners
400+
invokes
errorsservlet
FilterChain
Response
pair

Advanced

Reading Data From the InputStream

o.a.c.http11.InternalNioInputBuffer$SocketInputBuffer.doRead
o.a.c.http11.filters.ChunkedInputFilter.readBytes
o.a.c.http11.http11.filters.ChunkedInputFilter.parseChunkHeader
o.a.c.http11.http11.filters.ChunkedInputFilter.doRead
o.a.c.http11.http11.InternalNioInputBuffer.doRead
o.a.c.http11.Request.doRead
o.a.catalina.connector.InputBuffer.realReadBytes
o.a.t.u.buf.ByteChunk.substract
o.a.catalina.connector.InputBuffer.read
o.a.catalina.connector.CoyoteInputStream.read
comet.CometEchoServlet.echo

PerformanceTip
JspServlet.java
HandlesallJSPfiles
Getsdeployedintoeverywebappthrough
conf/web.xml
MappingdonethroughURLpatterns(perspec)
ItsaREGULARSERVLET!!
ConnectsintotheJasperengine

Advanced
HowareJSPfileshandled

ThroughtheJspServlet(sameinvocationpath)
JspServletWrappercreated

ContainsJspCompilationContext
HoldsaJSPclassloader
Invokescompileonthecompilationcontext
Processesannotations
Loads.classfilethroughclassloader
Finallyinvokesservice(req,resp)onthegeneratedservlet

o.a.jasper.compiler.Compiler

Generates.javafilefrom.jsp

o.a.jasper.compiler.JDTCompiler
Generates.classfilefrom.java

Advanced
DeploymentofApplications
<Context> - StandardContext
<Listener> - LifecycleListener
<Loader>
- WebappLoader
<Manager> - StandardManager
<Realm>
- No default
<Resources> - FileDirContext
<Valve>
- No default
</Context>
ContextRuleSet.javaparsescontextsinserver.xml

Advanced
Deploymentofapplications
ThedeployerisHostConfig.java
EachStandardHost.javaobjectholdsareferencetoa
HostConfig
Deployorder
context.xmlfiles
WARfiles
Directories

/ROOTishardcodedforpath=
RuntimedeploymenttriggeredbyLifecycleEvent

Advanced
Developing/Debugging

SVNRepoforTC6issimplified
trunk/javaallyouneed
svnco/antdownload/antbuildsthesystem
Runinsideadebuggerbyemulatingthe
catalina.sh/catalina.batifyouwish
Everythingisjava,breakpointsanywhere

PerformanceTip
Doesitscale
Yes,itsbeentestedwith16kconcurrentandactive
connectionsonaXmx512msystem
PerformanceincreaseswellasnewCPUsareadded
RAMisyourmax#concurrentconnectionlimitation
Simpletestsrunat
http://blog.covalent.net/roller/covalent/entry/20070308

PerformanceTip
Tuning

Mostlyintheapplicationitself,Tomcatdefaultispretty
good
Whenitcomesdowntonutsandbolts,thetuningisin
theconnectors
NIOconnector,byfarthemosttuningoptions(see
docs)
SocketandAppbuffersusuallythemostimportant
aspectforwritespeed
APRconnector,speedylittledevil,notasmany
options,butreliesonAPRbelowbeingwelltuned.

PerformanceTip
Tuningtheservletengine
Sure,itcanbedone,butnotthrough
configuration
Mostcommonbottlenecksturnouttobe
synchronizedstatementsorlocks
Comparedtothewebapportheconnector,
spendingtimetuningtheengineisnotworth
thetime

Conclusion

Notsodifficultonfirstimpression
Slightlyconfusingonsecondimpression
Onceyougetahangofit,gocrazy
Modulardesign,andnastydeps
Notthetypicaltextbookjavadesign
Findsomethinginteresting?wantto
contribute?takeinitiative,dontbeshy

Wanttojoin?
Ideasforneededprojects

Betterdeployer
Documentation
Administrationtool
BetterJMXsupport
RemoteandClusterdeployments
Livestatusdashboard
SIPsupport
Thelistgoeson,takeyourpick!

Q&A
LotsandLotscovered
Onlyadropinthesea,butenoughtogetyou
started
notenoughtime

fhanik@apache.organytime
dev@tomcat.apache.orgbebrave
http://people.apache.org/~fhanikforthe
presentation