memprof

a memory profiler for ruby

Aman Gupta
@tmm1
Ruby developers
know...
 Ruby
is

fatboyke (flickr)
 Ruby loves eating RAM

37prime (flickr)
this talk is about what
ruby does with your
RAM

let’s take a look inside the VM

ruby allocates
memory from
the OS
memory is
broken up
into slots
each slot
holds one
ruby object
 when you need an object, it’s
pulled off the freelist

a linked list called the

‘freelist’ points to all the
empy slots on the ruby heap
 when you need an object, it’s
pulled off the freelist

a linked list called the

‘freelist’ points to all the
empy slots on the ruby heap
 when you need an object, it’s
pulled off the freelist

a linked list called the

‘freelist’ points to all the
empy slots on the ruby heap
 when you need an object, it’s
pulled off the freelist

a linked list called the

‘freelist’ points to all the
empy slots on the ruby heap
 when you need an object, it’s
pulled off the freelist

a linked list called the

‘freelist’ points to all the
empy slots on the ruby heap
 when you need an object, it’s
pulled off the freelist

if the freelist is empty, GC is

run

a linked list called the

‘freelist’ points to all the
empy slots on the ruby heap
 when you need an object, it’s
pulled off the freelist

if the freelist is empty, GC is

run
GC finds non-reachable
objects and adds them to
the freelist

a linked list called the

‘freelist’ points to all the
empy slots on the ruby heap
 when you need an object, it’s
pulled off the freelist

if the freelist is empty, GC is

run
GC finds non-reachable
objects and adds them to
the freelist

if the freelist is still empty

(all slots were in use)

a linked list called the

‘freelist’ points to all the
empy slots on the ruby heap
 when you need an object, it’s
pulled off the freelist

if the freelist is empty, GC is

run
GC finds non-reachable
objects and adds them to
the freelist

if the freelist is still empty

(all slots were in use)
another heap is allocated
a linked list called the
‘freelist’ points to all the all the slots on the new
empy slots on the ruby heap heap are added to the
freelist
 turns
out,
Ruby’s
GC is
also one
of the
reasons it
can be so
antphotos (flickr)
slow
 Matz’ Ruby
Interpreter
(MRI 1.8)
has a...

john_lam (flickr)
 Conservative
lifeisaprayer (flickr)
 Stop
the
World
benimoto (flickr)
 Mark
and
Sweep
michaelgoodin (flickr)
 kiksbalayon (flickr)

Garbage
Collector
• conservative: the VM
hands out raw pointers to
ruby objects
• conservative: the VM
hands out raw pointers to
ruby objects

• stop the world: no ruby

code can execute during GC
• conservative: the VM
hands out raw pointers to
ruby objects

• stop the world: no ruby

code can execute during GC

• mark and sweep: mark all

objects in use, sweep away
unmarked objects
 more objects
=
longer GC

mckaysavage (flickr)
 longer GC
=
less time to run
your ruby code
kgrocki (flickr)
 fewer objects
=
better
performance
januskohl (flickr)
 improve performance
1. remove unnecessary object allocations
object allocations are not free
 improve performance
1. remove unnecessary object allocations
object allocations are not free

2. avoid leaked references

not really memory ‘leaks’

you’re holding a reference to an object you no

longer need. GC sees the reference, so it keeps
the object around
 the GC
follows
references
recursively,
so a reference
to classA
will ‘leak’ all
these objects
 useful tools
• ObjectSpace.each_object
• gdb.rb
• bleakhouse
• heap dumping patches
• memprof
types = Hash.new(0)
ObjectSpace.each_object do |obj|
types[obj.class] += 1
end
pp types.sort_by{ |klass,num| num }

[ ...,
[Module, 18],
[Class, 158],
[String, 1725]
]
mayu (flickr) delgrossodotcom (flickr)
on Ruby 1.9,
use ObjectSpace.count_objects
 useful tools
• ObjectSpace.each_object
• gdb.rb
• bleakhouse
• heap dumping patches
• memprof
honey_and_vinegar_real_estate (flickr)
gdb.rb: gdb hooks (gdb) ruby objects
HEAPS
SLOTS
8
1686252

for REE LIVE

FREE
893327 (52.98%)
792925 (47.02%)

scope 1641 (0.18%)

regexp 2255 (0.25%)
data 3539 (0.40%)
class 3680 (0.41%)
hash 6196 (0.69%)
object 8785 (0.98%)
array 13850 (1.55%)
string 105350 (11.79%)
node 742346 (83.10%)

(gdb) ruby objects strings

140 u 'lib'
158 u '0'
294 u '\n'
619 u ''

30503 unique strings

3187435 bytes
gdb.rb: gdb hooks (gdb) ruby objects
HEAPS
SLOTS
8
1686252

for REE LIVE

FREE
893327 (52.98%)
792925 (47.02%)

scope 1641 (0.18%)

• http://github.com/tmm1/gdb.rb regexp
data
class
2255
3539
3680
(0.25%)
(0.40%)
(0.41%)
hash 6196 (0.69%)
object 8785 (0.98%)
array 13850 (1.55%)
string 105350 (11.79%)
node 742346 (83.10%)

(gdb) ruby objects strings

140 u 'lib'
158 u '0'
294 u '\n'
619 u ''

30503 unique strings

3187435 bytes
gdb.rb: gdb hooks (gdb) ruby objects
HEAPS
SLOTS
8
1686252

for REE LIVE

FREE
893327 (52.98%)
792925 (47.02%)

scope 1641 (0.18%)

• http://github.com/tmm1/gdb.rb regexp
data
2255
3539
(0.25%)
(0.40%)

•
class 3680 (0.41%)
attach to a running REE process hash 6196 (0.69%)
and inspect the heap object
array
8785
13850
(0.98%)
(1.55%)

• number of nodes by type

string
node
105350
742346
(11.79%)
(83.10%)

• number of objects by class

• number of strings by content (gdb) ruby objects strings

•
140 u 'lib'
number of arrays/hash by size 158
294
u
u
'0'
'\n'
619 u ''

30503 unique strings

3187435 bytes
gdb.rb: gdb hooks (gdb) ruby objects
HEAPS
SLOTS
8
1686252

for REE LIVE

FREE
893327 (52.98%)
792925 (47.02%)

scope 1641 (0.18%)

• http://github.com/tmm1/gdb.rb regexp
data
2255
3539
(0.25%)
(0.40%)

•
class 3680 (0.41%)
attach to a running REE process hash 6196 (0.69%)
and inspect the heap object
array
8785
13850
(0.98%)
(1.55%)

• number of nodes by type

string
node
105350
742346
(11.79%)
(83.10%)

• number of objects by class

• number of strings by content (gdb) ruby objects strings

•
140 u 'lib'
number of arrays/hash by size 158
294
u
u
'0'
'\n'

• uses gdb7 + python scripting

619 u ''

30503 unique strings

• linux only 3187435 bytes

 fixing a leak in rails_warden
(gdb) ruby objects classes
1197 MIME::Type
2657 NewRelic::MetricSpec
2719 TZInfo::TimezoneTransitionInfo
4124 Warden::Manager
4124 MethodOverrideForAll
4124 AccountMiddleware
4124 Rack::Cookies
4125 ActiveRecord::ConnectionManagement
4125 ActionController::Session::CookieStore
4125 ActionController::Failsafe
4125 ActionController::ParamsParser
4125 Rack::Lock
4125 ActionController::Dispatcher
4125 ActiveRecord::QueryCache

middleware chain leaking per request

 god memory leaks
(gdb) ruby objects classes (gdb) ruby objects arrays
43 God::Process elements instances
43 God::Watch 94310 3
43 God::Driver 94311 3
43 God::DriverEventQueue 94314 2
43 God::Conditions::MemoryUsage 94316 1
43 God::Conditions::ProcessRunning
43 God::Behaviors::CleanPidFile 5369 arrays
45 Process::Status 2863364 elements
86 God::Metric
327 God::System::SlashProcPoller
327 God::System::Process
arrays with 94k+
406 God::DriverEvent elements!
 useful tools
• ObjectSpace.each_object
• gdb.rb
• bleak_house
• heap dumping patches
• memprof
harrywood (flickr)
 bleak_house
• http://github.com/fauna/bleak_house
• installs a custom patched version of ruby
 bleak_house
• http://github.com/fauna/bleak_house
• installs a custom patched version of ruby
• tells you what is leaking (like gdb.rb and
ObjectSpace), but also where the leak is
happening
191691 total objects
Final heap size 191691 filled, 220961 free
Displaying top 20 most common line/class pairs

89513 __null__:__null__:__node__
41438 __null__:__null__:String
2348 site_ruby/1.8/rubygems/specification.rb:557:Array
1508 gems/specifications/gettext-1.9.gemspec:14:String
 useful tools
• ObjectSpace.each_object
• gdb.rb
• bleakhouse
• heap dumping patches
• memprof
 100000 file.rb:123:String
useful, 100k strings on this line

but..
sometimes it’s not enough

what is actually inside these strings?

thaths (flickr)
 heap dumping patch
• simple patch to ruby VM (300 lines of C)
• http://gist.github.com/73674
• simple text based output format
0x154750 @ -e:1 is OBJECT of type: T
0x15476c @ -e:1 is HASH which has data
0x154788 @ -e:1 is ARRAY of len: 0
0x1547dc @ -e:1 is STRING len: 1 and val: T
0x154814 @ -e:1 is CLASS named: T inherits from Object
0x154a98 @ -e:1 is STRING len: 2 and val: hi
0x154b40 @ -e:1 is OBJECT of type: Range
$ wc -l /tmp/ruby.heap

 1571529 /tmp/ruby.heap
$ wc -l /tmp/ruby.heap

 1571529 /tmp/ruby.heap

$ cat /tmp/ruby.heap | awk '{ print $3 }' | sort |

uniq -c | sort -g | tail -1

 236840 memcached/memcached.rb:316
$ wc -l /tmp/ruby.heap

 1571529 /tmp/ruby.heap

$ cat /tmp/ruby.heap | awk '{ print $3 }' | sort |

uniq -c | sort -g | tail -1

 236840 memcached/memcached.rb:316

$ grep "memcached.rb:316" /tmp/ruby.heap | awk

'{ print $5 }' | sort | uniq -c | sort -g | tail -2

  64952 HASH
  123290 STRING
$ wc -l /tmp/ruby.heap

 1571529 /tmp/ruby.heap

$ cat /tmp/ruby.heap | awk '{ print $3 }' | sort |

uniq -c | sort -g | tail -1

 236840 memcached/memcached.rb:316

$ grep "memcached.rb:316" /tmp/ruby.heap | awk

'{ print $5 }' | sort | uniq -c | sort -g | tail -2

  64952 HASH
  123290 STRING

$ grep "memcached.rb:316" /tmp/ruby.heap | grep STRING

| awk '{ print $10 }' | sort | uniq -c | sort -g |
tail -2

  72095 int(11)
  79979 varchar(255)
 useful tools
• ObjectSpace.each_object
• gdb.rb
• bleakhouse
• heap dumping patches
• memprof
 memprof goals
• easy to use: no patching the VM, just require
the gem
• detailed: include file/line (bleak_house),
object contents (heap dumping patch), but also
information about references between objects
• simple analysis: allow processing via various
languages and databases using simple JSON data
format
 memprof
• http://github.com/ice799/memprof
• gem install memprof
• under active development on github
• works on x86_64 linux and x86_64 osx
• for best results, use an RVM built 1.8.x or REE
• ruby 1.9 support in the works
• 32bit support in the works
 memprof under the hood
• rewrites your Ruby binary in memory
• injects short trampolines for all calls to internal
VM functions to do tracking
• uses libdwarf and libelf to access VM internals
like the ruby heap slabs
• uses libyajl to dump out ruby objects as json
http://timetobleed.com/string-together-global-offset-tables-to-build-a-ruby-memory-profiler/
http://timetobleed.com/memprof-a-ruby-level-memory-profiler/
http://timetobleed.com/what-is-a-ruby-object-introducing-memprof-dump/
http://timetobleed.com/hot-patching-inlined-functions-with-x86_64-asm-metaprogramming/
http://timetobleed.com/rewrite-your-ruby-vm-at-runtime-to-hot-patch-useful-features/
assembly stub*
*slightly abbreviated

void handler(VALUE freed_object)

{
mark_object_freed(freed_object);
return;
}
http://bit.ly/memprof-abi
 memprof features
• memprof.track
• memprof.dump
• memprof.dump_all
• memprof.com
screenpunk (flickr)
 Memprof.track{
100.times{ "abc" }
100.times{ 1.23 + 1 }
100.times{ Module.new }
}

100 file.rb:2:String
100 file.rb:3:Float
100 file.rb:4:Module

• like bleak_house, but for a given block of code

• use Memprof::Middleware in your webapps to run
track per request
 memprof features
• memprof.track
• memprof.dump
• memprof.dump_all
• memprof.com
stephenr (flickr)
 Memprof.dump{
strings }
"hello" + "world"

{
"_id": "0x19c610", memory address of object
"file": "file.rb",
"line": 2,

"type": "string",
"class": "0x1ba7f0",
"class_name": "String",

"length": 10,
"data": "helloworld"
}
 Memprof.dump{
strings }
"hello" + "world"

{
"_id": "0x19c610", memory address of object
"file": "file.rb", file and line where string
"line": 2,
was created
"type": "string",
"class": "0x1ba7f0",
"class_name": "String",

"length": 10,
"data": "helloworld"
}
 Memprof.dump{
strings }
"hello" + "world"

{
"_id": "0x19c610", memory address of object
"file": "file.rb", file and line where string
"line": 2,
was created
"type": "string",
"class": "0x1ba7f0", address of the class
"class_name": "String", “String”

"length": 10,
"data": "helloworld"
}
 Memprof.dump{
strings }
"hello" + "world"

{
"_id": "0x19c610", memory address of object
"file": "file.rb", file and line where string
"line": 2,
was created
"type": "string",
"class": "0x1ba7f0", address of the class
"class_name": "String", “String”

"length": 10, length and contents

"data": "helloworld" of this string instance
}
 arrays
Memprof.dump{
[
1,
:b,
{
"_id": "0x19c5c0",
2.2,
"d"
"class": "0x1b0d18", ]
"class_name": "Array", }

"length": 4,
"data": [
1,
":b",

"0x19c750",
"0x19c598"
]
}
 arrays
Memprof.dump{
[
1,
:b,
{
"_id": "0x19c5c0",
2.2,
"d"
"class": "0x1b0d18", ]
"class_name": "Array", }

"length": 4,
"data": [
1, integers and symbols are
":b", stored in the array itself
"0x19c750",
"0x19c598"
]
}
 arrays
Memprof.dump{
[
1,
:b,
{
"_id": "0x19c5c0",
2.2,
"d"
"class": "0x1b0d18", ]
"class_name": "Array", }

"length": 4,
"data": [
1, integers and symbols are
":b", stored in the array itself
"0x19c750", floats and strings are
"0x19c598" separate ruby objects
]
}
 hashes
Memprof.dump{
{
:a => 1,
"b" => 2.2
{ }
"_id": "0x19c598", }
"type": "hash",
"class": "0x1af170",
"class_name": "Hash",

"default": null,

"length": 2,
"data": [
[ ":a", 1 ],
[ "0xc728", "0xc750" ]
]
}
 hashes
Memprof.dump{
{
:a => 1,
"b" => 2.2
{ }
"_id": "0x19c598", }
"type": "hash",
"class": "0x1af170",
"class_name": "Hash",

"default": null,

"length": 2,
"data": [
[ ":a", 1 ],
hash entries as key/value
[ "0xc728", "0xc750" ] pairs
]
}
 hashes
Memprof.dump{
{
:a => 1,
"b" => 2.2
{ }
"_id": "0x19c598", }
"type": "hash",
"class": "0x1af170",
"class_name": "Hash",

"default": null, no default proc

"length": 2,
"data": [
[ ":a", 1 ],
hash entries as key/value
[ "0xc728", "0xc750" ] pairs
]
}
 classes
Memprof.dump{
class Hello
@@var=1
Const=2
{ def world() end
"_id": "0x19c408",
end
"type": "class", }
"name": "Hello",
"super": "0x1bfa48",
"super_name": "Object",

"ivars": {
"@@var": 1,
"Const": 2
},
"methods": {
"world": "0x19c318"
}
}
 classes
Memprof.dump{
class Hello
@@var=1
Const=2
{ def world() end
"_id": "0x19c408",
end
"type": "class", }
"name": "Hello",
"super": "0x1bfa48", superclass object reference
"super_name": "Object",

"ivars": {
"@@var": 1,
"Const": 2
},
"methods": {
"world": "0x19c318"
}
}
 classes
Memprof.dump{
class Hello
@@var=1
Const=2
{ def world() end
"_id": "0x19c408",
end
"type": "class", }
"name": "Hello",
"super": "0x1bfa48", superclass object reference
"super_name": "Object",

"ivars": { class variables and constants

"@@var": 1, are stored in the instance
"Const": 2
}, variable table
"methods": {
"world": "0x19c318"
}
}
 classes
Memprof.dump{
class Hello
@@var=1
Const=2
{ def world() end
"_id": "0x19c408",
end
"type": "class", }
"name": "Hello",
"super": "0x1bfa48", superclass object reference
"super_name": "Object",

"ivars": { class variables and constants

"@@var": 1, are stored in the instance
"Const": 2
}, variable table
"methods": {
"world": "0x19c318" references to method objects
}
}
 memprof features
• memprof.track
• memprof.dump
• memprof.dump_all
• memprof.com
shyndarkly (flickr)
844steamtrain (flickr)
Memprof.dump_all("myapp_heap.json")

• dump out every single live object as json

• one per line to specified file
Memprof.dump_all("myapp_heap.json")

• dump out every single live object as json

• one per line to specified file
• analyze via
• jsawk/grep
• mongodb/couchdb
• custom ruby scripts
• libyajl + Boost Graph Library
 memprof features
• memprof.track
• memprof.dump
• memprof.dump_all
• memprof.com
 memprof.com
a web-based heap visualizer and leak analyzer
 memprof.com
a web-based heap visualizer and leak analyzer
 memprof.com
a web-based heap visualizer and leak analyzer
 memprof.com
a web-based heap visualizer and leak analyzer
 memprof.com
a web-based heap visualizer and leak analyzer
 memprof.com
a web-based heap visualizer and leak analyzer
 memprof.com
a web-based heap visualizer and leak analyzer
 memprof.com
a web-based heap visualizer and leak analyzer
 memprof.com
a web-based heap visualizer and leak analyzer
 memprof.com
a web-based heap visualizer and leak analyzer
plugging a leak in rails3
 plugging a leak in rails3
• in dev mode, rails3 is leaking 10mb per request
 plugging a leak in rails3
• in dev mode, rails3 is leaking 10mb per request
 plugging a leak in rails3
• in dev mode, rails3 is leaking 10mb per request

let’s use memprof to find it!

 plugging a leak in rails3
• in dev mode, rails3 is leaking 10mb per request

let’s use memprof to find it!

# in environment.rb
require `gem which memprof/signal`.strip
plugging a leak
in rails3
plugging a leak
in rails3
send the app some
requests so it leaks
$ ab -c 1 -n 30
http://localhost:3000/
plugging a leak
in rails3
send the app some
requests so it leaks
$ ab -c 1 -n 30
http://localhost:3000/

tell memprof to dump

out the entire heap to
json
$ memprof
--pid <pid>
--name <dump name>
--key <api key>
plugging a leak
in rails3
send the app some
requests so it leaks
$ ab -c 1 -n 30
http://localhost:3000/

tell memprof to dump

out the entire heap to
json
$ memprof
--pid <pid>
--name <dump name>
--key <api key>
2519 classes
 2519 classes
30 copies of
TestController
 2519 classes
30 copies of
TestController
 2519 classes
30 copies of
TestController

mongo query for all

TestController classes
 2519 classes
30 copies of
TestController

mongo query for all

TestController classes

details for one copy of

TestController
find references to object
find references to object
 find references to object

holding references
to all controllers
 find references to object

“leak” is on line 178

holding references
to all controllers
• In development mode, Rails reloads all your
application code on every request
• ActionView::Partials::PartialRenderer is caching
partials used by each controller as an optimization
• But.. it ends up holding a reference to every single
reloaded version of those controllers
• In development mode, Rails reloads all your
application code on every request
• ActionView::Partials::PartialRenderer is caching
partials used by each controller as an optimization
• But.. it ends up holding a reference to every single
reloaded version of those controllers
more* memprof features

• memprof.trace
• memprof::tracer

* currently under development

config.middleware.use(Memprof::Tracer)

{
"time": 4.3442,

"rails": {
"controller": "test",
"action": "index"
},

"request": {
"REQUEST_PATH": "/test,,
"REQUEST_METHOD": "GET"
},
config.middleware.use(Memprof::Tracer)

{
"time": 4.3442, total time for request

"rails": {
"controller": "test",
"action": "index"
},

"request": {
"REQUEST_PATH": "/test,,
"REQUEST_METHOD": "GET"
},
config.middleware.use(Memprof::Tracer)

{
"time": 4.3442, total time for request

"rails": { rails controller/action

"controller": "test",
"action": "index"
},

"request": {
"REQUEST_PATH": "/test,,
"REQUEST_METHOD": "GET"
},
config.middleware.use(Memprof::Tracer)

{
"time": 4.3442, total time for request

"rails": { rails controller/action

"controller": "test",
"action": "index"
},

"request": { request env info

"REQUEST_PATH": "/test,,
"REQUEST_METHOD": "GET"
},
config.middleware.use(Memprof::Tracer)

"mysql": {
"queries": 3, 3 mysql queries
"time": 0.00109302
},

"gc": {
"calls": 8,
"time": 2.04925
},
config.middleware.use(Memprof::Tracer)

"mysql": {
"queries": 3, 3 mysql queries
"time": 0.00109302
},

"gc": {
"calls": 8, 8 calls to GC
"time": 2.04925
},
config.middleware.use(Memprof::Tracer)

"mysql": {
"queries": 3, 3 mysql queries
"time": 0.00109302
},

"gc": {
"calls": 8, 8 calls to GC
"time": 2.04925 2 secs spent in GC
},
config.middleware.use(Memprof::Tracer)
"objects": {
"created": 3911103, 3 million objs created
"types": {
"none": 1168831,
"object": 1127,
"float": 627,
"string": 1334637,
"array": 609313,
"hash": 3676,
"match": 70211
}
}
}
config.middleware.use(Memprof::Tracer)
"objects": {
"created": 3911103, 3 million objs created
"types": {
"none": 1168831,
"object": 1127,
"float": 627,
"string": 1334637, lots of strings
"array": 609313,
"hash": 3676,
"match": 70211
}
}
}
config.middleware.use(Memprof::Tracer)
"objects": {
"created": 3911103, 3 million objs created
"types": {
"none": 1168831,
"object": 1127,
"float": 627,
"string": 1334637, lots of strings
"array": 609313, lots of arrays
"hash": 3676,
"match": 70211
}
}
}
config.middleware.use(Memprof::Tracer)
"objects": {
"created": 3911103, 3 million objs created
"types": {
"none": 1168831,
"object": 1127,
"float": 627,
"string": 1334637, lots of strings
"array": 609313, lots of arrays
"hash": 3676,
"match": 70211 regexp matches
}
}
}
config.middleware.use(Memprof::Tracer)
"objects": {
"created": 3911103, 3 million objs created
"types": {
"none": 1168831,
"object": 1127, object instances
"float": 627,
"string": 1334637, lots of strings
"array": 609313, lots of arrays
"hash": 3676,
"match": 70211 regexp matches
}
}
}
config.middleware.use(Memprof::Tracer)
"objects": {
"created": 3911103, 3 million objs created
"types": {
"none": 1168831, 1 million method calls
"object": 1127, object instances
"float": 627,
"string": 1334637, lots of strings
"array": 609313, lots of arrays
"hash": 3676,
"match": 70211 regexp matches
}
}
}
config.middleware.use(Memprof::Tracer)

guides#show action creates

an average of 429k objects
and maximum of 3.98M objects
config.middleware.use(Memprof::Tracer)

guides#show action creates

an average of 429k objects
and maximum of 3.98M objects
config.middleware.use(Memprof::Tracer)

guides#show action creates

an average of 429k objects
and maximum of 3.98M objects
config.middleware.use(Memprof::Tracer)

guides#show action creates

an average of 429k objects
and maximum of 3.98M objects
ttstam (flickr)
dno1967 (flickr)
 more objects
=
longer GC

mckaysavage (flickr)
 longer GC
=
less time to run
your ruby code
kgrocki (flickr)
 fewer objects
=
better
performance
januskohl (flickr)
Use these tools.
 Questions?
Aman Gupta
@tmm1

http://scribd.com/tmm1