2/4/2019 How to Check and Analyze Memory Usage in Solaris – The Geek Diary

THE GEEK DIARY

How to Check and Analyze Memory Usage in Solaris
HOWTOS | BASICS | CONCEPTS

By admin (https://www.thegeekdiary.com/author/sandeep_patil/)

Troubleshooting performance issues is an important skill every system admin must have. This post
is intended to give hints, where to look for in checking and troubleshooting memory usage. In
principle, investigation of memory usage is split in checking usage of kernel memory and user
memory.

Please be aware that in case of a memory-usage problem on a system, corrective actions usually
requires deep knowledge and must be performed with great care. First, you need to check how
much Memory is used in Kernel and how much is used in User Memory. This is important to decide,
which further troubleshooting steps are required.

A very useful mdb dcmd is ‘::memstat‘ (this command can take several minutes to complete). Below
example is from Solaris 11.2.

https://www.thegeekdiary.com/how-to-check-and-analyze-memory-usage-in-solaris/ 1/9
2/4/2019 How to Check and Analyze Memory Usage in Solaris – The Geek Diary

Succeed in negotiations
Build your global business network, negotiation
skill set and international perspective
LSE Online

# echo "::memstat" | mdb -k

Page Summary Pages Bytes %Tot
----------------- ---------------- ---------------- ----
Kernel 585584 4.4G 14%
Defdump prealloc 204802 1.5G 5%
Guest 0 0 0%
ZFS Metadata 21436 167.4M 0%
ZFS File Data 342833 2.6G 8%
Anon 56636 442.4M 1%
Exec and libs 1131 8.8M 0%
Page cache 4339 33.8M 0%
Free (cachelist) 8011 62.5M 0%
Free (freelist) 2969532 22.6G 71%
Total 4194304 32G

User memory usage : print out processes using most USER – memory
% prstat -s size – sorted by userland virtual memory consumption.
% prstat -s rss – sorted by userland physical memory consumption.

% prstat -s rss
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
4051 user1 297M 258M sleep 59 0 1:35:05 0.0% mysqld/10
26286 user2 229M 180M sleep 59 0 0:05:07 0.0% java/53

https://www.thegeekdiary.com/how-to-check-and-analyze-memory-usage-in-solaris/ 2/9
2/4/2019 How to Check and Analyze Memory Usage in Solaris – The Geek Diary

27101user2 237M 150M sleep 59 0 0:02:21 0.0% soffice.bin/5

23335user2 193M 135M sleep 59 0 0:12:33 0.0% firefox-bin/10
3727 noaccess 192M 131M sleep 59 0 0:36:22 0.0% java/18
22751 root 165M 131M sleep 59 0 1:13:12 0.0% java/46
1448 noaccess 192M 108M sleep 59 0 0:34:47 0.0% java/18
10115 root 129M 82M sleep 59 0 0:31:29 0.0% java/41
20274 root 136M 77M stop 59 0 0:04:08 0.0% java/25
3397 root 138M 76M sleep 59 0 0:12:42 0.0% java/37
12949 pgsql 81M 70M sleep 59 0 0:09:36 0.0% postgres/1
12945 pgsql 80M 70M sleep 59 0 0:00:05 0.0% postgres/1

User Memory Usage : shows Shared Memory and Semaphores

% ipcs -a

IPC status from

T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES

https://www.thegeekdiary.com/how-to-check-and-analyze-memory-usage-in-solaris/ 3/9
2/4/2019 How to Check and Analyze Memory Usage in Solaris – The Geek Diary

Message Queues:
q 0 0x55460272 -Rrw-rw---- root root root root 0 0 4194304
q 1 0x41460272 --rw-rw---- root root root root 0 0 4194304
q 2 0x4b460272 --rw-rw---- root root root root 0 0 4194304

T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ

Shared Memory:
m 0 0x50000b3f --rw-r--r-- root root root root 1 4
m 1 0x52574801 --rw-rw---- root oracle root oracle 35 1693450240
m 2 0x52574802 --rw-rw---- root oracle root oracle 35 1258291200
m 3 0x52594801 --rw-rw---- root oracle root oracle 12 241172480
m 4 0x52594802 --rw-rw---- root oracle root oracle 12 78643200
m 5 0x52584801 --rw-rw---- root oracle root oracle 13 125829120
m 6 0x52584802 --rw-rw---- root oracle root oracle 13 268435456
m 7 0x525a4801 --rw-rw---- root oracle root oracle 15 912261120
m 8 0x525a4802 --rw-rw---- root oracle root oracle 15 268435456
m 8201 0x4d2 --rw-rw-rw- root root root root 0 32008

T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS

Semaphores:
s 0 0x1 --ra-ra-ra- root root root root 1
s 1 0 --ra-ra---- root oracle root oracle 36
s 2 0 --ra-ra---- root oracle root oracle 13
s 3 0 --ra-ra---- root oracle root oracle 14
s 4 0 --ra-ra---- root oracle root oracle 16
s 5 0x4d2 --ra-ra-ra- root root root root 1
s 6 0x4d3 --ra-ra-ra- root root root root 1

User Memory Usage : lists User Memory usage of all processes (except PID 0,2,3)
# pmap -x /proc/* > /var/tmp/pmap-x

short list of total usage of these processes

% egrep "[0-9]:|^total" /var/tmp/pmap-x

1: /sbin/init
total Kb 2336 2080 128 -
1006: rlogin cores4

https://www.thegeekdiary.com/how-to-check-and-analyze-memory-usage-in-solaris/ 4/9
2/4/2019 How to Check and Analyze Memory Usage in Solaris – The Geek Diary

total Kb 2216 1696 80 -

1007: rlogin cores4
total Kb 2216 1696 104 -
115: /usr/sbin/nscd
total Kb 4208 3784 1704 -
-- snip --

User Memory Usage : check the usage of /tmp

% df -kl /tmp
Filesystem kbytes used avail capacity Mounted on
swap 1355552 2072 1353480 1% /tmp

Print the biggest 10 les and dirs in /tmp:

% du -akd /tmp/ | sort -n | tail -10

288 /tmp/SUNWut
328 /tmp/log
576 /tmp/ips2
584 /tmp/explo
608 /tmp/ipso
3408 /tmp/sshd-truss.out
17992 /tmp/truss.p
22624 /tmp/js
49208 /tmp

User Memory Usage : Overall Memory usage on system

% vmstat -p 3
memory page executable anonymous filesystem
swap free re mf fr de sr epi epo epf api apo apf fpi fpo fpf
19680912 27487976 21 94 0 0 0 0 0 0 0 0 0 14 0 0

https://www.thegeekdiary.com/how-to-check-and-analyze-memory-usage-in-solaris/ 5/9
2/4/2019 How to Check and Analyze Memory Usage in Solaris – The Geek Diary

3577608 11959480 0 20 0 0 0 0 0 0 0 0 0 0 0 0
3577328 11959240 0 5 0 0 0 0 0 0 0 0 0 0 0 0
3577328 11959112 38 207 0 0 0 0 0 0 0 0 0 0 0 0
3577280 11958944 0 1 0 0 0 0 0 0 0 0 0 0 0 0

scanrate ‘sr’ should be 0 or near zero

User Memory Usage : Swap usage

% swap -l
swapfile dev swaplo blocks free
/dev/dsk/c0t0d0s1 32,25 16 1946032 1946032

% swap -s
total: 399400k bytes allocated + 18152k reserved = 417552k used, 1355480k available

common kernel statistics

Print out all kernel statistics in a parse’able format.

% kstat -p > /var/tmp/kstat-p

kernel memory statistics

% kstat -p -c kmem_cache
% kstat -p -m vmem
% kstat -p -c vmem
% kstat -p | egrep zfs_file_data_buf | egrep mem_total

alternatively to kstat you can get kernel memory usage with kmastat prints kmastat bu ers.

# echo "::kmastat" | mdb -k > /var/tmp/kmastat

https://www.thegeekdiary.com/how-to-check-and-analyze-memory-usage-in-solaris/ 6/9
2/4/2019 How to Check and Analyze Memory Usage in Solaris – The Geek Diary

% more /var/tmp/kmastat
cache buf buf buf memory alloc lloc
name size in use total in use succeed fail
------------------------- ------ ------ ------ --------- --------- -----
kmem_magazine_1 16 470 508 8192 470 0
kmem_magazine_3 32 970 1016 32768 1164 0
kmem_magazine_7 64 1690 1778 114688 1715 0

Look for the highest numbers in column “memory in use” and for any numbers higher than ‘0’ in
column “alloc fail”.

Y O U M AY A L S O L I K E

https://www.thegeekdiary.com/how-to-check-and-analyze-memory-usage-in-solaris/ 7/9
 2/4/2019 How to Check and Analyze Memory Usage in Solaris – The Geek Diary

CHEAT SHEETS

LDOMs (OVM for SPARC) Command line reference (https://www.thegeekdiary.com/ldoms-ovm-for-sparc-

command-line-reference-cheat-sheet/)

Solaris Volume Manager (SVM) command line reference (https://www.thegeekdiary.com/solaris-volume-

manager-svm-command-line-reference-cheat-sheet/)

Solaris ZFS command line reference (https://www.thegeekdiary.com/solaris-zfs-command-line-reference-cheat-

sheet/)
RHEL 7 – RHCSA Notes (Cheat Sheets) (https://www.thegeekdiary.com/rhel-7-rhcsa-notes-cheat-sheets/)

CentOS / RHEL 7 : rewalld Cheat Sheet (https://www.thegeekdiary.com/centos-rhel-7- rewalld-command-line-

reference-cheat-sheet/)

systemd command line reference (cheat sheet) (https://www.thegeekdiary.com/centos-rhel-7-systemd-

command-line-reference-cheat-sheet/)
SOLARIS INTERVIEW QUESTIONS

https://www.thegeekdiary.com/how-to-check-and-analyze-memory-usage-in-solaris/ 8/9
 2/4/2019 How to Check and Analyze Memory Usage in Solaris – The Geek Diary

The ultimate Solaris Volume Manager (SVM) interview questions (https://www.thegeekdiary.com/the-ultimate-

solaris-volume-manager-svm-interview-questions/)

The ultimate Solaris zones (containers) interview questions (https://www.thegeekdiary.com/the-ultimate-solaris-

zones-containers-interview-questions/)

The ultimate Solaris networking interview questions (https://www.thegeekdiary.com/the-ultimate-solaris-

networking-interview-questions/)

The ultimate ZFS interview questions (https://www.thegeekdiary.com/the-ultimate-zfs-interview-questions/)

© 2019 · THE GEEK DIARY (HTTP://THEGEEKDIARY.COM)

https://www.thegeekdiary.com/how-to-check-and-analyze-memory-usage-in-solaris/ 9/9