Vous êtes sur la page 1sur 57

High Level Performance.

The easily identifiable performance


issues.
Bonnie LeBarron
IT Management Consultant
IBM Lab Services

2019 AIX Users Group


Session Objectives
• Take a look at CPU, Memory, IO, and various tunables to identify easy
to find performance issues.

2019 IBM Systems Technical University 2 © Copyright IBM Corporation 2019


NUMA or Affinity

2019 IBM Systems Technical University 3 © Copyright IBM Corporation 2019


NUMA

2019 IBM Systems Technical University


Affinity

2019 IBM Systems Technical University


Affinity FAR L

2019 IBM Systems Technical University


OCHDBNEW Far Affinity

2019 IBM Systems Technical University


Affinity Local J

lssrad –ad
Ref SRAD MEM CPU
0
0 24000 1-16

2019 IBM Systems Technical University


Affinity Near J

lssrad –ad
Ref SRAD MEM CPU
0
0 24000 1-8
1 12-24

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) – System View

lsmemopt -m kurtkP8 -r sys -o calcscore


curr_sys_score=90,predicted_sys_score=99,requested_lpar_
ids=none,protected_lpar_ids=none

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) - LPAR View
hscroot@mghmc2:~> lsmemopt -m kurtkP8 -r lpar -o calcscore
lpar_name=kk1vios1,lpar_id=1,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=kk1vios2,lpar_id=2,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gb_client5,lpar_id=3,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=SLES-12-SMT,lpar_id=4,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=yy_client3,lpar_id=5,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=HANA_n1,lpar_id=6,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=cc_server1,lpar_id=7,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gb_client6,lpar_id=8,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=cc_server2,lpar_id=9,curr_lpar_score=89,predicted_lpar_score=74
lpar_name=cc_server3,lpar_id=10,curr_lpar_score=100,predicted_lpar_score=90
lpar_name=cc_server4,lpar_id=11,curr_lpar_score=55,predicted_lpar_score=100
lpar_name=aaron_client1,lpar_id=12,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=jrc_client1,lpar_id=13,curr_lpar_score=4,predicted_lpar_score=100
lpar_name=bf_client1,lpar_id=14,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gb_client3,lpar_id=15,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gb_client9,lpar_id=16,curr_lpar_score=50,predicted_lpar_score=100
lpar_name=HANA_n2,lpar_id=18,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=HANA_n3,lpar_id=19,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=yy_client1,lpar_id=20,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gb_client8,lpar_id=21,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=HANA_n4,lpar_id=22,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=HANA_n5,lpar_id=23,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=yy_client2,lpar_id=24,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gb_client7,lpar_id=26,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=tk_client1,lpar_id=27,curr_lpar_score=none,predicted_lpar_score=none
lpar_name=yy_client4,lpar_id=28,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gb_client1,lpar_id=31,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gb_client4,lpar_id=41,curr_lpar_score=90,predicted_lpar_score=100
lpar_name=gb_client2,lpar_id=42,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gb_client10,lpar_id=43,curr_lpar_score=0,predicted_lpar_score=100
lpar_name=gw_client55,lpar_id=55,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gw_client61,lpar_id=61,curr_lpar_score=100,predicted_lpar_score=100
lpar_name=gj_client1,lpar_id=101,curr_lpar_score=100,predicted_lpar_score=100
hscroot@mghmc2:~>
2019 IBM Systems Technical University
Dynamic Platform Optimizer (DPO) - LPAR View

# hostname
gb-client9
# lssrad -av
REF1 SRAD MEM CPU
0
0 2851.31 0-3 8-11
1 3072.00 4-7
#

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) - LPAR View

# hostname
gb-client9
# lssrad -av
REF1 SRAD MEM CPU
0
0 2851.31 0-3 8-11
1 3072.00 4-7
#

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) - Start Sys DPO

optmem -m kurtkP8 -o start -t affinity

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) - HMC Enhanced UI

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) - HMC Enhanced UI

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) - HMC Enhanced UI

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) - HMC Enhanced UI
Score

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) - HMC Enhanced UI
Schedule

2019 IBM Systems Technical University


Dynamic Platform Optimizer (DPO) - HMC Enhanced UI
After DPO

2019 IBM Systems Technical University


Data Collection

2019 IBM Systems Technical University 21 © Copyright IBM Corporation 2019


NMON collection flags we use in LBS.

VIOS :

/usr/bin/nmon -f –s360 –c240 –m /yourdatacollectiondir -AXPVdO^

CLIENTS:

/usr/bin/nmon -f –s360 –c240 –m /yourdatacollectindir -XdMA^OPV

* Remember Tuning is an iterative process. Make changes and monitor. More changes may need to be made.

2019 IBM Systems Technical University 22 © Copyright IBM Corporation 2019


CPU

2019 IBM Systems Technical University 23 © Copyright IBM Corporation 2019


Size the CPU
• Look at the Average of the LPAR. This should be the EC assigned to the
LPAR.
• Look at the Peaks of the LPAR. This should be the VP assigned to the
LPAR.

**LBS has a Capacity Planning Tool that can do this for all of your
partitions.

© Copyright IBM Corporation 2014


2019 IBM Systems Technical University
CPU (Avg .4) EC=.2 VP=2
Where should they really be?

2019 IBM Systems Technical University 25 © Copyright IBM Corporation 2018


CPU (Avg .2) EC=.5 VP=6
Where should they really be?

2019 IBM Systems Technical University 26 © Copyright IBM Corporation 2018


CPU (Avg 2.7) EC=3 VP=5
Where should they really be?

2019 IBM Systems Technical University 27 © Copyright IBM Corporation 2018


Memory

2019 IBM Systems Technical University 28 © Copyright IBM Corporation 2018


Three indicators memory is low.

• Memory is low in the graph or vmstat.


• Free Frame Wait’s are occurring (vmstat –s)
• Scanning and Freeing is occurring (nmon or vmstat) > 4

2019 IBM Systems Technical University


Memory Too Low

2019 IBM Systems Technical University


Memory Too Low

2019 IBM Systems Technical University


Scanning and Freeing

2019 IBM Systems Technical University


Free Frame Waits

Every time the CPU is sitting and waiting for Memory to free
it adds to this number. Since Boot.

2019 IBM Systems Technical University


I/O

2019 IBM Systems Technical University


In Iostat or Nmon the Disk Read/Write
Service Times should be in band.

For Reads this number is < 8ms.

For Writes this number is < 3ms.

If these numbers are greater than the 8 for Reads, or the 3


for Writes, then these disks are considered hot and
should be looked at to alleviate the hot spots. By either
the SAN admin or by the DBA’s/APP’s adminsmoving
files to less busy disks.

2019 IBM Systems Technical University


Disk Read Service Times > 8

2019 IBM Systems Technical University


Disk Write Service Times > 3

2019 IBM Systems Technical University


I/O Bottlenecks Prior to the Disk

2019 IBM Systems Technical University


JFS2 Blocked due to lack of IO buffers

2019 IBM Systems Technical University


Commands

▪ To see Depleted IO buffers


− Vmstat –v (nmon BBBP tab has this as well) This is since boot. This is the JFS2
buffers. To tune this value this is an IOO tunable. First step is to tune
j2_dynamicBufferPreallocation. If you still have increases after you have maxed
this value. Then tune the j2_nBufferPerPagerDevice (which is a restricted tunable.
)

2019 IBM Systems Technical University


Commands

▪ To see Depleted IO buffers (JFS2 continued)


− Vmstat –v (nmon BBBP tab has this as well) This is since boot. This is the JFS2
buffers. To tune this value this is an IOO tunable. First step is to tune
j2_dynamicBufferPreallocation. (Does not require a reboot, max is 256) If you still
have increases after you have maxed this value. Then tune the
j2_nBufferPerPagerDevice (which is a restricted tunable. Requires a reboot)
−.

2019 IBM Systems Technical University


JFS2 Blocked due to lack of IO buffers
▪ To see Depleted IO buffers
− Vmstat –v (nmon BBBP tab has this as well) This is since boot. This is the JFS2
buffers. To tune this value this is an IOO tunable. First step is to tune
j2_dynamicBufferPreallocation. If you still have increases after you have maxed
this value. Then tune the j2_nBufferPerPagerDevice (which is a restricted tunable.
)

2019 IBM Systems Technical University


LVM Blocked due to lack of IO buffers

2019 IBM Systems Technical University


Commands

– To see Depleted IO buffers Vmstat –v (nmon BBBP tab has this as well)
This is since boot. This is at the VG level. To tune this value this is a lvmo
tunable
– Run lvmo –v <volumegroup> -a (look for global_blocked_io_count to see if
it is incrementing)

2019 IBM Systems Technical University


Commands

▪ To see Depleted IO buffers LVM level cont’d


▪ Vmstat –v (nmon BBBP tab has this as well) This is since boot. This is at
the VG level. To tune this value this is a lvmo tunable
▪ Run lvmo –v <volumegroup> -a (look for global_blocked_io_count to see if
it is incrementing)

▪ To increase these buffers :


▪ lvmo -v datavg -o pv_pbuf_count=2048 (Max is 16384)

▪ This will require a reboot of the system.

▪ Continue to monitor and increase if needed.

2019 IBM Systems Technical University


IO Blocked due to lack of Queue Depth buffers:
iostat -D

hdisk153 xfer: %tm_act bps tps bread bwrtn


2.5 274.7K 3.2 273.4K 1.3K
read: rps avgserv minserv maxserv timeouts fails
3.2 2.1 0.0 0.0 0 0
write: wps avgserv minserv maxserv timeouts fails
0.1 0.9 0.0 0.0 0 0
queue: avgtime mintime maxtime avgwqsz avgsqsz sqfull
0.0 0.0 0.0 0.0 0.0 20.0

This value isn’t as black and white as the previous buffers.


If sqfull is low like 1 or 2 then probably not an issue, if it
is higher then you may need to increase the queue_depth.

2019 IBM Systems Technical University


IO Blocked due to lack of num_cmd_ele buffers
fcstat output

This buffer isn’t as black and white as the first two, however, if you
Take these numbers below and divide by the number of seconds the
System has been up this will give you the number per second.

FC1 fcs0
FC1 fcs0 FC SCSI Adapter Driver Information
FC1 fcs0 No DMA Resource Count 2456790 Max_xfer_size
FC1 fcs0 No Adapter Elements Count 34238956 Num_cmd_elem
FC1 fcs0 No Command Resource Count 50234 Num_cmd_elem

2019 IBM Systems Technical University


Commands
• No DMA resource count > increase max_xfer_size
• No Adapter Elements Count > increase num_cmd_elems
• No command resource count > increase num_cmd_elems

** Ensure you do not overrun the FC adapter on the VIO with Clients
being to high.

2019 IBM Systems Technical University


Dropped network packets due to lack of buffers
• To determine if you are dropping packets
• entstat –d ent#
– Look at Packets Dropped:

2019 IBM Systems Technical University


Dropped network packets due to lack of buffers
• To determine if you are dropping packets
• entstat –d ent#
– Look at Packets Dropped:
•ETHERNET STATISTICS (ent4) :
Device Type: Virtual I/O Ethernet Adapter (l-lan)
Hardware Address: 3a:74:36:fd:a7:09

Transmit Statistics: Receive Statistics:


-------------------- -------------------
Packets: 207621462334 Packets: 581294665257
Bytes: 21512405442141 Bytes: 844536159722104
Interrupts: 0 Interrupts: 149392917398
Transmit Errors: 82 Receive Errors: 0
Packets Dropped: 82 Packets Dropped: 14957974
Bad Packets: 0

Max Packets on S/W Transmit Queue: 0


S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 0

Broadcast Packets: 237342439 Broadcast Packets: 12542211


Multicast Packets: 264437369 Multicast Packets: 12766154
No Carrier Sense: 0 CRC Errors: 0
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 0
Max Collision Errors: 0 No Resource Errors: 14957974

2019 IBM Systems Technical University


Determining which buffers are depleted.
• To determine if you are dropping packets
• entstat –d ent#
– Look at Packets Dropped:

Receive Information
Receive Buffers
Buffer Type Tiny Small Medium Large Huge
Min Buffers 512 512 128 24 24
Max Buffers 2048 2048 256 64 64
Allocated 885 512 128 24 24
Registered 512 512 128 24 24
History
Max Allocated 1353 2048 128 24 24
Lowest Registered 501 397 128 24 24

2019 IBM Systems Technical University


Commands
• To enable network buffers: smitty devices>change/show characteristics
of an ethernet adapter. Increase the min/max tiny, small, med, large
and huge to maximum. (This requires a reboot)

2019 IBM Systems Technical University


One last note
• Ensure you don’t have old tunables still set that have been changed.
• This can cause performance issues.

2019 IBM Systems Technical University


Commands

2019 IBM Systems Technical University


Commands

▪ To see Depleted NFS buffers


▪ Vmstat –v (nmon BBBP tab has this as well) This is since boot.
▪ client filesystem I/Os blocked with no fsbuf:
▪ Use the following commands to change them both, use the following order:
# nfso -o nfs_v3_vm_bufs= 20000
# nfso -o nfs_v3_pdts = 2

(or the NFS4 equivalents).


▪ Continue to monitor and increase if needed.

2019 IBM Systems Technical University


Softlive3 Blocked IO Buffers

2019 IBM Systems Technical University


Commands

2019 IBM Systems Technical University