DMX Performance

Performance Aspects of Open Systems on EMC Platforms (Symmetrix)
Alok Jain
09/12/08
Copyright 2008 Interscape Technologies Inc. All rights reserved.
Agenda
Storage Area Networks from Performance Perspective Open Systems and Performance EMC Storage Platforms Symmetrix Open Systems Tools Performance Metrics for Open Systems Performance Metrics for EMC Storage Performance Analysis using tools Storage Consolidation Planning & Considerations Use Case Study (time permitting)
Storage Performance (SNIA)

Most data centers are dealing with ever increasing amounts of data storage. Although the need for storage seems insatiable, array performance typically plateaus or worse, degrades post installation. Storage performance tuning is a necessary and ongoing activity..
Storage Area Networks and Performance
Data replication to remote site Storage IO performance issues can occur at any of these layers.
Open Systems and Storage Performance

Host
p Ap n io at ic l
Application performs an I/O

Database changes I/O subject to db_block_size, multiblock_read_count, and other parameters. File System changes I/O to conform to File System block size, fragment size, or memory page size LVM breaks up according to stripe width & I/O parameter settings Path manager may affect I/O channel and I/O size Driver may break up I/O (max_phys and other parameters) Hardware delivers the I/O as requested
Database
File System
LVM Path Manager DRIVER HARDWARE
SYMMETRIX SYMMETRIX
Symmetrix sees something very different than the initial size of the I/O!
Application and System Performance Key Metrics Three Ways to Measure Application Performance
Throughput (MB/s) Number of transactions per second Application run time
Four Ways to Measure IO Sub-System Performance

IO/sec (IO operations per sec) MB/sec (Bytes transferred/sec) I/O Service Times (average time to do an I/O) I/O Queue Lengths (outstanding IO waiting to be sent)
Types of IO & Performance Impacting Points There is a limited number of basic I/O operations
Random read Sequential read Random write Sequential write
Each type has cache hit and cache miss variants Block Size impacts the throughput (IO/sec) and Bandwidth (MB/s)
Application Work Load Characteristics

All applications perform the following:
Read/Write data to/from disk systems (milliseconds) Process data in CPU (microseconds) Write data back to disk system (milliseconds)
Performing an IO is an order of magnitude time consuming compared to a CPU operation
You need to know

Overall I/O rate % Read + read size % Write + write size % Random, % sequential % Cache hit (locality of reference)
8
Physical Drives Key Consideration

Number of disks and speed of disks can limit IO Subsystem performance The 7.2K RPM drives are good for sequential and large I/O sizes. A little slower at random access than the 10K RPM drives The 10K RPM drives give good, all-around performance 15K RPM Drives boost random I/O access rates
30% more random-access, small transfers than the 10K RPM drives They do not benefit much in large (64KB +) or sequential operations, as transfer rates are not as affected by rotational speed
If I/O is hitting the cache, theyre all the same speed

For writes, disk type is unimportant as long as write cache is not saturated Prefetch works at about the same rate for all of them What differs is how well they do random reads, and how fast they flush random writes from the cache
9
Drives can be roughly divided into families

The major distinction is the rotational speed
Faster rotation generally also means faster seek
Fastest Specification
Drive Capacity (Gbyte) Internal data rate (MB/sec) Disc Rotational Speed (r/min) Avg Rotational Latency (ms) Avg Read Seek (ms) Avg Write Seek (ms) Read Single Track Seek (ms) Write Single Track Seek (ms) Read Full Stroke Seek (ms) Write Full Stroke Seek (ms)
Fast
Not Fast
146GB 15k 73GB 15k 300GB 10k 146GB 10k 73GB 10k 500GB 7200 RPM RPM RPM RPM RPM RPM 146 68 - 114 15,000 2.00 3.50 4.00 0.27 0.45 7.40 7.90 73 68 - 114 15,000 2.00 3.50 4.00 0.27 0.45 7.40 7.90 300 47 - 94 10,000 3.00 4.70 5.40 0.20 0.50 9.70 10.40 146 47 - 94 10,000 3.00 4.70 5.40 0.20 0.50 9.70 10.40 73 47 - 94 10,000 3.00 4.70 5.40 0.20 0.50 9.70 10.40 500 47 - 94 7,200 4.17 8.50 9.50 0.80 1.00 16.00 16.55
10
RAID Level Considerations for IO Sub-Systems
RAID Level
RAID 0
RAID Group Configuration

Striped set without parity
Comments
Lowest data availability. Lowest cost. Highest performance. Not Supported in EMC Symmetrix Higher data availability. High cost. Best Sequential IO performance
RAID 1
Mirrored set
RAID 5
Striped set with distributed High data availability. Lower cost. parity Cost effective performance Striped set with dual distributed parity Striped + Mirrored Highest data availability. Low cost. Slower than R5 performance High data availability. High cost. Best performance for random reads & writes
RAID 6
RAID 1+0
11
EMC Symmetrix RAID 1 (Sample Symm View) RAID 1

Channel
Writes go to two drives Read from 1 drive

DMSP policy
Directory Channel Director Global Memory Disk Disk Director
High level of protection Most expensive $$$

2x raw storage
M1
Disk
M2
12
Striping At Host Level & Array level (Plaids for Best Performance) LUN = Logical Unit Number (drive letter/#) LUNs striped across multiple RAID groups (of same type)
Eliminate hot RAID groups, hot drives, hot back-end links Variations include, RAID 0+1, 1+0, 5+0, 10, or 50
13
Open Systems Performance Measurement Tools
OS Platform Solaris Linux AIX Windows
Tool Iostat & sar Iostat & sar Iostat & sar Perfmon
Command Example Iostat xtczn T d 5 Sar d (many options) Iostat xt 5 Sar d (many options) Iostat x 5 Sar d (many options) GUI based (with CLI options also)
14
Iostat Unix Based Systems (Solaris)

Cmds actively Cmds in Wait queue Being executed Wait tiime In queue Actual Service Time
Throughput Reads/sec & Writes
Bandwidth Read & Write KB/sec
From Performance perspective the bottlenecks can be identified by looking at Wait Queues, Wait Service Time and Active Service Time
15
Iostat Unix Based Systems (Solaris)

Performance Throughput Indicators
Reads/sec & Writes/sec Read Kbytes/sec & Write Kbytes/sec
Performance Bottlenecks Indicators

Active Service Time This can be application dependent. It should be compared to a baseline to determine the changes when the problem occurs Wait Queues present signifies IO sub-system is not able to keep up with demand at the LUN level Wait Service Time Any wait service time will introduce IO latency
16
Sar Unix based Systems (Solaris)

Avg Queues Throughput Bandwidth Total IO/sec total blocks/sec+ Avg. Wait Time (ms)
Avg. Service Time (ms)
SAR d output. The metrics are different compared to IOSTAT output. Reads and writes data is shown together %busy indicator is indicative only.
17
Perfmon Windows Based Systems
Windows based systems can use Perfmon to do performance analysis

18
Windows Host EMC Symmetrix Array
Windows Host Drives mapped to EMC Symmetrix physical disks. The color shows how utilized the physical disk spindles are in terms of IO performance. This provides a comprehensive view of Host devices in the Storage Array
19
EMC Storage Platforms Performance Measurement Tools
EMC Platform Symmetrix Clariion
Tool
Work Load Analyzer (WLA) Symstat (CLI) Navisphere Analyzer Navicli (CLI)
Example
WLA is GUI based Symstat sid 1234 c 5 I 5 GUI based Navicli h ip_address ..
20
Symmetrix Components from Performance Perspective

Symmetrix DMX Critical Component
%Cache Hits Fiber Adapter Channels Disk Directors Remote Fiber Director Device Utilizations Disk Utilizations (physical)
Symmetrix Key Performance Metrics (at eac component level)

Read/write IO/sec Read/write Kbytes/sec %cache hits Block Sizes Write Pending Slot Counts
21
EMC Symmetrix Global Memory Management & Data Flow
22
EMC DMX-4 Symmetrix Array Architecture
Front-End To Hosts
Cache
Back-End To Disks
23
EMC Symmetrix Tiering within arrays
24
EMC DMX-4 Symmetrix Disk Quadrant Layout
All drive types in all quadrants to leverage all Disk Adapters for Tiering
25
EMC Symmetrix Performance Viewer (Work Load Analyzer)
26
EMC Symmetrix IO/sec & MB/sec Metrics
Read IO/sec, Write IO/sec, Total IO/sec
Reads MB/sec, Write MB/sec, Total MB/sec
Monitor overall IO/sec and MB/sec for the whole array. Confirm the storage array is not over loaded
27
EMC Symmetrix Write Pending Slots Vs. Max WP Slots
Monitor overall Write Pending slot counts for the whole array. Crossing Max thresholds will impact performance
28
EMC Symmetrix %Cache Hits
Monitor %Cache Hits. For Open Systems anything below 70% may impact performance negatively
29
EMC Symmetrix Front-End Fiber Adapter CPU IOPS
Monitor Fibre Adapter CPU IO/sec on each individual FA CPU Identify busiest FA CPUs
30
EMC Symmetrix Front-End Fibre Adapter FA Port Utilization (based on IO/sec)
Identify FA Ports that are highly utilized. Anything over 90% may be of concern
31
EMC Symmetrix Remote Adapters IOPS & MBPS (used for remote replication)
Identify RA Ports that are doing heavy IOs. Generally most RAs will have similar IO profiles
32
EMC Symmetrix Disk Adapters (DA)
Identify DA Ports that are doing heavy IOs. Generally most DAs will have similar IO profiles if the hardware configuration is setup properly.
33
EMC Symmetrix Devices Sampled Average read and write times
Identify devices that are doing very high reads and writes. These devices then need to be mapped to the hosts that they are provisioned to
34
EMC Symmetrix Disk SCSI Command Loads
Identify physical disks that are very busy. Any 10K RPM disk doing over 120-140 IOPS points to hot disks which will create bottlenecks
35
EMC Symmetrix Disk to Device Mappings (Identify Bad Neighbors)
Identify busiest devices on the actual physical disks. This will point to hosts that may be starving other hosts for IO
36
EMC Symmetrix Component %Utilization HEAT Chart
This provides a one page view into the Symmetrix array. The %utilization colors points to how hot each of the components are
37
Questions?
38
Contact details
U.S. Office: 200 Centennial Avenue, Suite 200, Piscataway, NJ 08854 Office: 1.732.868.6320 Fax: 1.973.302.8135
Alok Jain Cell: 848.248.0100 Office: 732.868.6320 Email id.: alok.jain@interscapetech.com
39
Storage Consolidation Analysis Considerations & Use Case
40
IT Assets in General are Poorly Used
Typical Utilization Rates
Servers: PCs: Direct-attach storage: Network storage:
5-15% 10-20% 20-40% 60-80%
A typical x86 server consumes between 30% and 40% of its maximum power when idle
Sources: VMware, Microsoft, EMC
41
Storage Consolidation Energy Benefits

Storage consolidation saves energy Fewer and larger storage arrays
Consolidate and optimize workloads Maximize storage utilization
Requires understanding and planning

What are workloads? What are storage array capabilities? What are the performance characteristics? What are the data protection and business continuity requirements?
42
RAID 3+1 Annual Energy Cost Usable Terabyte vs Drive Size

1,000,000.00 Energy Cost @ $0.15/kWhr 800,000.00 600,000.00 400,000.00 200,000.00 0.00 0 500 1000 Usable Terabytes (TB) 73 GB Yearly Energy Cost 3+1 146 GB Yearly Energy Cost 3+1 300 GB Yearly Energy Cost 3+1 500 GB Yearly Energy Cost 3+1
Higher density disks brings the TCO down

43
Use Case: Storage Savings on Capital, Mgmt and Energy

Customer Example: Major Wireless Provider 44 Mid-tier systems consolidated to 2 High-end systems
Mid-Tier
885 TB in 146 GB drives
High-end Array
934 TB in 146 GB and 300 GB drives Power and
317 kWh $416,538
Cooling Annual Cost @.15 kWh
169 kWh
$222,066
1,296,480 kWh - Annual Energy Savings - $194,472

44
Use Case: Driving Factors Migration & Consolidation Planning Client is migrating Data Center form NY to NJ for various business reason Client would like to perform a detailed storage consolidation analysis as part of decision on migration strategy Client wants to look at their Tier-1 Storage environment which is mostly on EMC Symmetrix with SRDF Replication to another Data Center The consolidation analysis needs to model
Overall Storage Array Performance Storage Capacity Consolidation Requirements Technical Architecture at component levels
LUN Limitations Host Fan-Out ratios Device masking thresholds
45
ALL EMC Symmetrix Performance System IOPS & MBPS

Symmetrix Symmetrix ID Pair Alias Location 190100031 190100032 NA DC 1 190100011 190100012 NA DC 1 190100015 190100016 NA DC 1 187750019 187750020 NA DC 1 187720023 187720024 NA DC 1 187720025 187700026 NA DC 1 187720027 187720028 NA DC 1 187720013 187700014 NA DC 1 287750029 287750030 NA DC 1 187720021 187720022 NA DC 1 187720017 187720018 NA DC 1 190100032 NA DC 2 190100016 NA DC 2 190100012 NA DC 2 187750020 NA DC 2 187720024 NA DC 2 187720022 NA DC 2 287750030 NA DC 2 187720018 NA DC 2 187700014 NA DC 2 187720028 NA DC 2 187700026 NA DC 2 95th 95th Symm Max Max Percentile Percentile Avg Mode (IO/Sec) (IO/Sec) Avg (IO/Sec) (MB/Sec) (MB/Sec) (MB/Sec) R1 35474 26244 16614 531 408 230 R1 32126 23554 12358 726 483 217 R1 30582 21398 12666 45 347 176 R1 24750 21812 12715 316 227 128 R1 16707 10049 5685 161 97 43 R1 14022 9341 4005 210 118 46 R1 13298 8976 4683 195 150 68 R1 12849 8696 4735 220 114 53 R1 12215 7796 4065 193 140 51 R1 11512 8739 4308 215 125 50 R1 9977 7741 4453 145 101 48 R2 17003 11739 7147 249 168 78 R2 14983 11123 6784 173 97 41 R2 11703 7677 3672 295 207 154 R2 10755 6745 4174 90 70 28 R2 7720 3197 766 99 47 10 R2 5764 2858 1381 27 18 8 R2 5177 3356 1634 43 22 11 R2 5048 2180 971 24 18 6 R2 2859 544 188 38 7 2 R2 2387 1360 308 10 5 1 R2 212 67 46 3 0 0
S. No 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
TimeStamp 7/11/2008 21:45 7/13/2008 12:45 7/11/2008 22:15 7/10/2008 22:15 7/13/2008 0:45 7/11/2008 22:00 7/10/2008 21:15 7/10/2008 21:30 7/11/2008 22:30 7/11/2008 21:30 7/12/2008 2:30 7/11/2008 22:45 7/12/2008 19:30 7/11/2008 23:15 7/12/2008 22:30 7/11/2008 20:15 7/12/2008 22:45 7/10/2008 22:30 7/10/2008 21:45 7/12/2008 19:45 7/10/2008 20:15 7/11/2008 21:15
46
ALL EMC Symmetrix Performance Max System IOPS & MBPS R1+R2 View
No 1 2 3 4 5 6 7 8 9 10 11
Source Array 0031-0032 0015-0016 0011-0012 0019-0020 0023-0024 0027-0028 0029-0030 0025-0026 0013-0014 0021-0022 0017-0018
Model DMX3-24 - DMX3-24 DMX3-24 - DMX3-24 DMX3-24 - DMX3-24 2000P-M2 - 2000P-M2 2000S-M2 - 2000S-M2 2000S-M2 - 2000S-M2 2000S-M2 - 2000S-M2 2000S-M2 - 2000S-M2 2000S-M2 - 2000S-M2 2000S-M2 - 2000S-M2 2000S-M2 - 2000S-M2
Location DC1 - DC2 DC1 - DC2 DC1 - DC2 DC1 - DC2 DC1 - DC2 DC1 - DC2 DC1 - DC2 DC1 - DC2 DC1 - DC2 DC1 - DC2 DC1 - DC2
Symm Mode R1-R2 R1-R2 R1-R2 R1-R2 R1-R2 R1-R2 R1-R2 R1-R2 R1-R2 R1-R2 R1-R2
System Max Total System Max Total IOPS (R1+R2) MBPS (R1+R2) 43278 39648 35214 30579 17205 14621 14388 14066 13213 12927 11510 720 459 889 329 151 96 165 210 154 214 148
Aggregate IOPS & MBPS for each interval across the pairs. This reflects a more realistic failed over view
47
Consolidation Analysis Target Requirements
Target Config ID T1 T2 T3 T3
Target Model
Target Disk Size
Target Disk RPM
Target RAID Level RAID-5 RAID-1+0 RAID-1+0 RAID-1+0
Target RAID Config 3+1 4+4 4+4 4+4
DMX-4 DMX-4 DMX-4 DMX-4
300GB 300GB 300GB 400GB
15K 15K 10K 10K
48
Symmetrix Performance IOPS & MBPS Profiles
49
Source Storage Arrays For Consolidation Targets
50
Target Storage Array Specifications
51
Source Storage #1 Performance Baseline
52
Source Storage #2 Performance Baseline
53
Symmetrix Sample Aggregated System IOPS & MBPS
Sample shows two EMC Symmetrix IOPS and MBPS Consolidated View
54
Symmetrix Sample Aggregated RA IOPS & MBPS
Sample shows two EMC Symmetrix Replication Channel Consolidated View

55
Symmetrix Sample FA IOPS, MBPS, Masked Devices & FAN-OUT Analysis
Sample shown for two source array consolidation analysis view

56
Target Storage Array Performance Heat Chart (Model)
57
Symmetrix Sample Consolidated Target Storage Requirements
Sample shown for 2 source consolidation analysis view. Shows types of total device configs needed
58
ALL Symmetrix 2 to 1 Consolidation with RAID 1+0 Overall Status

Consol Source Group Array Pair 0011-0012 Model DMX3-24 DMX3-24 Location DC1-DC2 Symm Mode Bi-Dir System Max System Max Total IOPS Total MBPS (R1+R2) (R1+R2) 35214 889 Target Disk Recomm Target . Disk Storage Size (GB) Count %Utilized Status
2000S-M2 2000S-M2 DMX3-24 0015-0016 DMX3-24 0013-0014 2000S-M2 2000S-M2 2000P-M2 0019-0020 2000P-M2 0017-0018 2000S-M2 2000S-M2 2000S-M2 0023-0024 2000S-M2 0021-0022 2000S-M2 2000S-M2 2000S-M2 0027-0028 2000S-M2 0025-0026 2000S-M2 0029-0030 2000S-M2 0031-0032 DMX3-24 DMX3-24
DC1-DC2 DC1-DC2 DC1-DC2 DC1-DC2 DC1-DC2 DC1-DC2 DC1-DC2 DC1-DC2 DC1-DC2 DC1-DC2
Bi-Dir Bi-Dir Bi-Dir Bi-Dir Bi-Dir Bi-Dir Bi-Dir Bi-Dir Bi-Dir Bi-Dir
13213 39648 11510 30579 12927 17205 14066 14621 14388 43278
154 459 148 329 214 151 210 96 165 720
300
720
57%
OK
300
480
60%
OK
300
480
40%
OK
300
480
27%
OK
5 6
300 300
480 480
24% 72%
OK OK
59
Consolidation End Results

22 EMC DMX-2000 Symmetrix 73GB 15K x 6336 Drives 90% Storage utilized growth available ~500 Hosts 10 EMC DMX-4 Symmetrix
300GB 10K x 4800 Drives
~35% Storage Utilized with plenty of
~500 Hosts
Data Migration using Symmetrix SRDF Array based utility Lowered Operational and Management Cost Much less spinning disks with plenty of growth left Performance modeling helped in minimizing performance impact
60
Questions?
61

DMX Performance

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

DMX Performance

Transféré par

Droits d'auteur :

Formats disponibles

Performance Aspects of Open Systems on EMC Platforms (Symmetrix)

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Storage Performance (SNIA)

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Storage Area Networks and Performance

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Open Systems and Storage Performance

Application performs an I/O

LVM Path Manager DRIVER HARDWARE

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Four Ways to Measure IO Sub-System Performance

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Application Work Load Characteristics

You need to know

Physical Drives Key Consideration

If I/O is hitting the cache, theyre all the same speed

Drives can be roughly divided into families

Copyright 2008 Interscape Technologies Inc. All rights reserved.

RAID Level Considerations for IO Sub-Systems

RAID Group Configuration

Copyright 2008 Interscape Technologies Inc. All rights reserved.

EMC Symmetrix RAID 1 (Sample Symm View) RAID 1

Writes go to two drives Read from 1 drive

High level of protection Most expensive $$$

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Open Systems Performance Measurement Tools

OS Platform Solaris Linux AIX Windows

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Iostat Unix Based Systems (Solaris)

Throughput Reads/sec & Writes

Bandwidth Read & Write KB/sec

Iostat Unix Based Systems (Solaris)

Performance Bottlenecks Indicators

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Sar Unix based Systems (Solaris)

Avg. Service Time (ms)

Perfmon Windows Based Systems

Windows based systems can use Perfmon to do performance analysis

Windows Host EMC Symmetrix Array

EMC Storage Platforms Performance Measurement Tools

EMC Platform Symmetrix Clariion

Copyright 2008 Interscape Technologies Inc. All rights reserved.

Symmetrix Components from Performance Perspective

Symmetrix Key Performance Metrics (at eac component level)

Copyright 2008 Interscape Technologies Inc. All rights reserved.

EMC Symmetrix Global Memory Management & Data Flow

Copyright 2008 Interscape Technologies Inc. All rights reserved.

EMC DMX-4 Symmetrix Array Architecture

Copyright 2008 Interscape Technologies Inc. All rights reserved.

EMC Symmetrix Tiering within arrays

Copyright 2008 Interscape Technologies Inc. All rights reserved.

EMC DMX-4 Symmetrix Disk Quadrant Layout

Copyright 2008 Interscape Technologies Inc. All rights reserved.

EMC Symmetrix Performance Viewer (Work Load Analyzer)

Copyright 2008 Interscape Technologies Inc. All rights reserved.

EMC Symmetrix IO/sec & MB/sec Metrics

Read IO/sec, Write IO/sec, Total IO/sec

Reads MB/sec, Write MB/sec, Total MB/sec

EMC Symmetrix Write Pending Slots Vs. Max WP Slots

EMC Symmetrix %Cache Hits