INF-BCO1159

Architecting and Operating a VMware vSphere Metro Storage Cluster

Lee Dilworth, VMware, Inc. Duncan Epping, VMware, Inc.

#vmworldinf

Disclaimer

This session may contain product features that are

currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features
discussed or presented have not been determined.

Architecting and Operating a vSphere Metro Storage Cluster (vMSC)

Lee Dilworth Principal SE VMware (Twitter: @LeeDilworth) Duncan Epping Principal Architect VMware (Twitter: @DuncanYB)

2011 VMware Inc. All rights reserved

Interact!

If you use Twitter, feel free to tweet about this session (#BCO1159) Take pictures and share them on twitter / facebook
Signed copy of the vSphere 5.1 Clustering Deepdive for the best picture

Ask questions!
Signed copy of the vSphere 5.1 Clustering Deepdive for the best question

Blog about it
We would love to read your thoughts, your opinion, design decisions!

vSphere Metro Storage Cluster

Whats This All About - Typical vSphere vMSC Setup

vCenter Server

vMotion

vSphere Cluster
Site A hosts
ESXi ESXi ESXi ESXi ESXi

Site B hosts
ESXi ESXi ESXi

Active / Active Storage

Site A Datastore
6

Site B Datastore

What is a vSphere Metro Storage Cluster

Stretched Cluster Solution Requires:

Storage system that stretches across sites Stretched network across sites

Hardware Compatibility List (HCL) Certified vMSC

iSCSI Metro Cluster Storage FC Metro Cluster Storage

Latency Support Requirements

ESXi management network max supported latency 10

milliseconds Round Trip Time (RTT) Note: 10ms supported with Enterprise+ licenses only (Metro vMotion),
default is 5ms

Synchronous storage replication link is 5 milliseconds RTT

Note: some storage vendors have different support requirements!

CAMPUS

METRO / SYNC

Two Different Architectures (1/2)

Uniform host access configuration

ESXi hosts from both sites are all connected to a storage node in the storage
cluster across all sites. Paths are stretched across distance.

Two Different Architectures (2/2)

Non-Uniform host access configuration

ESXi hosts in each site are connected only to storage node(s) in the same site.
Paths are limited to local site.

10

Architecting a vSphere Metro Storage Cluster

11

Sounds Simple Right?

No, think about the whole solution its NOT just storage vSphere HA is not site aware!

vSphere DRS is not site aware!

vSphere Storage DRS is not site aware!

12

HA & DRS Site Awareness

What they think..

DRS HA

What youve actually got..

DRS HA

?? ? ?

13

Other Network Considerations

Network teams usually dont like the words Stretch and Cluster Network options are changing (OTV, EoMPLS) L3 Routing impacts (and options LISP?) Site-to-Site vMotion handle carefully Co-locate Multi-VM applications Consider application users site affinity affects data flow to! Consider east-west traffic Ingress point to the network? Load balanced / redundant?

14

Will Use Our Environment to Illustrate

Two sites

Four hosts in total

Stretched network Stretched storage One vCenter Server One vSphere HA Cluster

15

Site Awareness Why Should I Care?

VM to storage mapping Operational Simplicity Application Resiliency Site Affinity / Locality matters!

16

Site Awareness Using DRS Affinity

Host Group required per site Consider multi-tiers apps Groups require ongoing management Group dependent VMs
Site A VM group Site B VM group

Site A host group

17

Site B host group

DRS Affinity - Design Considerations

Use the should rules

HA does not violate must therefore avoid for these configurations

18

Site Awareness Using SDRS & Datastore Clusters

Cluster datastores based on

site affinity

Avoid unnecessary site-to-site

migrations

Set SDRS to Manual, take control,

migration *could* impact availability

Align VMs with storage /

site boundary

Group *similar* devices!

19

HA Design Considerations Admission Control

What about Admission Control?

We typically recommend setting it to 50%, to allow full site fail-over Admission control is not resource management Only guarantees power-on

20

HA Design Considerations Isolation Addresses

Isolation Addresses
Specify two, one at each site, using the advanced setting das.isolationaddress

isolation address 01

isolation address 02

21

HA Design Considerations HeartBeat Datastores

Each site needs a heartbeat datastore defined to ensure each

site can update heartbeat region for storage local to that site

With multiple storage systems consider increasing default

from 2 to 4 => 2 per site

22

HA Design Considerations Permanent Device Loss (PDL)

Ensure PDL enhancements are configured

ESXi Hosts - Set disk.terminateVMonPDLDefault to true in
/etc/vmware/settings

Cluster Advanced Option - Set Das.maskCleanShutdownEnabled

to true, in advanced settings

23

HA Design Considerations Split Brain

vSphere 5.0 HA master / slave concept

Default,1 master, responsible for HA restarts If master fails, a new one elected in ~15 seconds On partition there will be TWO masters

24

HA Design Considerations Isolation Response

Isolation response
Configure it based on your infrastructure! We cannot make this decision for you, however

25

Operating a vSphere Metro Storage Cluster

26

Maintaining the Configuration

HA / DRS settings (per-VM) DRS Affinity Group Members VM Dependencies Co-Locate? Restart Priorities (HA) Remember HA doesnt speak vApp (wont respect restart order)

DRS
HA

Should certain VMs be able to roam? Storage Device <-> DRA Affinity Group
Mappings

Storage Device Split Brain / Detachment

rules?

.automate if you can!!!!

27

So What About Automation / Orchestration?

Automation / orchestration is key Automate virtual machine provisioning

Validate virtual machine placement

Validate the VM-Host rules Validate the Datastore cluster

Some vendors offer tools!

28

Failure Scenarios

29

Face Your Fears!

Understand the possibilities Test them Test them again and keeping going until they feel normal!

30

Finding Your Fears

Seek out vendor KB articles Review Impact Tables

HP/Lefthand

Base POC Testing around tables

Start with biggest impact, get
confident with it

KB: 2020097

EMC VPLEX

Easy stuff last

Test with misconfigured VMs

Restart orders unset Incorrect affinity placement Learn to spot configuration drift Automate as much as possible

KB: 2007545

NetApp KB: 2031038

31

Defining Some Failure Terminology

All Paths Down (APD) Aaaarghhhh where has that device gone?
Incorrect storage removal i.e. yanked! Sudden storage failure No time for storage to tell us anything

Permanent Device Loss (PDL) Aaahhhh the device has gone,

OK I understand Much nicer than APD, graceful handing of state change Storage notifies of device state change via SCSI sense code Allows HA to failover VMs

Split Brain Hmmm the other half has disappeared, now what?
Election of second HA master Check heartbeat datastore region

Restart VMs (if needed)

32

More on PDL (And Why Does It Matter?)

Permanent Device Loss is a specific condition issued by the array

though a SCSI sense code

Virtual machines that do IO will be killed Only when disk.terminateVMOnPDLDefault and

das.maskCleanShutdownEnabled has been set to true!
2012-03-14T13:39:25.085Z cpu7:4499)WARNING: VSCSI: 4055: handle 8198(vscsi4:0):opened by wid 4499 (vmm0:fri-iscsi-02) has Permanent Device Loss. Killing world group leader 4491

What about an APD, will this help?

All Paths Down is a different condition. No action will be taken by HA during an APD event.

33

Scenario - Single Host Failure

A normal HA event No network or

datastore heartbeats

Host will be declared

dead

All VMs will be

restarted

Could violate affinity

rules

34

Scenario - Full Compute Failure in One Site

Normal HA event No datastore or

network heartbeats

All virtual machines

will be restarted

Note, max 32
concurrent restarts per host

Sequencing start
up order!

Will violate affinity

rules! (should rule)

35

Side Step Sequencing

One thing to point out with regards to the start up order

1. Agent virtual machines 2. FT secondary virtual machines 3. Virtual Machines configured with a restart priority of high 4. Virtual Machines configured with a medium restart priority 5. Virtual Machines configured with a low restart priority

This is no guarantee, if restart attempt fails HA continues with

the next virtual machine!

36

Scenario - Disk Shelf Failure

No impact on virtual
machines

Instant switch by
storage stack!

Might incur latency

for virtual machines in Frimley

No HA response
required

37

Scenario - Storage Partition

Virtual machines
remained running with no impact!

Remember the affinity

rules

Without affinity rules

this would result in APD condition

Will virtual machines

be restarted on the other site? Network heartbeats!

38

Scenario - Datacenter Partition (1/2)

Virtual machines
remained running with no impact!

Remember the affinity

rules

Without affinity rules

this would result in APD condition

Will virtual machines

be restarted from the other site? Storage not accessible!

39

Scenario - Datacenter Partition (2/2) - Restart of a VM!

But what if affinity

rules were violated?

Your virtual machine

would be available in both sites!

40

Scenario - Loss of Full Datacenter (1/2)

All virtual machines

will be restarted

Note in many cases

requires manual intervention from a storage perspective!

Run DRS when site

returns, to apply affinity rules and balance load!

41

Scenario - Loss of Full Datacenter (2/2)

What if the manual fail-over of storage is slow?

HA retries 5 times by default in ~ 30 minutes HA keeps a compatibility list so it knows where it can restart what The compatibility list contains VM / datastore / portgroup details

In many cases stretched architectures offer a witness

Used to determine the problem!

42

How About Combining vMSC with

43

Site Recovery Manager

Metro vMSC Site(s)

DR Site (SRM)

44

And What About

vCloud Director
Database considerations Multiple vCD cells! NFS share for the cells?

VXLAN
Keep in mind that VXLAN needs Edge Horse shoe! Single Point of Failure

45

Key Takeaways

Design a cluster that meets your needs dont forget Ops!

Understand HA / DRS play key part in your vMSC success Testing is critical, dont just test the easy stuff!

Document process changes, gain operational acceptance

Do not assume it is Next > Next > Finish Ongoing maintenance/checks will be required

Automate as much as you can!

46

Thank You

47

FILL OUT A SURVEY

EVERY COMPLETE SURVEY IS ENTERED INTO DRAWING FOR A $25 VMWARE COMPANY STORE GIFT CERTIFICATE

INF-BCO1159

Architecting and Operating a VMware vSphere Metro Storage Cluster

Lee Dilworth, VMware, Inc. Duncan Epping, VMware, Inc.

#vmworldinf