1

Virtual Machine Level Difference:

ESXi – Hypervisor Level – Comparison:

VMware vCenter Level Differences:

 2

The virtual
Storage vMotion RDM virtual RDM physical machine can
(SvMotion) compatibility mode compatibility mode change hosts Virtual machine Snapshots
ESXi/ESX 4.x Yes Yes Yes Virtual machine must not have
snapshots while performing
SvMotion.
ESXi 5.0 and Yes Yes Yes Yes
later

ESXi 6.0.x

Distributed Switch:-

VMware vSphere Distributed Switch (VDS) provides a centralized interface from which you can configure,
monitor and administer virtual machine access switching for the entire data center. The VDS provides

1. Simplified virtual machine network configuration

2. Enhanced network monitoring and troubleshooting capabilities
3. Support for advanced VMware vSphere networking features

These features are available only with a Distributed Switch:

1. Can shape inbound (RX) traffic.

2. Supports Private VLANs (PVLANs).
3. Provides potential customization of Data and Control Planes

VSphere 5.x provides these improvements to Distributed Switch functionality:

1. Increased visibility of inter-virtual machine traffic through Netflow.

2. Improved monitoring through port mirroring (dvMirror).
3. Support for LLDP (Link Layer Discovery Protocol), a vendor-neutral protocol.
4. The enhanced link aggregation feature provides choice in hashing algorithms and also increases the limit on
number of link aggregation groups.
5. Additional port security is enabled through traffic filtering support.
6. Improved single-root I/O virtualization (SR-IOV) support and 40GB NIC support.

VSphere 6.0 provides these improvements to Distributed Switch functionality:

1. Network IO Control – New support for per virtual machine Distributed vSwitch bandwidth reservations to
guarantee isolation and enforce limits on bandwidth.
2. Multicast Snooping - Supports IGMP snooping for IPv4 packet and MLD snooping for IPv6 packets in
VDS. Improves performance and scale with multicast traffic.
3. Multiple TCP/IP Stack for vMotion - Allows vMotion traffic a dedicated networking stack. Simplifies IP
address management with a dedicated default gateway for vMotion traffic.
 3

Private vLans

Usually you can separate traffic and secure your environment by using VLANS, but private VLANs allows further
segmentation and creation of private groups inside each of the VLAN. By using private VLANs (PVLANs) you are
splitting the broadcast domain into multiple isolated broadcast “subdomains”.

Private VLANs needs to be configured at the physical switch level (the switch must support PVLANs) and also on
the VMware vSphere distributed switch. (Ent. Plus is required).

There are different types of PVLANs:

Primary:

Promiscuous Primary VLAN – Imagine this VLAN as a kind of a router. All packets from the secondary VLANS
go through this VLAN. Packets which also goes downstream and so this type of VLAN is used to forward packets
downstream to all Secondary VLANs.

Secondary:

Isolated (Secondary) – VMs can communicate with other devices on the Promiscuous VLAN but not with other
VMs on the Isolated VLAN.

Community (Secondary) – VMs can communicate with other VMs on Promiscuous and also with those on the
same community VLAN. The graphics shows it all…

So where we configure PVLANs in vSphere?

VDS is a vSphere Enterprise Plus feature, as we need vSphere Distributed Switch (vDS) to configure PVLANs
 4

The next step is to create some PVLANs. You’ll be doing it at the vDS level, so select and right click the
vDS > Edit Settings > Private VLAN tab. Once there you can add some PVLANs. Notice the Secondary
Promiscuous was created automatically when you created the Primary private VLAN.

So in my example above I created Primary Private VLAN 500 which automatically created secondary PVLAN 500.
Then I only could create an Isolated Secondary VLAN 501 and Community VLAN 502.

Now we have those PVLANs created and this gives us the possibility to use them for new or existing port
groups. Example below I’m creating new port group with some name and after selecting the PVLAN, a new drop-
down menu appears which gives the option to choose an entry between the Isolated, or Community.

Private VLANS definitions:

 Promiscuous – A promiscuous port can communicate with all interfaces, including the isolated and
community ports within a PVLAN.
 Isolated – An isolated port has complete Layer 2 separation from the other ports within the same PVLAN, but
not from the promiscuous ports. PVLANs block all traffic to isolated ports except traffic from promiscuous
ports. Traffic from isolated port is forwarded only to promiscuous ports.
 5

 Community– Community ports communicate among themselves and with their promiscuous ports. These
interfaces are separated at Layer 2 from all other interfaces in other communities or isolated ports within their
PVLAN.

Types of port binding:-

These three different types of port binding determine when ports in a port group are assigned to virtual machines:

 Static Binding
 Dynamic Binding
 Ephemeral Binding

Static binding

When you connect a virtual machine to a port group configured with static binding, a port is immediately assigned
and reserved for it, guaranteeing connectivity at all times. The port is disconnected only when the virtual machine is
removed from the port group. You can connect a virtual machine to a static-binding port group only through vCenter
Server.

Note: Static binding is the default setting, recommended for general use.
Dynamic binding
In a port group configured with dynamic binding, a port is assigned to a virtual machine only when the virtual
machine is powered on and its NIC is in a connected state. The port is disconnected when the virtual machine is
powered off or the NIC of the virtual machine is disconnected. Virtual machines connected to a port group
configured with dynamic binding must be powered on and off through vCenter.

Dynamic binding can be used in environments where you have more virtual machines than available ports, but do
not plan to have a greater number of virtual machines active than you have available ports.

Note: Dynamic binding is deprecated from ESXi 5.0, but this option is still available in vSphere Client. It is strongly
recommended to use Static Binding for better performance.

Ephemeral binding

In a port group configured with ephemeral binding, a port is created and assigned to a virtual machine by the host
when the virtual machine is powered on and its NIC is in a connected state. When the virtual machine powers off or
the NIC of the virtual machine is disconnected, the port is deleted.

You can assign a virtual machine to a distributed port group with ephemeral port binding on ESX/ESXi and vCenter,
giving you the flexibility to manage virtual machine connections through the host when vCenter is down. Although
only ephemeral binding allows you to modify virtual machine network connections when vCenter is down, network
traffic is unaffected by vCenter failure regardless of port binding type.

Note: Ephemeral port groups must be used only for recovery purposes when you want to provision ports directly on
host bypassing vCenter Server, not for any other case. This is true for several reasons:

 Scalability

An ESX/ESXi 4.x host can support up to 1016 ephemeral port groups and an ESXi 5.x host can support up
to 256 ephemeral port groups. Since ephemeral port groups are always pushed to hosts, this effectively is
also the vCenter Server limit.
 6

 Performance

Every operation, including add-host and virtual machine power operation, is slower comparatively because
ports are created/destroyed in the operation code path. Virtual machine operations are far more frequent
than add-host or switch-operations, so ephemeral ports are more demanding in general.

 Non-persistent (that is, "ephemeral") ports

Port-level permissions and controls are lost across power cycles, so no historical context is saved.

Note: vSphere 5.0 has introduced a new advanced option for static port binding called Auto Expand. This port group
property allows a port group to expand automatically by a small predefined margin whenever the port group is about
to run out of ports. For vSphere 5.1 and 5.5, the Auto Expand feature is enabled by default. In vSphere 6.0 Auto
Expand is not available.
In vSphere 5.0 Auto Expand is disabled by default

Promiscuous Mode:-

Promiscuous mode is a security policy which can be defined at the virtual switch or port group level in vSphere
ESX/ESXi. A virtual machine, Service Console or VMkernel network interface in a portgroup which allows use of
promiscuous mode can see all network traffic traversing the virtual switch. By default, a guest operating system's
virtual network adapter only receives frames that are meant for it. Placing the guest's network adapter in
promiscuous mode causes it to receive all frames passed on the virtual switch that are allowed under the VLAN
policy for the associated portgroup. This can be useful for intrusion detection monitoring or if a sniffer needs to
analyze all traffic on the network segment. When promiscuous mode is enabled at the portgroup level, objects
defined within that portgroup have the option of receiving all incoming traffic on the vSwitch. Interfaces and virtual
machines within the portgroup will be able to see all traffic passing on the vSwitch, but all other portgroups within
the same virtual switch do not. When promiscuous mode is enabled at the virtual switch level, all portgroups within
the vSwitch will default to allowing promiscuous mode. However, promiscuous mode can be explicitly disabled at
one or more portgroups within the vSwitch, which override the vSwitch-defined default. If software within a virtual
machine is attempting to put the guest network adapter in promiscuous mode, contrary to the defined vSwitch or
portgroup security policy, it may be necessary to investigate if the virtual machine is running undesired software.

Difference between Promiscuous Mode and Netflow and Port mirroring

Port Mirroring has the ability to mirror all the traffic coming in or going out of particular virtual ports on vDS.
Promiscuous Mode cannot forward traffic to a particular port on the virtual switch, instead, promiscuous mode
simply repeats the traffic it receives to any virtual adapter that has entered promiscuous mode.
Promiscuous mode can present a security risk since any VM connected to that port group can capture the traffic.
Netflow capability on a VDS along with a Netflow collector tool helps monitor application flows and measures flow
performance over time and also helps in capacity planning so that I/O resources are utilized properly by different
applications, based on their needs. Netflow provides information about the data flowing across a port to a
monitoring system. Port mirror copies the actual data flowing across a port to another port.
You use port mirroring to troubleshoot network issues and copy some traffic into a destination where you typically
will have Wireshark or similar tool running. The easiest way is to install the network analyze tool into the VM you
want to troubleshoot, but you might not always want or even be allowed to install new code into a production VM.
Being able to listen and capture the traffic from the outside is great in such cases.
 7

Security Settings:-

o Promiscuous mode: Allows virtual adapters connected to this dvPortgroup to see all frames
passed on the host proxy switch that are allowed under the VLAN policy for the dvPortgroup
o Mac address changes: Allows virtual machines to receive frames with a Mac Address that is
different from the one configured in the VMX.
o Forged Transmits: Allows virtual machines to send frames with a Mac Address that is different
from the one specified in the VMX.

Traffic shaping policies:-

o Average Bandwidth: Target traffic rate cap that the switch tries to enforce. Every time a client
uses less than the defined Average Bandwidth, credit builds up.
o Peak Bandwidth: Extra bandwidth available, above the Average Bandwidth, for a short burst.
The availability of the burst depends on credit accumulated so far.
o Burst Size: Amount of traffic that can be transmitted or received at Peak speed. By
combining Peak Bandwidth and Burst Size, you can calculate the maximum allowed time for
the burst.

Vlan Tagging:-

There are 3 types of VLAN tagging available in VSphere.

1. Virtual Switch Tagging (VST)

2. External Switch Tagging (EST)
3.Virtual Guest Tagging (VGT)

There is no specific settings named "VLAN Tagging" is available in the vSphere host network settings. VLAN
tagging is determined by the VLAN value specified at the port group and it tells the vSwitch or Physical switch or
Virtual machines to how to handle the VLAN tagging.

1. Virtual Switch Tagging (VST)

1.1 VST uses 802.1q VLAN trunks and tagged traffic.

1.2 VLAN tagging for all packets is performed by the Virtual Switch before leaving the ESX/ESXI host
1.3 Port groups on the Virtual switch of ESX server should be configured with VLAN ID (1-4094)
1.4 vSwitch responsibility is to strip off the Vlan tag and send packet to virtual machine in corresponding port
group.
1.5 Reduces the number of Physical nics on the server by running all the VLANs over one physical nic. Better
 8

solution would be to keep 2 nics for redundancy.

1.6 Reduces number of cables from ESX server to physical switch.
1.7 The physical switch port connecting the uplink from the ESX should be configured as Trunk port.
1.8 virtual machine network Packet is delivered to vSwitch and before it is sent to physical switch the packet is
tagged with Vlan id according to the port group membership of originating virtual machine.

2. External Switch Tagging (EST)

2.1 In EST, ESX host doesn't see any Vlan tags and does not handle any VLAN tagging.
2.2 All the tagging operation is done by physical switch and virtual switch is not aware about that.
2.3 Number of physical nics = no of VLANs connected to ESX
2.4 Port groups on the Virtual switch of ESX server need not to be configured with the VLAN number or configure
VLAN ID 0 (if it is not native VLAN)
2.5 Count of NICS and cable connected to ESX is more as compared to VST approach.
2.6 The physical switch port connecting the uplink from the ESX should be configured as Access port assigned to
specific VLAN.
2.7 virtual machine network Packet is delivered to physical switch without any tagging operation performed by the
virtual switch
3. Virtual Guest Tagging (VGT)

3.1 you must install 8021.Q VLAN trunking driver inside virtual machine guest operating system.

3.2 All the VLAN tagging is performed by the virtual machine with use of trunking driver in the guest.

3.3 VLAN tags are understandable only between the virtual machine and external switch when frames are passed
to/from virtual switches.

3.4 Virtual Switch will not be involved or aware of this operation. VSwitch only forwards the packets from Virtual
machine to physical switch and will not perform any operation.

3.5 Port group of the virtual machine should be configured with VLAN ID 4095

3.6 The physical switch port connecting the uplink from the ESX should be configured as Trunk port
 9

Difference between vmfs3 and vmfs 5:-

New Unified 1MB File Block Size

Earlier versions of VMFS used 1, 2, 4 or 8MB file blocks. These larger blocks were needed to create large files
(>256GB). These different file blocks sizes are no longer needed to create large files on VMFS-5. Very large files
can now be created on VMFS-5 using the new unified 1MB file blocks. Earlier versions of VMFS will still have to
use larger file blocks to create large files.
Large Single Extent Volumes
In earlier versions of VMFS, the largest single extent was 2TB - 512 bytes. An extent is a partition on which one can
place a VMFS. To create a 64TB VMFS-5, one needed to create 32 x 2TB extents/partitions and join them together.
With VMFS-5, this limit for a single extent/partition has been increased to 64TB.
Smaller Sub-Blocks
VMFS-5 introduces smaller sub-blocks. Sub-blocks are now 8KB rather than 64KB as used in the earlier versions.
With VMFS-5, small files (< 8KB, but > 1KB) in size will consume only 8KB rather than 64KB. This will reduce
the amount of disk space stranded by small files. Also, there are many more sub-blocks in VMFS-5 than there were
in VMFS-3 (32,000 on VMFS-5 compared to approximately 4,000 on VMFS-3).
Small File Support
VMFS-5 introduces support for very small files. For files less than or equal to 1KB, VMFS-5 uses the file descriptor
location in the metadata for storage rather than file blocks. When these files grow beyond 1KB, they will then start
to use the new 8KB sub-blocks.
 10

Increased File Count

VMFS-5 introduces support for greater than 120,000 files, a four-fold increase when compared to the number of
files supported on VMFS-3, which was approximately 30,000.
GPT
VMFS-5 now uses GPT partition table rather that MBR table as used by earlier version of VMFS extending the
maximum partition size to 64TB which was limited to 2TB in earlier versions of VMFS.

GPT stands for GUID Partition Table. It’s a new standard that’s gradually replacing MBR. It’s associated with UEFI
— UEFI replaces the clunky old BIOS with something more modern, and GPT replaces the clunky old MBR
partitioning system with something more modern. It’s called GUID Partition Table because every partition on your
drive has a “globally unique identifier,” or GUID — a random string so long that every GPT partition on earth likely
has its own unique identifier.This system doesn’t have MBR’s limits. Drives can be much, much larger and size
limits will depend on the operating system and its file systems. GPT allows for a nearly unlimited amount of
partitions, and the limit here will be your operating system — Windows allows up to 128 partitions on a GPT drive,
and you don’t have to create an extended partition.

On an MBR disk, the partitioning and boot data is stored in one place. If this data is overwritten or corrupted, you’re
in trouble. In contrast, GPT stores multiple copies of this data across the disk, so it’s much more robust and can
recover if the data is corrupted. GPT also stores cyclic redundancy check (CRC) values to check that its data is intact
— if the data is corrupted, GPT can notice the problem and attempt to recover the damaged data from another
location on the disk. MBR had no way of knowing if its data was corrupted — you’d only see there was a problem
when the boot process failed or your drive’s partitions vanished.

UEFI

VSphere 5.0 supports booting ESXi hosts from the Unified Extensible Firmware Interface (UEFI). With UEFI, you
can boot systems from hard drives, CD-ROM drives, or USB media. Network booting or provisioning with VMware
Auto Deploy requires the legacy BIOS firmware and is not available with UEFI.

UEFI (Unified Extensible Firmware Interface) is a standard firmware interface for PCs, designed to
replace BIOS (basic input/output system). This standard was created by over 140 technology companies as part of
the UEFI consortium, including Microsoft. It's designed to improve software interoperability and address limitations
of BIOS. Some advantages of UEFI firmware include:

 Better security by helping to protect the pre-startup—or pre-boot—process against boot kit attacks.

 Faster startup times and resuming from hibernation.

 Support for drives larger than 2.2 terabytes (TB).

 Support for modern, 64-bit firmware device drivers that the system can use to address more than 17.2 billion
gigabytes (GB) of memory during startup.

 Capability to use BIOS with UEFI hardware.

 11

vNUMA:-

VMware introduced vNUMA (Virtual Non-Uniform Memory Access) in vSphere 5. It is a technology feature that
exposes the underlying NUMA architecture of the hypervisor to the VMs running on it. Assuming that those VMs
run operating systems that are NUMA-aware, they could potentially gain significant performance increases from
seeing the underlying NUMA architecture. In order to fully understand vNUMA and its benefits we must first
explain UMA (Unified Memory Architecture) and NUMA (Non-Uniform Memory Architecture).

UMA

Unified Memory Architecture is a Shared Memory Architecture where all the processors share the same physical
memory uniformly. This configuration is also known as a Symmetric Multi-Processing system or SMP. The graphic
below illustrates the UMA architecture:

As you can see both processors have direct access to the

same memory on the same bus and this access is uniform or equal, meaning that no processor would have a

performance advantage over another when accessing memory addresses. This architecture is suitable for general

purpose time critical applications used by multiple users. It is also suitable for large single programs in time critical

applications. The problem with UMA was that the requirement for much larger server systems resulted in more

processors sharing the same memory bus which increased memory access latency. This consequently impacted

operating system and application performance.

NUMA

NUMA architecture works by linking memory directly to a processor to create a NUMA node. Here, all processors

have access to all memory, but that memory access is not uniform or equal. The graphic below illustrates this:
 12

As you can see each processor has direct access to its own

memory, this is known as local memory. It can also access memory assigned to the other processor, this is known as

remote memory. Access to remote memory is significantly slower than local memory hence why the memory access

is non-uniform. Memory access times are not uniform and depend on the location of the memory and the node from

which it is accessed, as the technology’s name implies. The main benefit of NUMA architecture is memory latency

reduction and application memory performance improvement. To realize these performance benefits the operating

system must be NUMA-aware in order for it to place applications in specific NUMA nodes and to prevent them

from crossing NUMA-node boundaries.

vNUMA
As mentioned earlier, vNUMA exposes the underlying NUMA architecture of the hypervisor to the VMs running on
it. As long as those VMs run operating systems that are NUMA-aware they can make the most intelligent or
efficient use of the underlying processors and memory. VNUMA is a technology feature that was introduced with
vSphere 5. In earlier versions of vSphere, a VM that contained vCPUs spanning multiple physical sockets would
think it was running on a UMA system and would therefore adversely affect its NUMA-aware management features.
This could significantly impact the performance of the VM.With the increased demand for ever larger VMs, we are
now seeing wide or monster VMs becoming the norm in enterprise cloud environments. Especially when those VMs
are running critical workloads. The latest version of vSphere, at the time of writing is it vSphere 6, supports VMs
with 128 vCPUs and 4TB of memory. As VMs move towards these upper limits in terms of size, they will certainly
will span multiple NUMA nodes, this is where vNUMA can significantly improve system and application
performance of these large, high performance VMs.Below are some important points relating to vNUMA:
vNUMA requires VMs to run virtual hardware version 8 or above.

 The hypervisor must run vSphere 5.0 and above.

 The hypervisor must contain NUMA-enabled hardware.
 vNUMA is automatically enabled for VMs with more than 8 x vCPUs (so 9 x vCPUs or more).
 To enable vNUMA on VMs with 8 x vCPUs or less, it must be done manually. It can be set in the VM’s
Configuration Parameters.
 vNUMA will not be run on a VM with vCPU hot plug enabled, in fact, the will use UMA with interleaved
memory access instead.
 A VM’s vNUMA topology is set based on the NUMA topology of the hypervisor it is running on. It retains
the same vNUMA topology of the hypervisor it was started on even if it is migrated to another hypervisor in
the same cluster. This is why it is good practice to build clusters with identical physical hosts/hardware.

VM Sizing
VMware recommends sizing VMs so they align with physical NUMA boundaries. So, if your hypervisor has 8 x
cores per socket (octo-core), the NUMA node is assumed to contain 8 cores. Then size your virtual machines in
multiples of 8 x vCPUs (8 vCPUs, 16 vCPUs, 24 vCPUs, 32 vCPUs etc.).VMware recommends sizing VMs with
 13

the default value of 1 core per socket (with the number of virtual sockets therefore equal to the number of vCPUs).
So in our octo-core server if we require 8 x vCPUs we would configure the VM to have the configuration below:

By changing the configuration to 8 cores per socket, it still aligns correctly with the NUMA node as there are 8

cores:

However, this configuration can result in reduced performance. According to this VMware article, this configuration

is considered to be less than optimal and resulted in significant increases in application execution time. The

conclusion was that the corespersocket configuration of a virtual machine does indeed have an impact on

performance when the manually configured vNUMA topology does not optimally match the physical NUMA

topology. If VMs were sized incorrectly and did not match the underlying NUMA topology it would result in

reduced performance. So using our example where the hypervisor has 8 cores per socket, if a VM with 10 virtual

sockets (a total of 10 cores) were created, it would breach the NUMA boundary. 8 of the cores would come from an

assigned NUMA node and 2 would come from another NUMA node. This would result in reduced performance

as applications would be forced to access remote memory thereby incurring a performance hit. How much the

performance hit is depends on factors unique to that specific VM.

 14

VMware vSphere API for Array Integration (VAAI):-

VMware vSphere API for Array Integration (VAAI) - allows for hardware acceleration and offload certain

operations that originally occurs at the host level, to the storage system. With the storage hardware assistance, your

host performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth.

VAAI requires:

 ESXi/ESX 4.1 or later (with vSphere 4.1 NAS is not supported. NAS is supported since vSphere 5.0).
 Enterprise or Enterprise Plus licensing for ESXi/ESX hosts.
 Storage arrays that support VAAI storage-based hardware acceleration.

There are following fundamentals operations used in VAAI:

 Atomic Test & Set (ATS), which is used during creation and locking of files on the VMFS volume.
 Clone Blocks/Full Copy/XCOPY, which is used to copy or migrate data within the same physical array.
 Zero Blocks/Write same, which is used to zero-out disk regions.
 15

 Thin Provisioning in ESXi 5.x and later hosts, which allows the ESXi host to tell the array when the space
previously occupied by a virtual machine (whether it is deleted or migrated to another datastore) can be
reclaimed on thin provisioned LUNs.
 Block Delete in ESXi 5.x and later hosts, which allows for space to be reclaimed using the SCSI UNMAP
feature.
 Full File Clone – Like the Full Copy VAAI primitive provided for block arrays, this Full File Clone primitive
enables virtual disks to be cloned by the NAS device.
 Native Snapshot Support – Allows creation of virtual machine snapshots to be offloaded to the array.
 Extended Statistics – Enables visibility to space usage on NAS datastores and is useful for Thin Provisioning.
 Reserve Space – Enables creation of thick virtual disk files on NAS.

The VAAI limitations are as follow:

VAAI cannot be used when:

 The source and destination VMFS volumes have different block sizes
 The source file type is RDM and the destination file type is non-RDM (regular file)
 The source VMDK type is eagerzeroedthick and the destination VMDK type is thin
 The source or destination VMDK is any kind of sparse or hosted format
 Cloning a virtual machine that has snapshots because this process involves consolidating the snapshots into
the virtual disks of the target virtual machine.
 The logical address and/or transfer length in the requested operation is not aligned to the minimum alignment
required by the storage device (all datastores created with the vSphere Client are aligned automatically)
 The VMFS datastore has multiple LUNs/extents spread across different arrays
To check if you storage supports VAAI please refer to the VMware compatibility chart tool found here
 16

When your storage supports VAAI, you use vSphere Client and go to Host > Configuration > Storage, you can
see the Hardware Acceleration Status in the panel on the right side.

If you use Web Client (recommended :)), go to Datastore > Manage > Settings > General and checkDatastore
Capabilities:

For each storage device and datastore, it's displayed the hardware acceleration support status in the Hardware
Acceleration column of the Devices view and the Datastores view. The status values are Unknown, Supported,
and Not Supported. The initial value is Unknown. The status changes to Supported after the host successfully
performs the offload basic operations. If the offload operation fails, the status changes to Not Supported.

To determine if your storage device supports VAAI, test the Full Copy VAAI primitive:
1. Using the vSphere Client, browse the datastore and locate a virtual disk (VMDK) of at least 4 MB that is not
in use.
2. Copy the virtual disk to a new file.
3. Check the Hardware Acceleration status to verify that it changes from Unknown to either Supported or Not
Supported.
 17

To check exactly supported VAAI features by datastore, please run command (vSphere 5.x) via SSH:

Esxcli storage core device vaai status get

VMware vSphere API for Storage Awareness (VASA)

VMware vSphere API for Storage Awareness (VASA) - is a set of APIs that permits storage arrays to integrate with
vCenter for management functionality:

 Displays the features of the physical storage devices.

 Provides storage health status, configuration info & capacity.

Depending on storage and VASA provider, the attributes provided to vCenter can be as follow:

 Disk Type (SSD, SATA, SAS, FC)

 Snap space reserved
 Dedupe - Indicates space efficiency on the underlying NetApp LUN or a volume & Replication – Indicates
that the underlying NetApp storage is configured for SnapMirror or SnapVault.
 VV Type – VirtualCopy, PhysicalCopy, Base & Remotecopy – InRemotecopy, NotInRemotecopy -
HP storage and more.

ESXI Commands:-

services.sh – While Linux services are normally handled using the services command, ESXi services are handled
much the same way utilizing theservices.sh command. Services.sh can be passed with a stop, start, or restart flag to
perform the respected operation on all ESXi services.

services.sh restart – restart all ESXi services.

 18

/etc/init.d – The scripts located in /etc/init.d can be used to start/stop their respective services one at a time. If you
just wanted to restart the vCenter Server Agent (also known as the vpxa service), you could run/etc/init.d/vpxa
restart to restart it. On the other hand, services.sh restartwould restart all services.

/etc/init.d/vpxa restart – restart vCenter Agent on host

/etc/init.d/hostd restart
/etc/init.d/vpxa restart

cat /etc/chkconfig.db – view current running status of all ESXi services.

vmkping – We are all familiar with the functionality of the age old 'ping'command. But, vmkping takes this one step
further and allows you to use the IP stack of the VMkernel to send Internet Control Message Protocol (ICMP)
packets through specified interfaces. Meaning, you could send a ping packet out through the vMotion network,
rather than over the management network.

vmkfstools – If you ever need to manage VMFS volumes and virtual disks via the command line, then vmkfstools is
the command for you. The vmkfstools command allows you to create, clone, extend, rename and delete VMDK files.
In addition to the virtual disk options, you can also create, extend, grow and reclaim blocks from our file systems
with vmkfstools.

vmkfstools –i test.vmdk testclone.vmdk – clones test.vmdk to testclone.vmdk

esxtop – When it comes to performance monitoring and troubleshooting on an ESXi host, few tools can give you as
much information as esxtop. With similar functionality to the Linux top command, esxtop goes one step further by
gathering VMware-specific metrics as they compare to CPU, interrupt, memory, network, disk adapter, disk device,
disk VM and power management.

vim-cmd – Vim-cmd is a command space that is built over top of the hostd process, allowing the end user to script
and command almost every vSphere API. Vim-cmd has a number of sub ESXi commands dealing with different
portions of the virtual infrastructure and is very easy to use compared to its counterpart, vimsh.

esxcli hardware – The hardware namespace of esxcli can prove extremely useful when you are looking to get
information about the current hardware and setup of your ESXi host

esxcli hardware cpu list – retrieve CPU information (family, model and cache).
 19

esxcli hardware memory get – retrieve information about memory (available and non-uniform memory access).

esxcli iscsi – The iscsi namespace can be used to monitor and manage both hardware and software iSCSI setups.

esxcli iscsi software – can be used to enabled/disable the software iSCSI initiator.

esxcli iscsi adapter – can be used to setup discovery, CHAP and other settings for both your hardware and software
iSCSI adapters.

esxcli iscsi sessions – can be used to list established iSCSI sessions on the host.

esxcli network – The network namespace of esxcli is extremely valuable when looking to monitor and make changes
to anything and everything dealing with vSphere networking, including virtual switches, VMKernel network
interfaces, firewalls and physical network interface cards (NICs).

esxcli network nic – list and modify NIC information, such as name, wake on LAN, and speeds.

esxcli network vm list – list networking information about your VMs that have an active network port.

esxcli network vswitch – Commands to retrieve and manipulate options on VMware's standard and distributed
virtual switches.

esxcli network ip – Commands to manage VMkernel ports, including management, vMotion and Fault Tolerance
networks. Also contains the ability to modify any of the IP stack in regard to the host, including DNS,IPsec and
routing information.

esxcli software – The software namespace can be used to retrieve and install different pieces of software and drivers
on your ESXi host.

esxcli software vib list – list the software and drivers currently installed on the ESXi host

esxcli storage – This is perhaps one of the most used esxcli command namespaces and contains everything and
anything you need in order to manage the core storage attached to vSphere.

esxcli storage core device list – list the current storage devices

esxcli storage core device vaai status get –get the current status of VAAIsupport on your storage devices.
 20

esxcli system – This command gives you the ability to control ESXi advanced options, such as setting up syslog and
managing host status.

esxcli system maintenanceMode set –enabled yes/no – set the host into maintenance mode.

esxcli system settings advanced – View and change ESXi advanced settings (Hint: Use esxcli system settings
advanced list –d to view the settings that deviate from the default).

esxcli system syslog – Syslog information and configuration

esxcli vm – The VM namespace of ESXi can be used to list out various tidbits of information about the VMs
running on the host and shut them down forcibly if needed.

esxcli vm process list – List out process information for powered on VMs

esxcli vm process kill – Terminate running VM process, essentially shutting down or forcibly powering off a VM.

esxcli vsan – The VSAN namespace of ESXi contains a ton of commands dealing with VSAN setup and
maintenance, including data store, network, fault domain, and policy configuration.

esxcli vsan storage – commands for configuring local storage for use with VSAN, including adding and removing
physical disks and modifying auto claim.

esxcli vsan cluster – commands to leave/join VSAN clusters on the local host.

esxcli esxcli – That's right! The esxcli command has a namespace called "esxcli." By using the esxcli namespace,
you are able to get further information on any or all the commands that lie within the esxcli utility.esxcli esxcli
command list – list out every esxcli command on the system along with the functions it provides.

Run the esxcli storage core path list command to generate a list of all LUN paths currently connected to the ESXi
host.

You see output similar to:

fc.5001438005685fb5:5001438005685fb4-fc.50060160c46036df:50060167446036df-
naa.6006016094602800e07ff528b73ae011
UID: fc.5001438005685fb5:5001438005685fb4-
 21

fc.50060160c46036df:50060167446036df-naa.6006016094602800e07ff528b73ae011
Runtime Name: vmhba0:C0:T0:L23
Device: naa.6006016094602800e07ff528b73ae011
Device Display Name: DGC Fibre Channel Disk ( naa.6006016094602800e07ff528b73ae011)
Adapter: vmhba0
Channel: 0
Target: 0
LUN: 23
Plugin: NMP
State: active
Transport: fc
Adapter Identifier: fc.5001438005685fb5:5001438005685fb4
Target Identifier: fc.50060160c46036df:50060167446036df
Adapter Transport Details: WWNN: 50:01:43:80:05:68:5f:b5 WWPN:
50:01:43:80:05:68:5f:b4
Target Transport Details: WWNN: 50:06:01:60:c4:60:36:df WWPN:
50:06:01:67:44:60:36:df
 22

vim-cmd vmsvc/getallvms List all VMs running on the host. Also provides vmid,
required for commands below.

vim-cmd vmsvc/power.off vmid Power off specified VM.

vim-cmd vmsvc/power.on vmid Power off specified VM.

vim-cmd vmsvc/power.reboot vmid Reboot specified VM.

vim-cmd solo/registervm /vmfs/volume/datastore/subdir/vm- Register the VM stored at location on the ESX host inventory.
file.vmx

vim-cmd vmsvc/unregister vmid Unregister VM from the host. Does not remove the VM's files
from the datastore.

vim-cmd vmsvc/destroy vmid Delete the specified VM. The VMDK and VMX files will be
deleted from storage as well.

vim-cmd vmsvc/tools.install vmid Initiates an installation of VMWare Tools on the VM

vim-cmd hostsvc/maintenance_mode_enter Put the host into maintenance mode.

vim-cmd hostsvc/maintenance_mode_exit Take the host out of maintenance mode.

vim-cmd hostsvc/net/info Show networking information of the host.

chkconfig -l Show services running on the host. Can also be used to

change startup configuration.

esxtop Display list of processes and its usage of resources. Works

similar to linux top.

esxcfg-info Show host's configuration and information.

esxcfg-nics -l Show current NIC configuration.

esxcfg-vswitch -l Show current vSwitch configuration.

vmkerrcode -l Display a reference list of VMKernel return codes and

descriptions.

dcui Start the console UI (when accessing through SSH).

vsish Run the VMWare Interactive Shell (from SSH).

decodeSel /var/log/ipmi_sel.raw Read IPMI system log of physical server.

Esxtop

Although understanding all the metrics esxtop provides seem to be impossible using esxtop is fairly simple. When
you get the hang of it you will notice yourself staring at the metrics/thresholds more often than ever. The following
keys are the ones I use the most.

Open console session or ssh to ESX(i) and type:

 23

esxtop

By default the screen will be refreshed every 5 seconds, change this by typing:

s2

Changing views is easy, type the following keys for the associated views:

c = cpu
m = memory
n = network
i = interrupts
d = disk adapter
u = disk device (includes NFS as of 4.0 Update 2)
v = disk VM
p = power states

V = only show virtual machine worlds

e = Expand/Rollup CPU statistics, show details of all worlds associated with group (GID)
k = kill world, for tech support purposes only!
l = limit display to a single group (GID), enables you to focus on one VM
# = limiting the number of entitites, for instance the top 5

2 = highlight a row, moving down

8 = highlight a row, moving up
4 = remove selected row from view
e = statistics broken down per world
6 = statistics broken down per world

Memory reclamation

Memory reclamation, as the term suggests, is a method of reclaiming memory that has been assigned to VMs.

ESXi assigns memory resources to VMs according to the amount of memory those VMs have been configured to

use. If ESXi has free or spare memory after satisfying all VMs’ configured memory demands then there is no need

to reclaim memory. Memory reclamation only comes into play when the host begins to runs out of physical memory

and cannot allocate any more to VMs.

There are 4 memory reclamation techniques in total:

 Transparent Page sharing

 Ballooning

 Memory compression

 Hypervisor swapping
 24

Ballooning and Hypervisor swapping are dynamic in that they expand or contract the amount of memory allocated

to VMs based on the amount of free memory on the host.

TPS (Transparent Page Sharing)

In a typical virtual environment, it is very likely that a high proportion of VMs will be running the same operating

system. They would therefore load the same pages of data into memory. In such situations a hypervisor will employ

TPS to store just a single copy of the identical pages and securely eliminates those redundant copies of memory

pages – it is basically memory deduplication. TPS results in reduced host memory consumption by the VMs. TPS is

enabled by default and its effectiveness depends on the amount of identical pages that can be identified in memory

and subsequently reclaimed. TPS can save you up to 70% (VDI environments with many identical operation

systems), it is transparent to the VMs and incurs no performance penalty. With TPS, a workload running within a

VM often consumes less memory than it would when running on a physical machine.

Ballooning

Ballooning is a dynamic memory reclamation technique that is leveraged to reclaim memory pages when host

memory resources are in demand and available physical pages cannot meet requirements. A memory balloon driver

(vmmemctl) is loaded into the guest OS running within a VM. The memory balloon driver communicates directly

with the host and is made aware of it’s low memory status. The guest OS is neither aware of this communication nor

the low memory status of the host. The host will indicate the amount of physical pages it needs to reclaim, this

determines the balloon size. The balloon driver then inflates to the required size by allocating the required amount of

memory within the guest OS that is required by the host. It pins the allocated pages, so that the guest OS doesn’t

page or swap them out. The balloon driver then informs the host which pages it has allocated, the host then finds the

physical memory that backs the pages that have been pinned within the guest OS by the balloon driver and uses

them to satisfy memory demands of other running workloads. Ballooning creates artificial memory pressure with the

guest OS, which can force it to leverage its own native memory management techniques to handle these changing

conditions. Once memory pressure within the host has reduced, it will return those pages back to the guest OS. It

does this by deflating the balloon driver. VMware Tools contains the balloon driver and must be installed for

ballooning to work on a VM.

 25

Memory compression

Memory compression is employed when the host is suffering from memory pressure and ballooning has already

been used. It is enabled by default. It compresses and stores memory in a cache in the host’s main memory. ESXi

checks whether a page can be compressed by checking its compression ratio. Memory is compressed if a page’s

compression ratio is greater than 50%. Otherwise, no performance benefit is derived from compressing it and that

page is swapped out. As compressed memory resides in memory and can be retrieved by decompression, it is still

much faster than swapping which is slower because it involves writing pages to a file (hypervisor swapping).

Memory compression can significantly improve application performance when the host is under memory pressure.

Hypervisor swapping

Swapping is also a dynamic memory reclamation technique. It must be made clear that this is hypervisor swapping

and not guest OS swapping. When the other memory reclamation techniques have failed to combat memory pressure

on the host, swapping is used as a last resort. In this technique, the VM’s swap file, which is created when the VM is

powered on, is used to swap out memory from the VM in order to reclaim memory from it. The reason it is used as a

last resort is because swapping memory to a file is very slow and results in a significant performance impact. If you

have reached a point whereby hypervisor swapping is being utilized you need to fix the problem fast and order more

physical memory!

VC-5.1

Minimum hardware requirements for vCenter Server

vCenter Server Requirement

hardware

CPU Two 64-bit CPUs or one 64-bit dual-core processor.

Processor 2.0 GHz or faster Intel 64 or AMD 64 processor. The Itanium (IA64) processor is not supported.

Processor requirements may be higher if the database runs on the same machine.

Memory The amount of memory needed depends on your vCenter Server configuration.

 If vCenter Server is installed on a different host machine than vCenter Single Sign-On
and vCenter Inventory Service, 4 GB of RAM are required.
 26

 If vCenter Server, vCenter Single Sign-On, and vCenter Inventory Service are installed
on the same host machine (as with vCenter Simple Install), 10 GB of RAM are required.

Memory requirements may be higher if the vCenter Server database or vCenter Single Sign-On
database runs on the same machine as vCenter Server.

vCenter Server includes several Java services: VMware Virtual Center Management Web services
(tc Server), Inventory Service, and Profile-Driven Storage Service. When you install vCenter
Server, you select the size of your vCenter Server inventory to allocate memory for these services.
The inventory size determines the maximum JVM heap settings for the services. You can adjust
this setting after installation if the number of hosts in your environment changes. See the
recommendations in JVM heap settings for vCenter Server.
Disk storage The amount of disk storage needed depends on your vCenter Server configuration.

 If vCenter Server is installed on a different host machine than vCenter Single Sign-On
and vCenter Inventory Service, 4 GB are required.
 If vCenter Server, vCenter Single Sign-On, and vCenter Inventory Service are installed
on the same host machine (as with vCenter Simple Install), at least 40-60 GB of free disk
space are required after installation, depending on the size of your inventory. 100 GB are
recommended to allow for future growth of your inventory.

Disk storage requirements are higher if the vCenter Server database or vCenter Single Sign-On
database runs on the same machine as vCenter Server, depending on the size of those databases.

In vCenter Server 5.x, the default size for vCenter Server logs is 450 MB larger than in vCenter
Server 4.x. Make sure the disk space allotted to the log folder is sufficient for this increase.
Microsoft SQL Up to 2 GB free disk space to decompress the installation archive. Approximately 1.5 GB of these
Server 2008 R2 files are deleted after the installation is complete.
Express disk
storage

Network speed 1 Gbps

VC-5.5

Minimum hardware requirements for vCenter Server

vCenter Server Requirement

Hardware

CPU Two 64-bit CPUs or one 64-bit dual-core processor.

Processor 2.0 GHz or faster Intel 64 or AMD 64 processor. The Itanium (IA64) processor is not supported.
Processor requirements might be higher if the database runs on the same machine.

Memory The amount of memory needed depends on your vCenter Server configuration.
 27

 If vCenter Server is installed on a different host machine than vCenter Single Sign On
and vCenter Inventory Service, 4 GB of RAM are required.
 If vCenter Server, vCenter Single Sign On and vCenter Inventory Service are installed
on the same host machine (as with vCenter Simple Install), 12 GB of RAM are required.

Memory requirements are higher if the vCenter Server database or vCenter Single Sign On
database runs on the same machine as vCenter Server.

vCenter Server includes several Java services: VMware VirtualCenter Management Webservices
(tc Server), Inventory Service, and Profile-Driven Storage Service. When you install vCenter
Server, you select the size of your vCenter Server inventory to allocate memory for these
services. The inventory size determines the maximum JVM heap settings for the services. You
can change this setting after installation if the number of hosts in your environment changes. For
more information, see the recommendations in JVM Heap Settings for vCenter Server.

Disk storage The amount of disk storage needed for the vCenter Server installation depends on your vCenter
Server configuration.

 If vCenter Server is installed on a different host machine than vCenter Single Sign On
and vCenter Inventory Service, 4 GB are required.
 If vCenter Server, SSO, and vCenter Inventory Service are installed on the same host
machine (as with vCenter Simple Install), at least 40-60 GB of free disk space are
required after installation, depending on the size of your inventory. 100 GB are
recommended, to allow for future growth of your inventory.

You should provide more space to allow for future growth of your inventory. For
guidelines about the disk space required for vCenter Single Sign-On and Inventory
Service, see:

o Minimum hardware requirements for vCenter Single Sign-On, running on a
separate host machine from vCenter Server
o Minimum Hardware Requirements for vCenter Inventory Service, running on a
separate host machine from vCenter Server

Disk storage requirements are higher if the vCenter Server database runs on the same machine as
vCenter Server, depending on the size of those databases.

In vCenter Server 5.x, the default size for vCenter Server logs is 450 MB larger than in vCenter
Server 4.x. Ensure the disk space allotted to the log folder is sufficient for this increase.
Microsoft SQL Up to 2 GB free disk space to decompress the installation archive. Approximately 1.5 GB of these
Server 2008 R2 files are deleted after the installation is complete.
Express disk

Network speed 1 Gbps

Note: Installing vCenter Server on a network drive or USB flash drive is not supported.
 28

Migration 5.1 to 5.5

What is SSO..?

vCenter Single Sign-On is a new feature of vSphere 5.1 that is not just an authentication broker but also a security
token exchange providing a more secure way of accessing your vSphere solutions. What that means is that when you
previously logged into vCenter Server, you were authenticated with the provided username and password against the
Active Directory configured for vCenter Server. With vSphere 5.1 and vCenter Single SIgn-On you no longer log
directly into vCenter Server but with a security domain defined by your vSphere environment. When logging in to
vSphere 5.1 you actually pass authentication to the vCenter Single Sign-On server which can be configured with
multiple identity sources like Active Directory and OpenLDAP and on successful authentication, your username and
password is exchanged for a security token which is then used to access the vSphere components like vCenter
Server and vCenter Orchestrator etc.

We run through upgrading:

 Single Sign On
 Inventory Service
 vCenter Server
 Web Client
 vSphere Client
 Update Manager
 ESXi host

VI client Vs Web client

vSphere Web
Client Client

Connect direct to an ESXi Host

Custom Attributes
Tags

Maps
Single Sign On

Work In Progress

Pre-emptive Searching
Save Searches
vCenter Orchestrator
Workflow Integration
VM Hardware Version 9
 29

vSphere Update Manager

vSphere Data Recovery (VDR)

vSphere Data Protection (VDP)

vSphere Replication

Log Browser
Virtual Distributed Switch
Configuration Export
Virtual Distributed Switch
Health Checks
Virtual Distributed Switch
Elastic Port Groups
VXLAN Networking
Enhanced vMotion
(shared nothing)
vSAN Configuration
Upload Files to vSAN Datastore

Big Data Extensions

vSphere Flash Read Cache

VM Hardware Version 10

Manage >2TB VMDK

Roles and Permissions

• Privilege — The ability to perform a specific action or read a specific property. Examples include
powering on a virtual machine and creating an alarm.

• Role — A collection of privileges. Roles provide a way to aggregate all the individual privileges that
are required to perform a higher-level task, such as administer a virtual machine.

• Object — An entity upon which actions are performed. VirtualCenter objects are datacenters, folders,
resource pools, clusters, hosts, and virtual machines.

• System roles – System roles are permanent and the privileges associated with these roles cannot be
changed. The three system roles are: No Access, Read-Only, and Administrator. • Sample roles – Sample
roles are provided for
 30

The first small partition is just for booting the system and locating the hypervisor image which could be on one of
the next two partitions.

The actual system image is located on the first 250 MB partition, formatted with plain old FAT. The image
itself, s.v00, is a 124 MB compressed file, which is decompressed on boot and contains the hypervisor operating
system.The next 250 MB FAT partition is used for inplace upgrades and is empty at start if a clean installation was
done.
If the hypervisor itself should crash (“purple screen of death“) the 110 MB core dump partition is used for dumping
crash information.
In this 286 MB partition we have mostly ISO files with the VMware Tools for the various supported guest operating
system. We can also find the floppy images for the PVSCSI virtual disk controller, if in need to use these directly at
installation on Windows Server guests.
If installing ESXi on a local hard drive and this drive is larger than about 5 GB a “scratch” partition will be created.
This partition holds mostly the various log files for the vmkernel and other components. If installing on a very small
drive, about 1 GB, or on USB, this scratch partition will be missing and the log files will be in RAM only. This
 31

means that in the case of power failure the log files will be lost.
A partition formatted with VMFS will be automatically created on the rest of the space of the install drive. In ESXi
5.0 the new VMFS 5 will be used with a standard block size of 1 MB no matter the size of the disk and better
support for small files, e.g. VMX configuration and log files. When now using GPT (Guided Partition Table) we
could also use disks with sizes over 2 TB.
If using the ESXi Shell you can access the different partitions through file system mounts. For example, the
directory /bootbank is the first 250 MB partition where the actual hypervisor files are located. The /store directory is
the 286 MB large repository for VMware Tools and other files.

The compressed image in the first 250 MB “boot bank” partition is larger in 5.0 than in 4.1, (where the image was
70 MB), but still a very small disk foot print with about 124 MB in 5.0. If you later update the active image it will
first be copied to the alternative boot bank, as a “last known good configuration”.
If the update should fail you could press SHIFT + R early in the boot sequence and select to boot the system from
the alternative boot bank.

Files in a VM

Extension File Name Description

.log <vmname>.log or vmware.log This is the file that keeps a log of key VMware Workstation
activity. This file can be useful in troubleshooting if you encounter
problems. This file is stored in the directory that holds the
configuration (.vmx) file of the virtual machine.

.nvram <vmname>.nvram Or nvram This is the file that stores the state of the virtual machine's BIOS.
 32

.vmdk <vmname>.vmdk This is a virtual disk file, which stores the contents of the virtual
machine's hard disk drive.

A virtual disk is made up of one or more.vmdk files. If you have

specified that the virtual disk should be split into 2GB chunks, the
number of .vmdk files depends on the size of the virtual disk. As
data is added to a virtual disk, the .vmdk files grow in size, to a
maximum of 2GB each. (If you specify that all space should be
allocated when you create the disk, these files start at the maximum
size and do not grow.) Almost all of a .vmdk file's content is the
virtual machine's data, with a small portion allotted to virtual
machine overhead.
If the virtual machine is connected directly to a physical disk, rather
than to a virtual disk, the .vmdk file stores information about the
partitions the virtual machine is allowed to access.
Earlier VMware products used the extension.dsk for virtual disk
files.

<diskname>-<###>.vmdk This is a redo-log file, created automatically when a virtual machine

has one or more snapshots. This file stores changes made to a
virtual disk while the virtual machine is running. There may be
more than one such file. The ### indicates a unique suffix added
automatically by VMware Workstation to avoid duplicate file
names.

.vmem <uuid>.vmem The virtual machine's paging file, which backs up the guest main
memory on the host file system. This file exists only when the
virtual machine is running, or if the virtual machine has crashed.

<snapshot_name_and_number> Each snapshot of a virtual machine that is powered on has an

associated .vmem file, which contains the guest's main memory,
saved as part of the snapshot.

.vmsd <vmname>.vmsd This is a centralized file for storing information and metadata about
snapshots.

.vmsn <vmname>-Snapshot.vmsn This is the snapshot state file, which stores the running state of a
virtual machine at the time you take that snapshot

<vmname>- This is the file which stores the state of a snapshot

Snapshot<###>.vmsn

.vmss <vmname>.vmss This is the suspended state file, which stores the state of a
suspended virtual machine
 33

.Some earlier VMware products used the extension .std for

suspended state files

.vmtm <vmname>.vmtm This is the configuration file containing team data.

.vmx <vmname>.vmx This is the primary configuration file, which stores settings chosen
in the New Virtual Machine Wizard or virtual machine settings
editor. If you created the virtual machine under an earlier version of
VMware Workstation on a Linux host, this file may have
a .cfg extension

.vmxf <vmname>.vmxf This is a supplemental configuration file for virtual machines that
are in a team. Note that the .vmxf file remains if a virtual machine is
removed from the team.

VMDK Files

All virtual disks are made up of two files, a large data file equal to the size of the virtual disk and a small text disk
descriptor file

 The –.vmdk file

the descriptor file describes the size and geometry of the virtual disk file. The descriptor file also contains a
pointer to the large data file as well as information on the virtual disks drive sectors, heads, cylinders and disk
adapter type.
 The –flat.vmdk file
this is the virtual disk data file that is created when you add a virtual hard drive to your VM that is not an
RDM. One of these files is created for each virtual hard drive that a VM has configured. The size will vary
based on the maximum size of the disk, and the type of provisioning used (i.e. thick or thin)
 The –delta.vmdk file
these virtual disk data files are only used when snapshots are created of a virtual machine. When a snapshot is
created, all writes to the original –flat.vmdk are halted and it becomes read-only; changes to the virtual disk
are then written to these –delta files instead. A delta file will be created for each snapshot that you create for a
VM and their file names will be incremented numerically (i.e., myvm01-delta.vmdk, myvm-02-delta.vmdk).
These files are automatically deleted when the snapshot is deleted after they are merged back into the original
–flat.vmdk file.
 The -rdm.vmdk file
this is the mapping file for the RDM that manages mapping data for the RDM device. The mapping file is
presented to the ESXi host as an ordinary disk file, available for the usual file system operations. However, to
the virtual machine the storage virtualization layer presents the mapped device as a virtual SCSI device. The
metadata in the mapping file includes the location of the mapped device (name resolution) and the locking
state of the mapped device. If you do a directory listing you will see that these files will appear to take up the
same amount of disk space on the VMFS volume as the actual size of the LUN that it is mapped to, but in
reality they just appear that way and their size is very small. One of these files is created for each RDM that is
created on a VM.
 34

Thick Provision Lazy Zeroed

1. Creates a virtual disk in a default thick format.
2. Space required for the virtual disk is allocated when the virtual disk is created.
3. Data remaining on the physical device is not erased during creation, but is zeroed out on demand at a
later time on first write from the virtual machine.
4. Using the default flat virtual disk format does not zero out or eliminate the possibility of recovering
deleted files or restoring old data that might be present on this allocated space.
5. You cannot convert a flat disk to a thin disk.

Thick Provision Eager Zeroed

1. A type of thick virtual disk that supports clustering features such as Fault Tolerance.
2. Space required for the virtual disk is allocated at creation time.
3. In contrast to the flat format, the data remaining on the physical device is zeroed out when the virtual
disk is created.
4. It might take much longer to create disks in this format than to create other types of disks.

Thin Provision
1. It provides on on-demand allocation of blocks of data.
2. All the space allocated at the time of creation of virtual disk is not utilized on the hard disk, rather only
the size with utilized data is locked and the size increases as the amount of data is increased on the disk.
3. With thin provisioning, storage capacity utilization efficiency can be automatically driven up towards
100% with very little administrative overhead.

How does vMotion works?

VMotion steps, at a high level (vSphere 4.1):
1. Shadow VM created on the destination host.
2. Copy each memory page from the source to the destination via the vMotion network. This is known as
preCopy.
3. Perform another pass over the VM’s memory, copying any pages that changed during the last preCopy iteration.
4. Continue this iterative memory copying until no changed pages (outstanding to be-copied pages) remain.
5. Stun the VM on the source and resume it on the destination.
When the preCopy cannot converge, vMotion needs to decide whether to fail the vMotion or to proceed with
switchover to the destination anyway. It makes this decision by estimating the time required to transmit all the
remaining outstanding pages. By default, if this time is below 100 seconds vMotion will proceed with the
switchover. If it will take more than 100 seconds the vMotion will fail (timeout) with no impact on the VM.

In the event the VM passes the 100 second check, VMotion will stun the source and start running on the
destination. While the destination runs, the source will transmit the remaining pages to the destination using the
“quick resume” capability introduced with vSphere 4.1.

Live migration of a virtual machine from one physical server to another with VMware VMotion is enabled by three
underlying technologies.

First, the entire state of a virtual machine is encapsulated by a set of files stored on shared storage such as Fibre
Channel or iSCSI Storage Area Network (SAN) or Network Attached Storage (NAS). VMware vStorage VMFS
allows multiple installations of VMware ESX® to access the same virtual machine files concurrently.

Second, the active memory and precise execution state of the virtual machine is rapidly transferred over a high speed
network, allowing the virtual machine to instantaneously switch from running on the source ESX host to the
 35

destination ESX host. VMotion keeps the transfer period imperceptible to users by keeping track of on-going
memory transactions in a bitmap. Once the entire memory and system state has been copied over to the target ESX
host, VMotion suspends the source virtual machine, copies the bitmap to the target ESX host, and resumes the
virtual machine on the target ESX host. This entire process takes less than two seconds on a Gigabit Ethernet
network.

Third, the networks being used by the virtual machine are also virtualized by the underlying ESX host, ensuring that
even after the migration, the virtual machine network identity and network connections are preserved. VMotion
manages the virtual MAC address as part of the process. Once the destination machine is activated, VMotion pings
the network router to ensure that it is aware of the new physical location of the virtual MAC address.

Since the migration of a virtual machine with VMotion preserves the precise execution state, the network identity,
and the active network connections, the result is zero downtime and no disruption to users.

5.0 vmotion enhancements:-

There are some fundamental changes when it comes to vMotion scalability and performance in vSphere 5.0. Most of
these changes have one common goal: being able to vMotion ANY type of workload. It doesn’t matter if you have a
virtual machine with 32GB of memory that is rapidly changing memory pages any more with the the following
enhancements:

1. Multi-NIC vMotion support

2. Stun During Page Send (SDPS)

Multi-NIC vMotion Support

One of the most substantial and visible changes is multi-NIC vMotion capabilities. vMotion is now capable of using
multiple NICs concurrently to decrease the amount of time a vMotion takes. That means that even a single vMotion
can leverage all of the configured vMotion NICs. Prior vSphere 5.0, only a single NIC was used for a vMotion
enabled VMkernel. Enabling multiple NICs for your vMotion enabled VMkernel’s will remove some of the
constraints from a bandwidth/throughput perspective that are associated with large and memory active virtual
machines. The following list shows the currently supported maximum number of NICs for multi-NIC vMotion:

 1GbE – 16 NICs supported

 10GbE – 4 NICs supported

It is important to realize that in the case of 10GbE interfaces, it is only possible to use the full bandwidth when the
server is equipped with the latest PCI Express busses. Ensure that your server hardware is capable of taking full
advantage of these capabilities when this is a requirement.

Stun During Page Send

A couple of months back I described this cool vSphere 4.1 vMotion enhancement called Quick Resume and now it
is replaced with Stun During Page Send, or also often referred to as “Slowdown During Page Send” is a feature that
“slowsdown” the vCPU of the virtual machine that is being vMotioned. Simply said, vMotion will track the rate at
which the guest pages are changed, or as the engineers prefer to call it, “dirtied”. The rate at which this occurs is
compared to the vMotion transmission rate. If the rate at which the pages are dirtied exceeds the transmission rate,
the source vCPUs will be placed in a sleep state to decrease the rate at which pages are dirtied and to allow the
vMotion process to complete. It is good to know that the vCPUs will only be put to sleep for a few milliseconds at a
time at most. SDPS injects frequent, tiny sleeps, disrupting the virtual machine’s workload just enough to guarantee
vMotion can keep up with the memory page change rate to allow for a successful and non-disruptive completion of
 36

the process. You could say that, thanks to SDPS, you can vMotion any type of workload regardless of how
aggressive it is.

It is important to realize that SDPS only slows down a virtual machine in the cases where the memory page change
rate would have previously caused a vMotion to fail.

This technology is also what enables the increase in accepted latency for long distance vMotion. Pre-vSphere 5.0,
the maximum supported latency for vMotion was 5ms. As you can imagine, this restricted many customers from
enabling cross-site clusters. As of vSphere 5.0, the maximum supported latency has been doubled to 10ms for
environments using Enterprise Plus. This should allow more customers to enable DRS between sites when all the
required infrastructure components are available like, for instance, shared storage.

vSphere 5.1 vMotion deepdive

vSphere 5.1 vMotion enables a virtual machine to change its datastore and host simultaneously, even if the two hosts
don't have any shared storage in common. For me this is by far the coolest feature in the vSphere 5.1 release, as this
technology opens up new possibilities and lays the foundation of true portability of virtual machines. As long as two
hosts have (L2) network connection we can live migrate virtual machines. The new vMotion provides a new level of
ease and flexibility for virtual machine migrations and the beauty of this is that it spans the complete range of
customers. It lowers the barrier for vMotion use for small SMB shops, allowing them to leverage local disk and
simpler setups, while big datacenter customers can now migrate virtual machines between clusters that may not have
a common set of datastores between them.

Let’s have a look at what the feature actually does. In essence, this technology combines vMotion and Storage
vMotion. But instead of either copying the compute state to another host or the disks to another datastore, it is a
unified migration where both the compute state and the disk are transferred to different host and datastore. All is
done via the vMotion network (usually). The moment the new vMotion was announced at VMworld, I started to
receive questions. Here are the most interesting ones that allows me to give you a little more insight of this new
enhancement.

Migration type
One of the questions I have received is, will the new vMotion always move the disk over the network? This depends
on the vMotion type you have selected. When selecting the migration type; three options are available:

This may be obvious to most, but I just want to highlight it again. A Storage vMotion will never move the compute
state of a VM to another host while migrating the data to another datastore. Therefore when you just want to move a
VM to another host, select vMotion, when you only want to change datastores, select Storage vMotion.

Which network will it use?

vMotion will use the designated vMotion network to copy the compute state and the disks to the destination host
when copying disk data between non-shared disks. This means that you need to the extra load into account when the
disk data is being transferred. Luckily the vMotion team improved the vMotion stack to reduce the overhead as
much as possible.
 37

Does the new vMotion support multi-NIC for disk migration?

The disk data is picked up by the vMotion code, this means vMotion transparently load balances the disk data traffic
over all available vMotion vmknics. vSphere 5.1 vMotion leverages all the enhancements introduced in the vSphere
5.0 such as Multi-NIC support and SDPS. Duncan wrote a nice article on these two features.

Is there any limitation to the new vMotion when the virtual machine is using shared vs. unshared swap ?

No, either will work, just as with the traditional vMotion.

Will the new vMotion features be leveraged by DRS/DPM/Storage DRS ?

In vSphere 5.1 DRS, DPM and Storage DRS will not issue a vMotion that copies data between datastores. DRS and
DPM remains to leverage traditional vMotion, while Storage DRS issues storage vMotions to move data between
datastores in the datastore cluster. Maintenance mode, a part of the DRS stack, will not issue a data moving vMotion
operation. Data moving vMotion operations are more expensive than traditional vMotion and the cost/risk benefit
must be taken into account when making migration decisions. A major overhaul of the DRS algorithm code is
necessary to include this into the framework, and this was not feasible during this release.

How many concurrent vMotion operations that copies data between datastores can I run simultaneously?

A vMotion that copies data between datastores will count against the limitations of concurrent vMotion and Storage
vMotion of a host. In vSphere 5.1 one cannot perform more than 2 concurrent Storage vMotions per host. As a result
no more than 2 concurrent vMotions that copy data will be allowed.

How is disk data migration via vMotion different from a Storage vMotion?

The main difference between vMotion and Storage vMotion is that vMotion does not “touch” the storage subsystem
for copy operations of non-shared datastores, but transfers the disk data via an Ethernet network. Due to the
possibilities of longer distances and higher latency, disk data is transferred asynchronously. To cope with higher
latencies, a lot of changes were made to the buffer structure of the vMotion process. However if vMotion detects
that the Guest OS issues I/O faster than the network transfer rate, or that the destination datastore is not keeping up
with the incoming changes, vMotion can switch to synchronous mirror mode to ensure the correctness of data.
 38

I understand that the vMotion module transmits the disk data to the destination, but how are changed blocks
during migration time handled?
For disk data migration vMotion uses the same architecture as Storage vMotion to handle disk content. There are
two major components in play – bulk copy and the mirror mode driver. vMotion kicks off a bulk copy and copies as
much as content possible to the destination datastore via the vMotion network. During this bulk copy, blocks can be
changed, some blocks are not yet copied, but some of them can already reside on the destination datastore. If the
Guest OS changes blocks that are already copied by the bulk copy process, the mirror mode drive will write them to
the source and destination datastore, keeping them both in lock-step. The mirror mode driver ignores all the blocks
that are changed but not yet copied, as the ongoing bulk copy will pick them up. To keep the IO performance as high
as possible, a buffer is available for the mirror mode driver. If high latencies are detected on the vMotion network,
the mirror mode driver can write the changes to the buffer instead of delaying the I/O writes to both source and
destination disk.

What is copied first, disk data or the memory state?

If data is copied from non-shared datastores, vMotion must migrate the disk data and the memory across the
vMotion network. It must also process additional changes that occur during the copy process. The challenge is to get
to a point where the number of changed blocks and memory are so small that they can be copied over and switch
over the virtual machine between the hosts before any new changes are made to either disk or memory. Usually the
change rate of memory is much higher than the change rate of disk and therefore the vMotion process start of with
the bulk copy of the disk data. After the bulk data process is completed and the mirror mode driver processes all
ongoing changes, vMotion starts copying the memory state of the virtual machine.

But what if I share datastores between hosts; can I still use this feature and leverage the storage network?

Yes and this is very cool piece of code, to avoid overhead as much as possible, the storage network will be leveraged
if both the source and destination host have access to the destination datastore. For instance, if a virtual machine
resides on a local datastore and needs to be copied to a datastore located on a SAN, vMotion will use the storage
network to which the source host is connected. In essence a Storage vMotion is used to avoid vMotion network
utilization and additional host CPU cycles.
 39

Because you use Storage vMotion, will vMotion leverage VAAI hardware offloading?

If both the source and destination host are connected to the destination datastore and the datastore is located on an
array that has VAAI enabled, Storage vMotion will offload the copy process to the array.

Hold on, you are mentioning Storage vMotion, but I have Essential Plus license, do I need to upgrade to
Standard?

To be honest I try to keep away from the licensing debate as far as I can, but this seems to be the most popular
question. If you have an Essential Plus license you can leverage all these enhancements of vMotion in vSphere 5.1.
You are not required to use a standard license if you are going to migrate to a shared storage destination datastore.

vSphere 5.0: Storage vMotion and the Mirror Driver

There’s a cool and exciting new feature as part of Storage vMotion in vSphere 5.0. This new feature is called Mirror
Mode and it enables faster and highly efficient Storage vMotion processes. But what is it exactly, and what does it
replace?

Prior to vSphere 5.0 we used a mechanism called Change Block Tracking (CBT), to ensure that blocks which were
already copied to the destination were marked as changed and copied during the iteration. Although CBT was
efficient compared to legacy mechanisms (snapshots), the Storage vMotion engineers came up with an even more
elegant and efficient solution which is called Mirror Mode. Mirror Mode does exactly what you would expect it to
do; it mirrors the I/O. In other words, when a virtual machine that is being Storage vMotioned writes to disk, the
write will be committed to both the source and the destination disk. The write will only be acknowledged to the
virtual machine when both the source and the destination have acknowledged the write. Because of this, it is
unnecessary to do re-iterative copies and the Storage vMotion process will complete faster than ever before.

The questions remain: How does this work? Where does Mirror Mode reside? Is this something that happens inside
or outside of the guest? A diagram will make this more obvious.
 40

By leveraging DISKLIB, the Mirror Driver can be enabled for the virtual machine that needs to be Storage
vMotioned. Before this driver can be enabled, the virtual machine will need to be stunned and of course unstunned
after it has been enabled. The new driver leverages the datamover to do a single-pass block copy of the source disk
to the destination disk. Additionally, the Mirror Driver will mirror writes between the two disks. Not only has
efficiency increased but also migration time predictability, making it easier to plan migrations. I’ve seen data where
the “down time” associated with the final copy pass was virtually eliminated (from 13seconds down to 0.22
seconds) in the case of rapid changing disks, but also the migrations time went from 2900 seconds back to 1900
seconds. Check this great paper by Ali Mashtizadeh for more details.

The Storage vMotion process is fairly straight forward and not as complex as one might expect.

1. The virtual machine working directory is copied by VPXA to the destination datastore.
2. A “shadow” virtual machine is started on the destination datastore using the copied files. The “shadow”
virtual machine idles, waiting for the copying of the virtual machine disk file(s) to complete.
3. Storage vMotion enables the Storage vMotion Mirror driver to mirror writes of already copied blocks to the
destination.
4. In a single pass, a copy of the virtual machine disk file(s) is completed to the target datastore while
mirroring I/O.
5. Storage vMotion invokes a Fast Suspend and Resume of the virtual machine (similar to vMotion) to
transfer the running virtual machine over to the idling shadow virtual machine.
6. After the Fast Suspend and Resume completes, the old home directory and VM disk files are deleted from
the source datastore.
1. It should be noted that the shadow virtual machine is only created in the case that the virtual
machine home directory is moved. If and when it is a “disks-only Storage vMotion, the virtual
machine will simply be stunned and unstunned.

Mulipathing:-

 Most Recently Used (MRU): The VMW_PSP_MRU policy selects the first working path, discovered at
system boot time. If this path becomes unavailable, the ESXi/ESX host switches to an alternative path and
continues to use the new path while it is available. This is the default policy for Logical Unit Numbers
(LUNs) presented from an Active/Passive array. ESXi/ESX does not return to the previous path if, or when,
it returns; it remains on the working path until it, for any reason, fails.
 41

Note: The preferred flag, while sometimes visible, is not applicable to the MRU pathing policy and can be
disregarded.

 Fixed (Fixed): The VMW_PSP_FIXED policy uses the designated preferred path flag, if it is configured.
Otherwise, it uses the first working path discovered at system boot time. If the ESXi/ESX host cannot use
the preferred path or it becomes unavailable, the ESXi/ESX host selects an alternative available path. The
host automatically returns to the previously defined preferred path as soon as it becomes available again.
This is the default policy for LUNs presented from an Active/Active storage array.

 Round Robin (RR): The VMW_PSP_RR policy uses an automatic path selection, rotating through all
available paths, enabling the distribution of load across the configured paths.

o For Active/Passive storage arrays, only the paths to the active controller will be used in the Round
Robin policy.
o For Active/Active storage arrays, all paths will be used in the Round Robin policy.

Note: For logical units associated with Microsoft Cluster Service (MSCS) and Microsoft Failover
Clustering virtual machines, the Round Robin pathing policy is supported only on ESXi 5.5 and later.

Destination TCP Ports UDP Ports

Source

Source 445, 139, 137, 138

Converter server
computer 9089,9090

Virtual Center 443

Converter server

Converter 443
Converter client
server

ESX 443, 902

Source computer

Ports to be opened for P2V-TCP Ports: 443, 902

To initiate a Telnet test to a port from Windows:

1. Open a command prompt. For more information, see Opening a command or shell prompt (1003892).
2. In the command prompt window, type:-

telnet server port

Where server is the hostname or IP address of the server, and port is the port that you want to connect to.

3. Press Enter.

Note: To exit the Telnet application, press Ctrl + ], then type quit.
 42

Depending on the application that uses the port, you may only see a blank screen with a cursor in the corner; this is
normal. Some common outputs of a successful connection attempt are:

 Connecting to port 902 on an ESXi/ESX host:

C:\>telnet server 902

Connecting...

220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP,

MKSDisplayProtocol:VNC

 Connecting to port 25 on a mail server:

C:\>telnet server 25
Connecting...

220 server ESMTP Sendmail 8.13.3/8.13.3;

 Connecting to port 443 on the vCenter Server:

C:\>telnet server 443

Connecting...

220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP,

MKSDisplayProtocol:VNC

If Telnet is unable to connect to the port, the output is similar to:

C:\>telnet server 902
Connecting To server...

Could not open connection to the host, on port 902: Connect failed
If the connection is refused, a firewall may be blocking that port from your source to the destination server. For
more information, see Required ports for configuring an external firewall to allow ESX and vCenter Server traffic
(1005189).

Testing port connectivity with Telnet from Linux or Mac OS X

To initiate a Telnet test to a port from Linux or Mac OS X:
1. Open a shell prompt. For more information.

2. In the shell prompt window, type:

telnet server port

Where server is the hostname or IP address of the server, and port is the port that you want to connect to.

3. Press Enter.
 43

Tasks to perform before conversion

To prepare for conversion:

1. If the source is a domain controller, special considerations must be made. VMware does not recommend virtualizing
an active domain controller with Converter.

2. If the source is Microsoft Exchange, SQL, or other database server, VMware recommends that the application
(Microsoft Exchange/SQL) and database services be shut down prior to conversion. This minimizes any chance of
corrupted database tables or stale data in the destination virtual machine.

3. Disable the real-time antivirus scanning during the conversion.

4. Verify that you are using or have downloaded the latest version of VMware Converter.
If you have previously installed or attempted a conversion with an earlier version of VMware Converter, a previous
version may still be installed verify/uninstall it.
5. Install VMware Converter directly to the source operating system using the local Administrator account, if you are
going to use remote hot clone feature you may choose a custom installation to only install the converter agent. If the
source server is running Windows NT or Windows 2000, you must reboot it after installing VMware Converter or
Converter does not start.
Note: In some cases, a domain administrator account may be used depending on your environment, local and group
policies, and account permissions.
6. If the NIC on the source machine is compatible with TOE (TCP Offload Engine), you need to disable it by running
this command in a command prompt on the source machine:

netsh int tcp set global chimney=disabled

7. Confirm that the source has 200 MB of free disk space on its system volume. This space is required to operate the
disk snapshot features in Converter.
Note: It is possible to separate the source partitions in different destination volumes during the conversion.
8. Run VMware Converter as a local administrator. Using a local administrator account removes any possible
permissions issues. If you are performing a remote conversion, be sure to specify the login user as the Administrator
account.
9. Run the System Configuration Utility(msconfig) on the source server to reduce the number of services and
applications running on startup, all software except for All Microsoft Services and VMware Converter Service.
10. If you have static IP addresses assigned, assign the interfaces DHCP addresses prior to conversion, if possible.
11. If the source is a virtual machine created in Microsoft Virtual PC, remove the Virtual PC Additions, prior to
conversion.
12. If the destination is an ESX host:

1. Connect to the server using its IP address instead of DNS host name. Using the host name of the ESX host
may expose issues with DNS name resolution that can prevent the Converter from connecting.
2. Confirm that the source server can access the destination ESX host directly using ports 443 and 902, even if
using VirtualCenter. Authenticate to the ESX host using the root account.
3. If the source server contains a hard drive or partition larger than 256GB, ensure that the destination
datastore's block size is 2MB, 4MB, or 8MB, and not the default 1MB size. The 1 MB default block size
cannot accommodate a file larger than 256 GB. The block size is no longer used on a VMFS 5 datastore
connected to an ESXi 5.0 Host.
4. Confirm that you are providing a unique name for the target virtual machine. Use the Virtual Infrastructure
(VI) client to confirm that the name is not already in use.
 44

Tasks to perform after conversion has completed

After conversion has completed:

1. Review the virtual hardware settings:
 Adjust the number of virtual NICs. If you need to customize the host name or IP address, leave all NICs
disconnected but present.
 Remove any unnecessary devices such as USB controllers (if running on ESX), COM ports or floppy drives
2. Start the virtual machine in Safe Mode.
3. Click Start > Control Panel > Add / Remove Programs. Remove any unnecessary programs used to install or
support device drivers, such a RAID management tools, network teaming or management software, wireless card
management software, and video and sound drivers. Do not restart if prompted by an uninstall program.
4. Restart the virtual machine into Normal mode.
5. Remove any additional devices or device drivers that were used to support hardware on the physical server. Use
either the Device Manager or Control Panel, depending on the version of Windows, to remove unnecessary devices.
It may also be necessary to view the Event Log to clear any remaining device startup failure messages.

Note: To remove the hidden devices from the Windows operating system, follow these steps:
1. Click Start, point to All Programs, point to Accessories, and then click Command Prompt.
2. At a command prompt, type the following command , and then press ENTER:
3. set devmgr_show_nonpresent_devices=1
4. Type the following command at command prompt and then press Enter: start devmgmt.msc
5. Troubleshoot the devices and drivers in Device Manager.
NOTE: Click Show hidden devices on the View menu in Device Manager before you can see devices that
are not connected to the computer.
6. When you finish troubleshooting, close Device Manager.
7. Type exit at the command prompt.
8.
Note that when you close the command prompt window, Window clears
thedevmgr_show_nonpresent_devices=1 variable that you set in step 2 and prevents ghosted devices from
being displayed when you click Show hidden devices.

6. VMware recommends changing the HAL in the virtual machine to uniprocessor if the source server is configured
with multi-CPU hardware abstraction layer (HAL), and the destination virtual machine is configured to use a single
CPU.
7. Install VMware Tools and restart if prompted.
8. If required, customize the virtual machine's identity. VMware recommends using the Microsoft Sysprep utility to
accomplish this, however it can also be accomplished by manually changing its computer host name, IP address, and
any other required unique identification.
9. If the System Configuration Utility(msconfig) was used prior to conversion, select the Normal startup option to
change switch back to a normal boot configuration.
10. Apply any previously removed static IP address settings, as required.
11. Reconnect any disconnected virtual NICs, as required.

Converting a Linux Virtual machine:-

If you want to convert a Linux machine which does not have GUI (graphical user interface). I’ll try to help. I
suppose that you have a Linux server, an workstation under XP and your infrastructure ESX.
 45

The steps you need to do:

01.) Download Vmware Converter for linux here.

02.) Extract the converter package on your linux server: “tar xf VMware-converter-4.0.0-
146302.tar.gz”
03.) Start the installation “cd vmware-converter-distrib/ && ./vmware-install.pl”
04.) You can just accept the majority of the options just make sure that you activate the
remote access “Do you want to enable remote access in Converter Standalone
Server?yes”
05.) Usually a linux server has an Apache installed on port 80, so good thing to do is to add
additional port for http proxy : “What port do you want the HTTP proxy to use?
[80] 8080 and What port do you want the HTTPS proxy to use? [443] 444
06.) Also you should go and allow root to login localy “/etc/ssh/sshd_config change the line
PermitRootLogin yes
07.) Then you just leave your linux server and go to your Windows XP or Vista Workstation
and connect to your linux server remotely via http for example “http://192.168.0.1:444”
08.) Download and install the VMware Converter client.
09.) Start the Converter Client and go to Administration > Connect to another server

10.) In the credentials window just enter the IP address followed by the port number, then the root
login and password.
 46

11.) Enter the credentials once more and choose Linux as a OS family.
 47

12.) Then Just follow the assistant for the rest of the steps. As a destination you choose the ESX server IP address or
hostname.

How to split drives in a P2V:-

VMware converter (version 3.0.3). In this update a new feature was introduced that makes P2V-ing a little bit easier.
It is finally possible to convert individual volumes on a single physical disk from the source physical machine to
separate and independent virtual disks across different datastores. Thank you VMware!

Select: Create a separate disk for each volume

Select the appropriate datastore for each disk.

H/W version or VMware tools upgrade-Which is first..?

When you upgrade from virtual hardware version 3 to version 7, the upgrade is irreversible, even if you take a
virtual machine backup or snapshot before performing the upgrade.
 48

When you upgrade from virtual hardware version 4 to version 7 the upgrade is reversible if you take a virtual
machine backup or snapshot before performing the upgrade

VMware recommends that before you upgrade the virtual hardware, first upgrade VMware Tools on the virtual
machine. This is especially important for virtual machines with Microsoft Windows guest operating systems. On
Microsoft Windows virtual machines, if you upgrade the virtual hardware before you upgrade VMware Tools, the
virtual machine might lose its network settings.

Essentials for Essentials Plus Standard Advanced Enterprise Enterprise Plus

retail for retail

Center Server vCenter Server vCenter Server vCenter vCenter vCenter vCenter Server

compatibility for Essentials, for Essentials, Server Server Server Foundation and

vCenter Server vCenter Server Foundation Foundation Foundation Standard

Foundation and Foundation and and Standard and Standard and Standard

vCenter Server vCenter Server

Standard Standard

Cores per 6 6 6 12 6 12

processor

vSMP support 4-way 4-way 4-way 4-way 4-way 8-way

Memory per 256 GB 256 GB 256 GB 256 GB 256 GB No license limit,

physical server but ESX and

ESXi 4.0

currently support
 49

up to 1 TB of

memory

Thin provisioning X X X X X X

vCenter Agent X X X X X X

Update Manager X X X X X X

VMsafe X X X X X X

vStorage APIs X X X X X X

High availability X X X X X

Data recovery X X X X

Hot add X X X

Fault tolerance X X X

vShield Zones X X X
 50

VMotion X X X

Storage VMotion X X

Distributed X X

Resource

Scheduler and

Distributed Power

Management

vNetwork X

Distributed Switch

Host Profiles X

Third-party X

multipathing

Pre-Requisites

vMotion

1. Host must be licensed for vMotion

2. Configure host with at least one vMotion n/w interface (vmkernel port group)
3. Shared storage (this has been compromised in 5.1)
4. Same VLAN and VLAN label
5. GigaBit ethernet network required between hosts
 51

6. Processor compatibility between hosts

7. vMotion does not support migration of applications clustered using Microsoft clustering service
8. No CD ROM attached
9. No affinity is enabled
10. VMware tools should be installed

Fault Tolerance

Host requirements

 Hosts should have license for FT feature

 Hosts must have processors from the FT compatible processor group

Cluster requirements

 There should be at least two hosts with the same host build number
 The hosts should have the same datastores and networks
 FT logging and vMotion networking should be configured
 VMware HA should be enabled

VM requirements

 VM must be stored in vmdk or virtual RDM files that are thick provisioned
 VM files should be on shared storage
 Only VMs with single vCPU is supported by FT till 5.5, sphere 6.0 support up to 4vCPU
 VM should be running on one of the supported operating systems

Does FT supports failback?

So we have seen failover is transparent with FT and without any disruption of services. What about failback? What
happens when primary server running the primary VM comes back online after recovering from failure. What
happens now? Will the original primary (which got failed) becomes secondary or will it become primary again and
force the new primary (which was secondary before the failure) to become secondary.This was some questions
which kept me waiting for a long time before I got a correct explanation. I discussed this with many of the
colleagues of mine and each one have their own version of answer.

So answers for above question is “NO, FT doesn’t supports failback“. Even after the physical server which comes
online after failure, it is not going to disrupt the current pair of primary-secondary FT VM. The original primary VM
 52

which gone down due to host failure never comes back online again. All the memory pointers of failed primary VM
is deleted. There can’t be more than 2 VM’s at any given time in a FT pair.

Distributed Resource Scheduler (DRS)

 Shared storage is required for the hosts in the DRS cluster

 Processor compatibility of hosts in the DRS cluster
 Should meet all vMotion prerequisites
 DRS is required for the working of DPM

High Availability

 All hosts must be licensed for HA

 All hosts must have access to the same management network
 All hosts must have access to the same VM networks and datastores
 For VMM, VMware tools must be installed
 All hosts must have DNS configured

Storage vMotion

■ Virtual machine disks must be in persistent mode or be raw device mappings (RDMs). For virtual compatibility
mode RDMs, you can migrate the mapping file or convert to thick-provisioned or thin-provisioned disks during
migration as long as the destination is not an NFS datastore. If you convert the mapping file, a new virtual disk
is created and the contents of the mapped LUN are copied to this disk. For physical compatibility mode RDMs,
you can migrate the mapping file only.
■ Migration of virtual machines during VMware Tools installation is not supported.

■ Because VMFS3 datastores do not support large capacity virtual disks, you cannot move virtual disks greater
than 2TB from a VMFS5 datastore to a VMFS3 datastore.
■ The host on which the virtual machine is running must have a license that includes Storage vMotion.

■ ESX/ESXi 3.5 hosts must be licensed and configured for vMotion. ESX/ESXi 4.0 and later hosts do not require
vMotion configuration in order to perform migration with Storage vMotion.
■ The host on which the virtual machine is running must have access to both the source and target datastores.

What is a snapshot?
A snapshot is a “point in time image” of a virtual guest operating system (VM). That snapshot contains an image of
the VMs disk, RAM, and devices at the time the snapshot was taken. With the snapshot, you can return the VM to that
point in time, whenever you choose. You can take snapshots of your VMs, no matter what guest OS you have and the
snapshot functionality can be used for features like performing image level backups of the VMs without ever shutting
them down.
 53

Snapshot:-

In VMware, snapshot refers to the copy of the virtual machine disk file (VMDK) at a certain point in time. In
addition to the VMDK file, it copies the content in vRAM and VM settings at that point of time.

Snapshot creation
when you create a snapshot, the disk which was writable becomes read-only and a new delta file will be created.
This delta file will contain all changes (delta) to the original disk files. When you create multiple snapshots, new
delta files are created and the previous delta files become read-only. With multiple snapshots each delta file can
potentially grow as large as the original disk file.

Snapshot deletion

The process for deleting multiple snapshots has changed across vSphere versions.

vSphere 4.0 versions and VMware Infrastructure 3 (VI3) :

If a VM has 3 active snapshots and you deleted them, the following process occurs:

 Snapshot 3 is copied to Snapshot 2, which is then copied to Snapshot 1.

 Snapshot 1 is copied to the original disk file, and the helper snapshot (which writes any changes occur
during the snapshot) is copied to the original disk file.
 This process requires extra disk space because each snapshot grows as the previous snapshot is added to it.
If there isn't sufficient free disk space on the data store, the snapshots cannot be committed.

vSphere 4.0 versions and later :

Each snapshot is merged directly into the original disk, instead of merging with the previous snapshot. Because each
snapshot is directly merged into the original one at a time, no extra disk space is needed, except for the helper file.

Snapshots that have been active for a very long time (thereby becoming extremely large) can take a very long time
to commit when deleted. The amount of time the snapshot takes to commit varies depending on the VM's activity
level; it will commit faster if it is powered off.

Snapshot files

*--delta.vmdk file: This is the differential file created when you take a snapshot of a VM. It is also known as the
redo-log file. A delta file will be created for each snapshot that you create for a VM. An extra delta helper file will
also be created to hold any disk changes when a snapshot is being deleted or reverted. It contains any changes that
are made to the VM's disk while the snapshot is deleted. The size of the helper delta file varies and it's based on how
long the snapshot takes to delete.

*.vmsd file: This file is used to store metadata and information about snapshots. This file does not cleanup
completely after the snapshots are taken. Once you delete a snapshot, it will still increment the snapshot's last unique
identifier for the next snapshot.

*.vmsn file: This is the snapshot state file, which stores the exact running state of a virtual machine at the time you
take that snapshot. This file will either be small or large depending on if you select to preserve the VM's memory as
part of the snapshot. If you do choose to preserve the VM's memory, then this file will be a few megabytes larger
than the maximum RAM memory allocated to the VM.
This file is similar to the VMware suspended state (.vmss) file. A .vmsn file will be created for each snapshot taken
on the VM; these files are automatically deleted when the snapshot is removed.
 54

Snapshot and its performance

1. When you first create a snapshot, your VM activity will pause briefly; if you ping a VM while creating a
snapshot you will notice a few timeouts.
2. If a snapshot is active, the performance of the VM will be degraded because the host writes to delta files
differently and less efficiently than it does to standard VMDK files.
3. The delta file grows by each 16 MB increment (discussed in part one of this series), it will cause another
metadata lock. This can affect your VMs and hosts.
4. Deleting/committing a snapshot creates a metadata lock.
5. Snapshot you are deleting can create greatly reduced performance on its VM while the delta files are being
committed; this will be more noticeable if the VM is very busy.

Things to remember

 It is not possible to expand a VM's virtual disk while a snapshot is running, if you try the vmkfstools
command, you receive an error: Failed to extend the disk. Failed to lock the file. With the vSphere Client, if
you edit a VM's settings when a snapshot is running, and then select one of its virtual disks, the option to
resize the disk is grayed out. But once the snapshot is deleted, you can resize the virtual disk.
 If you have a VM with more than one disk and you wish to exclude a disk from being included in a
snapshot, you must edit the VM's settings by changing the disk mode to Independent (make sure you select
Persistent). The independent setting provides you the means to control how each disk functions
independently, there is no difference to the disk file or structure. Once a disk is Independent it will not be
included in any snapshots. You will not be able to include memory snapshots on a VM that has independent
disks.
 The size of a snapshot file can never exceed the size of the original disk file. There will be an overhead disk
space that contains information used to manage the snapshots. This varies according to the VMFS block
size. For eg: 1 MB block sized VMFS volume will have a maximum overhead of 2 GB.
 Snapshot files will initially be small (16 MB), but will grow as writes are made to the VM's disk files.
Snapshots grow in 16 MB increments to help reduce SCSI reservation conflicts. When requests are made to
change a block on the original disk, it is instead changed in the delta file.

What is Fault Tolerance ?

FT is a technology used by VMware to provide zero downtime for a virtual machine

Continuous protection for a VM even in case of a host failure
FT is enabled only on a VM basis
When enabled, FT creates a secondary VM in another ESXi
Uses Record and Replay technology & vLockstep technology
 55

Record and Replay technology

This technology records all events in the primary VM and replays in the secondary VM( Even a mouse click) but
only the primary VM produces output such as disk writes and network transmits

 It is done using a FT logging NIC

 The output of the secondary VM is suppressed by the ESXi and is not in the network until the primary fails
 If the primary fails, the secondary becomes the new primary

vLockstep technology

 This technology uses a second identical processor to monitor and verify the operations of the first processor
 Both processors receive the same inputs so the operation state of both processors will be identical
 Fault Tolerance feature relies on vLockstep technology to sync the CPU operations of a VM between two
hosts so they are in identical states

Limitations

 FT requires newer processor models from AMD and Intel which supports vLockstep technology
 All hosts must be running the same build of VMware ESX
 Shared storage is required as a single vmdk is being used by two VMs in two separate hosts
 HA must be enabled
 All VM disks must be “thick”
 No devices like USB, physical CD-ROM, floppy, physical RDMs are permitted
 Snapshots must be removed before enabling FT
 Taking snapshot of a VM in FT is not possible
 Storage vMotion operation is not permitted for a VM in FT
 FT enabled virtual machine is automatically configured as DRS-disabled but initially the secondary VM is
placed by DRS when FT is enabled

HA:-

vSphere 5.0 uses an agent called “FDM” Fault Domain Manager

1. No more primary/secondary node concept as in its predecessors

2. New master/slave concept with an automated election process
3. vpxa(vcenter agent) dependency removed
4. HA talks directly to hostd instead of using a translator to talk to vpxa
5. FDM agent communicates with vCenter to retrieve information about the status of virtual machines and vCenter is
used to display the protection status of virtual machines.
6. HA is no longer dependent on DNS.
7. Character limit that HA imposed on the hostname has been lifted (Previously 26 chars).
 56

8. If you add ESX/ESXi 4.1 or prior hosts to a vSphere 5.0 vCenter Server, the new vSphere HA Agent (FDM) will be
installed

Master/Slave concept
One of the nodes in your cluster becomes the Master and the rest become Slaves.
Master responsibilities:-
1. Monitors availability of hosts / VMs in cluster
2. Manages VM restarts after VM/host failures
3. Maintains list of VMs available in each ESXi host
4. Restarting failed virtual machines
5. Exchanging state with vCenter
6. Monitor the state of slaves

Slave responsibilities:-

1. Monitor their running VMs and send Status to Master and perform restarts on request from Master
2. Monitors Master Node Health
3. If the master should fail, participates in master election
Master-election algorithm:-
1. Takes 15 to 25s (depends on reason for election).
2. Elects participating host with the greatest number of mounted datastores.
3. Managed Object ID is used if there is a tie.

An election is held when:-

1. vSphere HA is enabled initially.

2. Master’s host fails or enters maintenance mode.
3. A Management Network partition occurs.
Heartbeating:-

1. Two different Heartbeat mechanisms

2. Network heartbeat mechanism.
3. Datastore Heartbeat mechanism (New – Used when network is unavailable)

Network heartbeat mechanism:-

1. Sends heartbeat between slaves and master in every second.

2. Election using UDP and master-slave communication using TCP.
3. When a slave isn’t receiving any heartbeats from the master, it will try to determine whether it is isolated or has
failed.
4. Prior vSphere 5.0, virtual machine restarts were always initiated, even if only the management network of the
host was isolated and the virtual machines were still running

Datastore heartbeating:-

1. Adds a new level of resiliency and allows HA to make distinction between a failed host and an isolated /
partitioned host.
2. Prevents Unnecessary Restarts
3. Two different files used : PowerOn file , Host hb file
4. Uses ‘PowerOn’ File to determine Isolation
 57

5. Datastore heartbeat mechanism is only used in case the master has lost network connectivity with the slaves
6. 2 datastores are automatically selected by vCenter for this mechanism
7. For VMFS datastores, the Master reads the VMFS heartbeat region(Uses locking mechanism).
8. For NFS datastores, the Master monitors a heartbeat file that is periodically touched by the Slaves.
9. File created by each hosts in datastore (Host-<number>-hb).
10. Virtual Machine availability is reported by a file created by each Slave which lists the powered on VMs. (host-
<number>-poweron)

Locking mechanism

1. HA leverages the existing VMFS files system locking mechanism

2. The locking mechanism uses a so called “heartbeat region” which is updated as long as the lock on a file exists
3. Host needs to have at least one open file on the volume to update heartbeat region
4. Per-host file is created on the designated heart beating datastores to ensure heartbeat
5. HA will simply check whether the heartbeat region has been updated
Isolated vs Partitioned
1. Host is considered to be either Isolated or Partitioned when it loses network access to a master but has not
failed
2. Isolation address is the IP address the ESXi hosts uses to check on isolation when no heartbeats are
received
3. VMware HA will use the default gateway as an isolation address (Normally)

Isolated

1. Is not receiving heartbeats from the master.

2. Is not receiving any election traffic
3. Cannot ping the isolation address

Partitioned

1. Is not receiving heartbeats from the master

2. Is receiving election traffic
3. (at some point a new master will be elected at which the state will be reported to vCenter)

When multiple hosts are isolated but can still communicate amongst each other over the management networks,
it is called a network partition.
When a network partition exists, a master election process will be issued.
By default the isolation response is triggered after ~30 seconds with vSphere 5.x
Failed Master Host:-
1. Master Election Initiated.
2. New Master Elected
3. New Master Restarts all VMs on the Protected list with Not Running State
Failed Slave Host:-

1. Master Check Network heartbeat

2. Master Checks Datastore Heartbeat
3. Master Restarts VMs Affected
Isolation Responses:-
1. Power Off
2. Leave Powered On
3. Shut Down
 58

Isolation Detection:-
1. Slaves will Hold Single Server Election and Check Ping Address
2. Master will Check Ping Address
3. Master Restarts VMs Affected
Isolation of a slave:-
1. T0 – Isolation of the host (slave)
2. T10s – Slave enters “election state”
3. T25s – Slave elects itself as master
4. T25s – Slave pings “isolation addresses”
5. T30s – Slave declares itself isolated and “triggers” isolation response

Isolation of a master

1. T0 – Isolation of the host (master)

2. T0 – Master pings “isolation addresses”
3. T5 – Master declares itself isolated and “triggers” isolation response

Master declares a host dead when:

1. Master can’t communicate with it over the network
2. Host is not connected to master
3. Host does not respond to ICMP pings
4. Master observes no storage heartbeats
Results in:
a. Master attempts to restart all VMs from host
b. Restarts on network-reachable hosts and its own host
Master declares a host partitioned when:
a. Master can’t communicate with it over the network
b. Master can see its storage heartbeats
Results in:
a. One master exists in each partition
b. VC reports one master’s view of the cluster
c. Only one master “owns” any one VM
d. A VM running in the “other” partition will be monitored via the heartbeat datastores
e. restarted if it fails (in master’s partition)
f. When partition is resolved, all but one master abdicates
A host is isolated when:
a. It sees no vSphere HA network traffic.
b. It cannot ping the isolation addresses
Results in:
a. Host invokes (improved) Isolation response
b. Checks first if a master “owns” a VM
c. Applied if VM is owned or datastore is inaccessible
Master
a. Restarts those VMs powered off or that fail later
b. Reports host isolated if both can access its heartbeat datastores, otherwise dead
 59

Determine if a slave is alive

a. Rely on heartbeats issued to slave’s HB datastores
b. Each FDM opens a file on each of its HB datastores for heartbeating purposes
c. Files contain no information. On VMFS datastores, file will have the minimum-allowed file size
d. Files are named X-hb, where X is the (SDK API) moID of the host
e. Master periodically reads heartbeats of all partitioned / isolated slaves
Determine the set of VMs running on a slave
a. A FDM writes a list of powered on VMs into a file on each of its HB datastores
b. Master periodically reads the files of all partitioned/isolated slaves
c. Each poweron file contains at most 140 KB of info. On VMFS datastores, actual disk usage is
determined by the file-sizes supported by the VMFS version
d. They are named X-powereon, where X is the (SDK API) moID of the host

Protected-vm files are used

a. When recovering from a master failure
b. To determine whether a master is responsible for a given VM
FDMs create a directory (.vSphere-HA) in root of each relevant datastore
Within it, they create a subdirectory for each cluster using the datastore.
You can migrate virtual machine disks with Storage vMotion if the virtual machine and its host meet the following
resource and configuration requirements:

SCRATCH PARTITION
The scratch partition is *not* used for the typical logs which most of us are familiar with. In vSphere the scratch
partition is used for storing the vm-support command output (support-bundle); it is also where the logs are stored
initially, by default, in those cases when the partition gets automatically created – see below. You would typically
provide this information when logging a support call with VMware

How to configure scratch location for ESXi logs? Follow those steps

Connect to vCenter Server or the ESXi host using the vSphere Client > Select your host > Configuration >
Storage
Right click datastore (local or shared) and select Browse
Create folder for this ESXi host named for example “scratch”> close the datastore browser.
Select the host > configuration > Advanced (software section) > ScratchConfig > Change the
ScratchConfig.ConfiguredScratchLocation

For example:
 60

/vmfs/volumes/DatastoreName/ESXi5-01/scratch

Click OK > go and put the host in maintenance mode and reboot for the configuration to take effect.

ESX 4.x 2050 to 2250 UDP ESXi/ESX Host ESXi/ESX Host VMware HA

Raw device mapping (RDM) is method to provide direct access to a LUN on a iscsi or fibre channel storage system
for a virtual machine. RDM is basically a Mapping file acts as a proxy for a raw physical storage device placed in a
VMFS volume. Virtual Machine can directly access the storage device using RDM and RDM contains metadata
which controls the disk access to the physical device.

Use cases for Raw Device Mapping

1. For Microsoft cluster configuration in Virtual machine (Virtual-to-virtual or physical-to-virtual).

2. for configuring a virtual machine to use N_Port ID Virtualization (NPIV)

3. For running SAN management software (Storage resource management software, storage array snapshot software,
replication software,etc) inside a virtual machine

4. For any application running in a virtual machine that needs to access a device using hardware-specific SCSI
commands

5. RDM is useful in physical-to-virtual conversion operations by avoiding migration of a large data LUN to a
VMDK.

Physical compatibility mode

1. Physical mode specifies minimal SCSI virtualization of the mapped device, allowing the greatest flexibility
for SAN management software.
2. The VMkernel passes all SCSI commands to the device, with one exception - The REPORT
LUNs command is virtualized, so that the VMkernel can isolate the LUN to the owning virtual machine.
Otherwise, all physical characteristics of the underlying hardware are exposed.

Physical mode is useful while running SAN management agents or other SCSI target-based software in the
virtual machine.
3. Physical mode also allows virtual-to-physical clustering for cost-effective high availability.
4. Virtual Machine Snapshots are not available when the RDM is used in physical compatibility mode.
5. You can use this mode for Physical-to-virtual clustering and cluster-across-boxes.
6. VMFS5 supports greater than 2 TB disk size for RDMs in physical compatibility mode only.
7. You cannot relocate larger than 2 TB RDMs to datastores other than VMFS5.
8. You cannot convert larger than 2 TB RDMs to virtual disks, or perform other operations that involve RDM
to virtual disk conversion. Such operations include cloning.
 61

Virtual compatibility mode

1. Virtual mode specifies full virtualization of the mapped device.

2. VMkernel sends only READ and WRITE to the mapped device. The mapped device appears to the guest
operating system exactly the same as a virtual disk file in a VMFS volume.
3. The real hardware characteristics are hidden.
4. If you are using a raw disk in virtual mode, you can realize the benefits of VMFS, such as advanced file
locking for data protection and snapshots for streamlining development processes.
5. Virtual mode is more portable across storage hardware than physical mode, presenting the same behavior
as a virtual disk file.
6. You can use this mode for both Cluster-in-a-box and cluster-across-boxes.

Note: RDM is not available for direct-attached block devices or certain RAID devices. You cannot map a disk
partition as RDM. RDMs require the mapped device to be a whole LUN.

ESXi 5.1 Host Log Files

Logs for an ESXi 5.1 host are grouped according to the source component:

 /var/log/auth.log: ESXi Shell authentication success and failure.

 /var/log/dhclient.log: DHCP client service, including discovery, address lease requests and renewals.

 /var/log/esxupdate.log: ESXi patch and update installation logs.

 /var/log/lacp.log: Link Aggregation Control Protocol logs.

 /var/log/hostd.log: Host management service logs, including virtual machine and host Task and Events,
communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections.

 /var/log/hostd-probe.log: Host management service responsiveness checker.

 /var/log/rhttpproxy.log: HTTP connections proxied on behalf of other ESXi host webservices.

 /var/log/shell.log: ESXi Shell usage logs, including enable/disable and every command entered.
/var/log/sysboot.log: Early VMkernel startup and module loading.

 /var/log/boot.gz: A compressed file that contains boot log information and can be read using zcat
/var/log/boot.gz|more.

 /var/log/syslog.log: Management service initialization, watchdogs, scheduled tasks and DCUI use.

 /var/log/usb.log: USB device arbitration events, such as discovery and pass-through to virtual machines.

 /var/log/vobd.log: VMkernel Observation events, similar to vob.component.event.

 /var/log/vmkernel.log: Core VMkernel logs, including device discovery, storage and networking device and
driver events, and virtual machine startup.

 /var/log/vmkwarning.log: A summary of Warning and Alert log messages excerpted from the VMkernel
logs.
 62

 /var/log/vmksummary.log: A summary of ESXi host startup and shutdown, and an hourly heartbeat with
uptime, number of virtual machines running, and service resource consumption.

 /var/log/Xorg.log: Video acceleration.

Enhanced vMotion Compatibility (EVC)

Enhanced vMotion Compatibility (EVC) simplifies vMotion compatibility issues across CPU generations. EVC
automatically configures server CPUs with Intel FlexMigration or AMD-V Extended Migration technologies to be
compatible with older servers.

After EVC is enabled for a cluster in the vCenter Server inventory, all hosts in that cluster are configured to present
identical CPU features and ensure CPU compatibility for vMotion. The features presented by each host are
determined by selecting a predefined EVC baseline. vCenter Server does not permit the addition of hosts that cannot
be automatically configured to be compatible with the EVC baseline.

Where does SRM shine:

 Disaster Recovery
 Orchestration
 Testing
 Reporting
 Disaster Avoidance (will incur downtime when VMs failover to other site)

Where does a Stretched Cluster solution shine:

 Workload mobility
 Cross-site automated load balancing
 Enhanced downtime avoidance
 Disaster Avoidance (VMs can be vMotioned, no downtime incurred!)

Stretched Cluster:-

A stretched cluster is a deployment model in which two or more virtualization host servers are part of the
same logical cluster but are located in separate geographical locations. In a stretched cluster the servers act
as a single system to provide high availability (HA) and load balancing despite not being located in the
same facility. A stretched cluster offers the advantage of enabling easy migration of virtual machines from
one geographic location to another while maintaining network connections with the other servers within the
cluster. This scenario can be useful if, for example, a hurricane or other natural disaster were expected to
hit one location. In this case, critical virtual machines could be migrated to servers within the stretched
cluster that are physically located in a different facility
 63

VMware vSphere Metro Storage Cluster (VMware vMSC)

 VMware vSphere Metro Storage Cluster (VMware vMSC) is a configuration option, introduced in vSphere
5, that allows for the use of stretched clusters that are spread across geographical locations. This
configuration allows organizations to perform load balancing and non-disruptive live migrations between
active data centers.

 VMware vMSC can give organizations many of the benefits that a local high-availability cluster provides,
but with geographically separate sites. Stretched clustering, also called distributed clustering, allows an
organization to move virtual machines (VMs) between two data centers for failover or proactive load
balancing. VMs in a metro storage cluster can be live migrated between sites with vSphere vMotion
and vSphere Storage vMotion. The configuration is designed for disaster avoidance in environments where
downtime cannot be tolerated, but should not be used as an organization's primary disaster recovery
approach.

 VMware vMSC was introduced in vSphere 5. VMware vMSC is only practical for organizations that
require active site balancing and very high availability (HA) standards, because the approach requires a
high bandwidth, low latency connection between the two sites. The maximum supported network
latency between the two sites is 10 milliseconds round-trip time, which limits the geographic distance in
which a stretched cluster will work to less than 100 kilometers, in most cases. vMSC supports Fibre
Channel (FC), iSCSI, network file system (NFS) and Fibre Channel over Ethernet (FCoE) protocols, but
vMSC links require low latency and high-bandwidth network connectivity for best operation.

Troubleshooting Storage I/O Control (1022091)

Symptoms

You experience one or more of these issues:

 Storage I/O Control (SIOC) is not performing as expected.

 I/O from virtual machines does not get prioritized under congestion circumstances, or it gets prioritized
when there is no real congestion.
 SIOC rules are intermittently applied to virtual machines on the same host.

Purpose

This article assists you in determining if SIOC has been correctly configured and provides steps to enable logging.

Resolution

Storage I/O Control (SIOC) is used to control the I/O usage of a virtual machine and to gradually enforce the
predefined I/O share levels. SIOC is supported on Fibre Channel and iSCSI connected storage in ESX/ESXi 4.1 and
5.0. With ESXi 5.0 support for NFS with SIOC was also added. Datastores with multiple extents or Raw Device
Mapping (RDM) are currently not supported

Notes:
 64

 Before using SIOC on datastores that are backed by arrays with automated storage tiering capabilities,
check the VMware Storage/SAN Compatibility Guide to ensure that your automated tiered storage array is
certified to be compatible with SIOC.
 Before enabling SIOC, ensure that datastores are managed by a single vCenter Server.

Note: Storage I/O Control (SIOC) requires Enterprise Plus licensing. Without this license, the option to enable SIOC
is grayed out.
Enabling Storage I/O Control

To enable SIOC:

1. Select a datastore in the vSphere Client inventory and click the Configuration tab.
2. Click Properties.
3. Under Storage I/O Control, select Enabled.
4. Click Close.

Note: This setting is specific to the datastore and not to the host.

If you experience problems with SIOC or if the number of hosts connected to the datastore has changed since
enabling SIOC:

1. Disable SIOC and save the changes by clicking OK.

2. Enable SIOC and save the changes.

Determining if the threshold value has been modified

To determine if the threshold value has been modified:

1. Select a datastore in the vSphere Client inventory and click the Configuration tab.
2. Click Properties.
3. Under Storage I/O Control, click Advanced.
4. Check if the value is 30ms. If it is not 30, reset it to the default value of 30.

Ensuring virtual machines have disk shares assigned according to their importance

By default, all virtual machines have the same number of shares and IOPS limit. IOPS are the number of I/O
operations per second. By default, IOPS are unlimited. If these defaults are not changed, then I/O control does not
prioritize virtual machines.

To see the shares of all the virtual machines on the cluster, choose the cluster, click Resource Allocation, then
click Storage.

To change the vDisk shares and limit:

1. Choose a virtual machine in the vSphere Client inventory.

2. Click the Summary tab and click Edit Settings.
3. Click the Resources tab and click Disk.
o Choose a virtual hard disk from the list and click the Share column to select the relative amount
of shares to allocate to the virtual machine (Low, Normal, or High).
o You can also click Custom and enter a user-defined share value.

4. Click the Limit - IOPS column and enter the upper limit of storage resources to allocate to the virtual
machine.
 65

5. Click OK.

Note: The above process is used to set the resource consumption limits of individual vdisks in a virtual machine
even when SIOC is not enabled. These settings are specific to the individual guest, and not the host, although they
are used by SIOC.
Enabling and disabling SIOC logging on the host

These logs are used for troubleshooting purposes. If you are sending the support logs to VMware, enable SIOC
logging before collecting the logs. VMware recommends that you disable logging after the troubleshooting activity
is complete.

To enable logging:

1. Click Host > Configuration.

2. In the left pane, click Software and then click Advanced Settings.
3. In the Misc section, select the Misc.SIOControlLogLevel parameter.
4. Set the value to 7 for complete logging.

For example:

Min value: 0 (no logging)

Max value: 7

After changing the log level, you see these changes logged in the/var/log/vmkernel logs:

May 27 07:41:10 wdc-tse-h53 vmkernel: 76:14:16:03.334 cpu7:4110)Config: 297: "SIOControlLoglevel"

= 1, Old Value: 0, (Status: 0x0)

5. Perform the action that is failing or repeat the procedure to replicate the observed issue.

To disable logging:

1. Click Host > Configuration.

2. In the left pane, click Software and then click Advanced Settings.
3. Set the Misc.SIOControlLogLevel value to 0.
4. After changing the log level, you see these changes logged in the /var/log/vmkernel logs.

May 27 07:41:31 wdc-tse-h53 vmkernel: 76:14:16:24.521 cpu4:4111)Config: 297: "SIOControlLoglevel"

= 0, Old Value: 1, (Status: 0x0)

Note: SIOC log files are saved in /var/log/messages. In ESXi 5.0 or later versions, the SIOC log files are saved
in/var/log/storagerm.log.
Run this command to stop and start SIOC on ESX;

/etc/init.d/vmware-late {start|stop|status|restart}
Run this command to stop and start SIOC on ESXi:

/etc/init.d/storageRM {start|stop|status|restart}

Below are some suggestions you can implement to improve disk I/O performance issues:
 66

Suggestion Details

Use non-growable or When creating a production virtual machine, VMware recommends that the virtual hard disk be
preallocated VMDK configured to preallocated mode. If existing disks are not in preallocated mode, use the vmware-
disks vdiskmanager tool to convert the disks. Consult the product's User Guide for more information.

When a snapshot is created, the VMware product produces an additional delta file. Each successive
snapshot produces an additional file. When a disk operation is performed within the guest, the disk
I/O is recreated by parsing each snapshot delta file in the chain. This produces additional disk
Remove or reduce
overhead on the host because more than one file must be opened and processed to recreate the I/O
snapshots
data for the guest operating system. For best performance, remove all snapshots in the guest
operating system or store performance-sensitive data on an independent virtual disk. Consult the
product's User Guide for information on configuring independent virtual disks.

Use separate physical Install the host operating system onto a separate hard disk from the virtual machines. Also store the
and virtual hard disks paging file or swap partition on a different drive than the host operating system.

Run disk defragmentation software on the host and in the guest operating system. Fragmentation of
Optimize the drive
both the .vmdk files and within the guest can double the impact from fragmentation.

Implementing partitions inside the guest operating system or host can improve performance by
creating fragmentation boundaries and can reduce further fragmentation. For example, consider
Use partitions storing the small, often modified files of the operating system away from large files such as
database or Microsoft Exchange stores by using a separate partition. Also consider storing the
virtual disk files (.vmdk files) on their own partition or disk on the host

Certain RAID configurations can impact read or write performance positively and negatively.
Use RAID or adjust the When using a RAID 5 configuration, consider adding more disks to the array. This generally
RAID configuration or improves the performance of the array. Using mirroring can improve read performance but may
add disks to the array degrade write performance. If write performance is primarily impaired, consider a different RAID
type to host the virtual machine.

Check for disk Disk encryption can reduce disk performance. Try moving the virtual machine to a non-encrypted
encryption volume and test if performance has improved.

Ensure the existing

Often disk problems such as bad sectors or failing controllers can impact performance because I/O
physical hardware is
and bad cluster auto-recovery can cause sudden interruptions in I/O operations to the device.
healthy and performing
Perform a hardware and file system diagnostic to verify if this is impacting performance.
as expected

It is important to consider the performance characteristics of the physical disk hardware. In general,
hardware RAID and independent disk controllers perform better than software RAID and
integrated disk controllers. When an independent controller is used, often it is possible to configure
Upgrade or choose or increase the cache memory to yield better performance. Consult the hardware vendor for more
different physical disk information. Typically older hardware performs slower than newer hardware. Hard disks used in
hardware laptop or notebook computers are often far slower than drives used in desktop computers. SCSI
hard disks typically perform much faster than those used in regular desktops and notebooks. Hard
disks connected over USB typically perform slower than directly attached local disks (such as IDE,
SATA, and SCSI). Flash-based USB thumb drives typically perform slower than hard drives.
 67

Review the performance specifications provided by the disk manufacturer on critical metrics such
as spindle speed, average seek time (latency), and burst data transfer rates. Higher spindle speeds,
lower seek times, and higher transfer rates perform better. When comparing flash-based drives,
observe both the read and write transfer performance ratings.

Adding these settings to a virtual machine can reduce the I/O load on the hard disk, however these
adjustments require additional memory on the host. Only add these settings if there is sufficient
free memory on the host to accommodate all the memory allocated to the virtual machine,
otherwise you may cause a memory starvation condition that can reduce performance of all the
running virtual machines or possibly affect the host operating system. Use these settings with
caution.

Open the .vmx file for the affected virtual machine while it is powered off. Add the following lines
to the file using a text editor.

Edit the virtual Note: If you are using VMware Server, you may need to restart the VMware Authorization Service
machine settings to (vmware-authd) for changes to take effect.
reduce I/O usage by
MemTrimRate = "0"
using more host
mainMem.useNamedFile = "FALSE"
memory
sched.mem.pshare.enable = "FALSE"
prefvmx.useRecommendedLockedMemSize = "TRUE"

Note: If you are using a Linux host, use the following entry instead of mainMem.useNamedFile =
"FALSE". The mainMem.useNamedFile entry only applies to Windows Hosts.

mainmem.backing = "swap"

Maximum snapshots

snapshot.maxSnapshots = "n" where n = max number of snapshots and n <= 496

BLADES

ML, DL & BL indicate the families for HPs lineup of Industry Standard Servers: ProLiant.
 68

BL = Blades= HP BladeSystem. Up to 16 server &/or storge blades + Networking, sharing power and cooling in 10u
enclosure.

DL = Density Line = Rack mount ProLiant server. Here, HP puts as many features as possible in as small an
enclosure as possible. 1 U (rack unit, 1.75") is the smallest. It is a complete server and does not share components
(like power and cooling) with other servers (like blades).

ML = Maximized Line. These are typically ProLiant servers in a tower enclosures but there are several in rack
enclosures. Here HP gives you more slots for option cards and more drive bays and typically, more memory slots.
They are also complete servers that do not share components like blades.

Why Blade:-

A blade server is a stripped down server computer with a modular design optimized to minimize the use of physical
space and energy. Whereas a standard rack-mount server can function with (at least) a power cord and network
cable, blade servers have many components removed to save space, minimize power consumption and other
considerations, while still having all the functional components to be considered a computer. [1] Unlike a rack-mount
server, a blade server needs a blade enclosure. A blade enclosure, which can hold multiple blade servers, provides
services such as power, cooling, networking, various interconnects and management. Together, blades and the blade
enclosure form a blade system (although BladeSystem from Hewlett-Packard is a specific product name). Different
blade providers have differing principles regarding what to include in the blade itself, and in the blade system
altogether.

In a standard server-rack configuration, one rack unit or 1U—19 inches (480 mm) wide and 1.75 inches (44 mm)
tall—defines the minimum possible size of any equipment. The principal benefit and justification of blade
computing relates to lifting this restriction so as to reduce size requirements. The most common computer
rack form-factor is 42U high, which limits the number of discrete computer devices directly mountable in a rack to
42 components. Blades do not have this limitation. As of 2014, densities of up to 180 servers per blade system (or
1440 servers per rack) are achievable with blade systems

Blade enclosure

Enclosure (or chassis) performs many of the non-core computing services found in most computers. Non-blade
systems typically use bulky, hot and space-inefficient components, and may duplicate these across many computers
that may or may not perform at capacity. By locating these services in one place and sharing them between the blade
computers, the overall utilization becomes higher. The specifics of which services are provided may vary by vendor.
 69

HP BladeSystem c7000 enclosure (populated with 16 blades), with two 3U UPS units below.
Power

Computers operate over a range of DC voltages, but utilities deliver power as AC, and at higher voltages than
required within computers. Converting this current requires one or more power supply units (or PSUs). To ensure
that the failure of one power source does not affect the operation of the computer, even entry-level servers may have
redundant power supplies, again adding to the bulk and heat output of the design.

The blade enclosure's power supply provides a single power source for all blades within the enclosure. This single
power source may come as a power supply in the enclosure or as a dedicated separate PSU supplying DC to multiple
enclosures.[3][4] This setup reduces the number of PSUs required to provide a resilient power supply.

The popularity of blade servers, and their own appetite for power, has led to an increase in the number of rack-
mountable uninterruptible power supply (or UPS) units, including units targeted specifically towards blade servers
(such as the BladeUPS).
Cooling

During operation, electrical and mechanical components produce heat, which a system must dissipate to ensure the
proper functioning of its components. Most blade enclosures, like most computing systems, remove heat by
using fans.

A frequently underestimated problem when designing high-performance computer systems involves the conflict
between the amount of heat a system generates and the ability of its fans to remove the heat. The blade's shared
power and cooling means that it does not generate as much heat as traditional servers. Newer blade-enclosures
feature variable-speed fans and control logic, or even liquid cooling systems[5][6] that adjust to meet the system's
cooling requirements.

At the same time, the increased density of blade-server configurations can still result in higher overall demands for
cooling with racks populated at over 50% full. This is especially true with early-generation blades. In absolute
terms, a fully populated rack of blade servers is likely to require more cooling capacity than a fully populated rack of
standard 1U servers. This is because one can fit up to 128 blade servers in the same rack that will only hold 42 1U
rack mount servers.[7]
 70

Networking

Blade servers generally include integrated or optional network interface controllers for Ethernet or host
adapters for Fibre Channel storage systems or converged network adapter to combine storage and data via one Fibre
Channel over Ethernet interface. In many blades at least one interface is embedded on the motherboard and extra
interfaces can be added using mezzanine cards.

A blade enclosure can provide individual external ports to which each network interface on a blade will connect.
Alternatively, a blade enclosure can aggregate network interfaces into interconnect devices (such as switches) built
into the blade enclosure or in networking blades.[8][9]

Storage

While computers typically use hard disks to store operating systems, applications and data, these are not necessarily
required locally. Many storage connection methods (e.g. FireWire, SATA, E-
SATA, SCSI, SASDAS, FC and iSCSI) are readily moved outside the server, though not all are used in enterprise-
level installations. Implementing these connection interfaces within the computer presents similar challenges to the
networking interfaces (indeed iSCSI runs over the network interface), and similarly these can be removed from the
blade and presented individually or aggregated either on the chassis or through other blades.

The ability to boot the blade from a storage area network (SAN) allows for an entirely disk-free blade, an example
of which implementation is the Intel Modular Server System.

All versions of the enclosure occupy 10 rack units and can accommodate up to 16 half-height blade servers. It
includes space for 6 power supplies (single-phase, three-phase or a −48V DC), 10 cooling fans, 8 single-wide (such
as Gigabit Ethernet or FC) or 4 double-wide (such as 40Gb Ethernet or Infiniband) interconnect modules (that
allows for up to 4 redundant interconnect fabrics).

In a c3000, all internal NICs of the blades are wired to a single interconnect module bay. If you lose that
interconnect module, you will lose all network connectivity unless you have added extra NICs in the mezzanine
slots and an extra LAN I/O module(s) in the appropriate slot(s).
In a c7000, the internal NICs are split between two I/O modules in slots 1 and 2. The downside is that you will need
two I/O modules to make all your internal NICs useable, but the upside is that it will be possible to configure them
in a redundant fashion (i.e. with no extra NICs, only half of your LAN bandwidth will be lost if you lose an I/O
module).

You must understand that the I/O routing from the blades to the I/O modules is hardwired: you cannot just configure
any NIC or HBA to use any I/O module you wish. This is true in both c3000 and c7000 enclosures. The I/O routing
diagrams are among the most important tools when designing a configuration for c-class enclosures. You'll find
them in the documentation of each enclosure.
 71

ESXi upgrade

3.54.0/4.15.x6.0

4.1 VC upgrade

As was the case with vCenter Server 4.1, vCenter Server 5.0 only supports being installed on a 64 bit OS.
If you are currently running vCenter 4.0 on a 32-bit OS you cannot upgrade. You must create a new server
(physical or virtual) with a supported 64-bit OS and either start with a clean installation and database or
migrate your installation and database to a new 64-bit OS server and then perform the upgrade.

Before upgrading you should backup both the database and the SSL certificates in
C:\ProgramData\VMware\VMware VirtualCenter\SSL\

To back up the vCenter Server database:

1. Download and install Microsoft SQL Server Management Studio Express.

To download Microsoft SQL Server Management Studio Express, see these Microsoft Download locations:

o MS SQL Studio Express 2005

o MS SQL Studio Express 2008

Note: The preceding links were correct as of October 10, 2013. If you find a link is broken,
provide feedback and a VMware employee will update the link.

2. Open SQL Server Management Studio Express and connect to the vCenter Server database server or
instance.

Note: If you are not using a dedicated SQL server, the database resides on vCenter Server. To verify the
location and database name:

a. Open the ODBC connector.

b. Click the System DSN tab.
c. Find the vCenter Server item.
d. Click configure.
e. Proceed through the steps and note the SQL Server and the default database.
 72

3. Stop all vCenter Server services on the vCenter Server system. This ensures that no service is out of sync
during the backup. For more information, see Stopping, starting, or restarting vCenter services (1003895).
4. Expand Databases and look for the vCenter Server database, usually named as VIM_VCDB, unless you
renamed it during the initial creation.
5. Right-click the vCenter Server database and Click Tasks > Back Up.
6. Select your backup options and Click OK.
7. Copy the backup file to a safe location during maintenance or upgrades. Ensure to perform this backup
before performing any operation on vCenter Server, as it holds current and historical data.
8. Start all vCenter Server services on the vCenter Server system once the backup has completed. For more
information, see Stopping, starting, or restarting vCenter services (1003895).

1) Make DB backup
2) Stop vCenter services, set them to "Manual startup"
3) Backup SSL keys
4) Remove vCenter server from domain
5) Shut down vCenter server
7) Make Windows 64bit clean installation on new server, use same IP and name as for old vCenter server
8) Restore SSL keys from backup
9) Make 2 ODBC connections, 64bit for vCenter and 32bit for VUM
10) Install vCenter, point to existing database.
You will have old vCenter as a rollback, but you have to restore old vCenter DB from backup first.
You can upgrade an ESXi 5.0.x, ESXi 5.1.x, or ESXi 5.5.х host directly to 6.0, using any of these methods.

Upgrade Methods Upgrade from Upgrade from ESXi 5.0 to ESXi Upgrade from ESXi 5.1 to ESXi
ESX/ESXi 4.x 5.5 5.5
to ESXi 5.5

vSphere Update Yes Yes Yes

Manager

Interactive upgrade Yes Yes Yes

from CD
DVD or USB drive

Scripted upgrade Yes Yes Yes

vSphere Auto No Yes, if the ESXi 5.0.x host was Yes, if the ESXi 5.1.x host was
Deploy deployed using Auto Deploy deployed using Auto Deploy

esxcli No Yes Yes

 73

Product Standard Enterprise Enterprise Plus

Cores per CPU No more restrictions No more restrictions No more restrictions
CPU Entitlement Per CPU Per CPU Per CPU
vRAM Entitlement Unlimited Unlimited Unlimited
vCPU Entitlement 8-Way 32-Way (V5.0: 8-Wege) 64-Way (V5.0: 32-
Wege)
SUSE Linux Enterprise yes yes yes
Server for VMware
vCenter Server vCenter Server vCenter Server vCenter Server
Compatibility Foundation & Standard Foundation & Standard Foundation & Standard
Thin Provisioning Yes Yes Yes
Update Manager (*) Yes Yes Yes
Data Recovery (*) Yes Yes Yes
High Availability (*) Yes Yes Yes
vMotion (*) Yes Yes Yes
Storage APIs for Data Yes Yes Yes
Protection
Hot Add yes yes yes
Replication yes yes yes
vShield Zones (*) yes yes yes
Fault Tolerance yes yes yes
Storage vMotion (*) yes yes yes
Virtual Serial Port Yes Yes
Concentrator
Storage APIs for Memory Yes Yes
Integration
Storage APIs for Multi- Yes Yes
pathing
Distributed Resource Yes Yes
Scheduler + DPM
Storage I/O Control Yes
Network I/O Control Yes
Distributed Switch Yes
Host Profiles Yes
Auto-deploy Yes
Storage DRS Yes
Profile-based Storage Yes
Connection
 74

Configurations Maximums-5.1,5.5,6.0 1
Storage vMotion with RDMs and Snapshots 2
VDS-vSphere Distributed Switch 2
Private vLans 3
Types of Port Binding 5
Security Settings 7
Traffic shaping policies 7
Vlan Tagging 7
Difference between vmfs3 and vmfs 5 9
UEFI 10
vNUMA 10
vSphere APIs for Array Integration-VAAI 13
vSphere APIs for Storage Awareness-VAAS 16
Commands-ESXi 17