Vmware ESXi 5 Configuration Tests

Background:
These tests are conducted to judge the feasibility to migrate & upgrade the current virtualization
infrastructure from Vmware server 2 to ESXi 5. Currently we have 2 host servers (SUSE SLES 11 with
Vmware Server2), running 5 VMs (Kerio Firewall, ADC, DHCP, Aspire, Linux Update Server ) and the
setup is running quite well for almost 10 months.
The problem with this setup is that in case of any disaster (for example dead server or HDD) the failover
procedure is not very smooth and elegant. What we usually do is copy the VMs from our primary server
to backup server , then attach/map LAN cards to proper VMs via web console and run the VMs. Though
it seems quite easy, in reality it has some severe drawbacks. These include:
1) The host servers are running in Active/Passive mode meaning when the Primary server is
running, the Backup server remains shutdown and during the disaster we power up Backup and
shutdown Primary. The complete procedure is manual.
2) VMs doesnt sync among both servers automatically. We have to manually track and update the
VMs to Backup server after sometime (mostly by-weekly) so that both severs remain in sync and
contains the updated copy of all VMs. This takes long time as the combined size of all VMs is
around 50GB 60GB.
3) Mapping of LAN cards in both servers is different thereby causing confusion when connecting
LAN cables into server.
4) Performance of VMs can be slower since the Vmware software is running on top of existing OS
which causes overhead. Though we havent seen such problem in our environment (180+ users)
but it makes sense to eliminate the host OS completely.
5) Sometimes an outdated copy of the VM can be run if it doesnt cause problem. For example, we
have SUSE DHCP and Kerio firewall whom we doesnt update often and during a problem we
runs an outdated copy of these VMs. But never ever try to run an older version of AD/ADC VM
because if the USN number of DC/ADC mismatches then itll stop replicating information and
becomes a non-functional. Itll also cause inconsistency in Active Directory database and creates
problem which are very difficult to diagnose and troubleshoot. We learned it the hard-way.
Even after having the above problems we dont have any stability or performance issues from using
Vmware Server 2 and it can be confidently used as a test bed before migrating completely towards
virtualization.
Evolution:
As the next logical step well migrate our virtualization infrastructure to ESXi 5 hypervisor. But before
continuing we want to be sure that it is beneficial to use in our environment in terms of fault-tolerance,
high availability, better manageability and foremost, the cost (hardware).
We have planned a series of tests during July 2012 to check different configurations of ESXi 5, check
failover options & timing, check high availability and finally check fault-tolerance.
Configuration 1
Tests conducted on: July 1, 2012
Tests conducted by: RSK
Tests conducted at: PEvo Headoffice
Total time spent: Almost 8 hours
Software used: Vmware (vsphere) ESXi 5, Vmware VSphere Client, FreeNAS
Scenario:








Reason to Test this Configuration:
To see if we can use software based NAS connected via 100Mpbs LAN link (It is required for failover and
HA).
Explanation:
ESXi 5 host PC is connected to FreeNAS software storage server via 100Mbps LAN. FreeNAS would
become an iSCSI Target and VM datastore.
I wanted to test that what would be the level of performance if I move the VM datastore from ESX host
server to a storage server? Since we dont have any dedicated hardware NAS weve built one using
FreeNAS software.
After configuring iSCSI on FreeNAS we make it available to ESXi, and create 2 VMs in it. Windows Server
2003 & SUSE SLES 11 is installed as a test. The average performance during installation was very good
and it doesnt seem in a least bit that the storage system is separate from ESXi host itself. Read/Write
access speed was very good, following figures shows the different stats.

ESXi 5
Host
Server
FreeNAS
Storage Server
Core2Duo 2.8Ghz
7GB RAM
80GB HDD
6 Gigabit LAN
Core2Duo Xeon
2.8Ghz
1GB RAM
250GB HDD
6 Gigabit LAN
Connected via 100Mbps LAN
FreeNAS Stats (During both OS installation)

















































SUSE SLES 11 Stats


















Average traffic transfer during installation from ESXi host to NAS server (1min stats)












































Windows Server 2003 Stats

















Average traffic transfer during installation from ESXi host to NAS server (1min stats)









































The Crash Tests:
A series of crash tests have been performed after the installation of VMs to check the stability of
external storage and ESXi since there is a risk of single point of failure as everything is connected via a
simple CAT6 cable. The tests finding are listed below.
Crash Test 1

Crash Performed:
Disconnect LAN connectivity for 5mins between ESXi server & NAS server while the VMs are
powered on.
During Crash:
Vsphere client hangs
VMs OS services remain offline (no ping, no mstsc etc.)
After Recovering from Crash
VMs come online and didnt power off or restart. No damage to VMs.

Crash Test 2

Crash Performed:
Restart ESXi host normally. NAS remains ON.
During Crash:
VMs OS services remain offline (no ping, no mstsc etc.)
After Recovering from Crash
It was observed that some VM doesnt powered off gracefully. It happens due to 2 reasons.
First, if Vmware Tools isnt installed inside VM and secondly if ESXi
Configuration->Startup/Shutdown settings isnt set to Guest Shutdown (Default is Power
off).

Crash Test 3

Crash Performed:
Set ESXi to autostart VM upon boot and check if it happens or not.
During Crash:
N/A
After Recovering from Crash
All VMs started successfully.

Crash Test 4

Crash Performed:
Start/Power ON ESXi host with VM set to auto start. NAS is disconnected/powered OFF.
During Crash:
ESXi boots up extremely slow since it tries to connect to NAS during boot up.
After Recovering from Crash
All VMs started successfully.