Vous êtes sur la page 1sur 58

Acropolis Advanced Administration Guide

Acropolis
Contents

1. Cluster Management..................................................................................... 4
Controller VM Access.................................................................................................................................. 4
Port Requirements............................................................................................................................ 4
Admin Access to Controller VM....................................................................................................... 4
Starting a Nutanix Cluster........................................................................................................................... 6
Stopping a Cluster....................................................................................................................................... 7
Destroying a Cluster.................................................................................................................................... 8
Creating Clusters from a Multiblock Cluster............................................................................................... 9
Cluster IP Address Configuration..................................................................................................... 9
Configuring the Cluster................................................................................................................... 10
Verifying IPv6 Link-Local Connectivity............................................................................................13
Failing from one Site to Another............................................................................................................... 15
Disaster failover.............................................................................................................................. 15
Planned failover.............................................................................................................................. 15
Fingerprinting Existing vDisks....................................................................................................................16

2. Changing Passwords.................................................................................. 17
Changing User Passwords........................................................................................................................ 17
Changing the SMI-S Provider Password (Hyper-V)....................................................................... 18
Changing the Controller VM Password (Nutanix User).............................................................................19
Changing the Controller VM Password (Admin User).............................................................................. 19

3. Cluster IP Address Configuration............................................................. 21


Network Configuration (Virtual Interfaces, Virtual Switches, and IP Addresses)...................................... 21
Changing Controller VM IP Addresses..................................................................................................... 23
Preparing to Set IP Addresses.......................................................................................................23
Preparing to Change Controller VM IP Addresses........................................................................ 24
Changing Controller VM IP Addresses...........................................................................................26
Completing Controller VM IP Address Change..............................................................................27
Changing a Controller VM IP Address (manual)............................................................................28

4. Creating a Windows Guest VM Failover Cluster...................................... 29

5. Acropolis Dynamic Scheduling in AHV.................................................... 31


Disabling Acropolis Dynamic Scheduling.................................................................................................. 31
Enabling Acropolis Dynamic Scheduling...................................................................................................32

6. Logs.............................................................................................................. 33
Sending Logs to a Remote Syslog Server................................................................................................33
Configuring the Remote Syslog Server Settings............................................................................ 34
Common Log Files.................................................................................................................................... 35
Nutanix Logs Root.......................................................................................................................... 36
Self-Monitoring (sysstats) Logs.......................................................................................................36
/home/nutanix/data/logs/cassandra.................................................................................................36

ii
Controller VM Log Files.................................................................................................................. 36
Correlating the FATAL log to the INFO file............................................................................................... 39
Stargate Logs.............................................................................................................................................40
Cassandra Logs.........................................................................................................................................41
Prism Gateway Log................................................................................................................................... 42
Zookeeper Logs......................................................................................................................................... 43
Genesis.out................................................................................................................................................ 43
Diagnosing a Genesis Failure........................................................................................................ 44
ESXi Log Files........................................................................................................................................... 45
Nutanix Calm Log Files............................................................................................................................. 45

7. Troubleshooting Tools................................................................................ 47
Nutanix Cluster Check (NCC)................................................................................................................... 47
Installing NCC from an Installer File.............................................................................................. 48
Upgrading NCC Software............................................................................................................... 50
NCC Usage..................................................................................................................................... 51
Diagnostics VMs........................................................................................................................................ 52
Running a Test Using the Diagnostics VMs................................................................................... 53
Diagnostics Output..........................................................................................................................53
Syscheck Utility.......................................................................................................................................... 54
Using Syscheck Utility.................................................................................................................... 54

8. Controller VM Memory Configurations..................................................... 55


CVM Memory and vCPU Configurations (G5/Broadwell)..........................................................................55
Platform Workload Translation (G5/Broadwell)...............................................................................56
CVM Memory and vCPU Configurations (G4/Haswell/Ivy Bridge)............................................................57
CVM Memory Configurations for Features................................................................................................58

iii
1
Cluster Management
Although each host in a Nutanix cluster runs a hypervisor independent of other hosts in the cluster, some
operations affect the entire cluster.

Controller VM Access
Most administrative functions of a Nutanix cluster can be performed through the web console or nCLI.
Nutanix recommends using these interfaces whenever possible and disabling Controller VM SSH access
with password or key authentication. Some functions, however, require logging on to a Controller VM
with SSH. Exercise caution whenever connecting directly to a Controller VM as the risk of causing cluster
issues is increased.

Warning: When you connect to a Controller VM with SSH, ensure that the SSH client does not import or
change any locale settings. The Nutanix software is not localized, and executing commands with any locale
other than en_US.UTF-8 can cause severe cluster issues.
To check the locale used in an SSH session, run /usr/bin/locale. If any environment variables
are set to anything other than en_US.UTF-8, reconnect with an SSH configuration that does not
import or change any locale settings.

Port Requirements
Nutanix uses a number of ports for internal communication. The following unique ports are required for
external access to Controller VMs in a Nutanix cluster.

Table 1: Table

Purpose Port Numbers


Remote site replication 2009 and 2020
Cluster and IP address configuration 2100
Remote support tunnel (outgoing connection to service centers 80 or 8443
nsc01.nutanix.net and nsc02.nutanix.net)
Management interface (web console, nCLI) 9440

Admin Access to Controller VM


You can access the Controller VM as the admin user (admin user name and password) with SSH. For
security reasons, the password of the admin user must meet complexity requirements. When you log on to
the Controller VM as the admin user for the first time, you are prompted to change the default password.
Following are the default credentials of the admin user:
• User name: admin
• Password: Nutanix/4u
The password must meet the following complexity requirements:
• At least 8 characters long

AOS | Cluster Management | 4


• At least 1 lowercase letter
• At least 1 uppercase letter
• At least 1 number
• At least 1 special character
• At least 4 characters difference from the old password
• Must not be among the last 5 passwords
• Must not have more than 2 consecutive occurrences of a character
• Must not be longer than 199 characters
After you have successfully changed the password, the new password is synchronized across all Controller
VMs and interfaces (Prism web console, nCLI, and SSH).

Note:
• As an admin user, you cannot access nCLI by using the default credentials. If you are logging
in as the admin user for the first time, you must SSH to the Controller VM or log on through the
Prism web console. Also, you cannot change the default password of the admin user through
nCLI. To change the default password of the admin user, you must SSH to the Controller VM or
log on through the Prism web console.
• When you make an attempt to log in to the Prism web console for the first time after you upgrade
to AOS 5.1 from an earlier AOS version, you can use your existing admin user password to log in
and then change the existing password (you are prompted) to adhere to the password complexity
requirements. However, if you are logging in to the Controller VM with SSH for the first time after
the upgrade as the admin user, you must use the default admin user password (Nutanix/4u)
and then change the default password (you are prompted) to adhere to the password complexity
requirements.

By default, the admin user password does not have an expiry date, but you can change the password at
any time.
When you change the admin user password, you must update any applications and scripts using the
admin user credentials for authentication. Nutanix recommends that you create a user assigned with the
admin role instead of using the admin user for authentication. The Prism Web Console Guide describes
authentication and roles.
Following are the default credentials to access a Controller VM.

Table 2: Controller VM Credentials

Interface Target User Name Password

SSH client Nutanix Controller VM admin Nutanix/4u


nutanix nutanix/4u
Prism web console Nutanix Controller VM admin Nutanix/4u

Accessing the Controller VM Using the Admin Account

Perform the following procedure to log on to the Controller VM by using the admin user with SSH for the
first time.

1. Log on to the Controller VM with SSH by using the management IP address of the Controller VM and
the following credentials.
• User name: admin
• Password: Nutanix/4u

AOS | Cluster Management | 5


You are now prompted to change the default password.

2. Respond to the prompts, providing the current and new admin user password.
Changing password for admin.
Old Password:
New password:
Retype new password:
Password changed.

The password must meet the following complexity requirements:


• At least 8 characters long
• At least 1 lowercase letter
• At least 1 uppercase letter
• At least 1 number
• At least 1 special character
• At least 4 characters difference from the old password
• Must not be among the last 5 passwords
• Must not have more than 2 consecutive occurrences of a character
• Must not be longer than 199 characters
For information about logging on to a Controller VM by using the admin user account through the Prism
web console, see Logging Into The Web Console in the Prism Web Console guide.

Starting a Nutanix Cluster

1. Log on to any Controller VM in the cluster with SSH.

2. Start the Nutanix cluster.


nutanix@cvm$ cluster start

If the cluster starts properly, output similar to the following is displayed for each node in the cluster:
CVM: 10.1.64.60 Up
Zeus UP [5362, 5391, 5392, 10848, 10977,
10992]
Scavenger UP [6174, 6215, 6216, 6217]
SSLTerminator UP [7705, 7742, 7743, 7744]
SecureFileSync UP [7710, 7761, 7762, 7763]
Medusa UP [8029, 8073, 8074, 8176, 8221]
DynamicRingChanger UP [8324, 8366, 8367, 8426]
Pithos UP [8328, 8399, 8400, 8418]
Hera UP [8347, 8408, 8409, 8410]
Stargate UP [8742, 8771, 8772, 9037, 9045]
InsightsDB UP [8774, 8805, 8806, 8939]
InsightsDataTransfer UP [8785, 8840, 8841, 8886, 8888,
8889, 8890]
Ergon UP [8814, 8862, 8863, 8864]
Cerebro UP [8850, 8914, 8915, 9288]
Chronos UP [8870, 8975, 8976, 9031]
Curator UP [8885, 8931, 8932, 9243]
Prism UP [3545, 3572, 3573, 3627, 4004,
4076]
CIM UP [8990, 9042, 9043, 9084]
AlertManager UP [9017, 9081, 9082, 9324]
Arithmos UP [9055, 9217, 9218, 9353]
Catalog UP [9110, 9178, 9179, 9180]

AOS | Cluster Management | 6


Acropolis UP [9201, 9321, 9322, 9323]
Atlas UP [9221, 9316, 9317, 9318]
Uhura UP [9390, 9447, 9448, 9449]
Snmp UP [9418, 9513, 9514, 9516]
SysStatCollector UP [9451, 9510, 9511, 9518]
Tunnel UP [9480, 9543, 9544]
ClusterHealth UP [9521, 9619, 9620, 9947, 9976,
9977, 10301]
Janus UP [9532, 9624, 9625]
NutanixGuestTools UP [9572, 9650, 9651, 9674]
MinervaCVM UP [10174, 10200, 10201, 10202,
10371]
ClusterConfig UP [10205, 10233, 10234, 10236]
APLOSEngine UP [10231, 10261, 10262, 10263]
APLOS UP [10343, 10368, 10369, 10370,
10502, 10503]
Lazan UP [10377, 10402, 10403, 10404]
Orion UP [10409, 10449, 10450, 10474]
Delphi UP [10418, 10466, 10467, 10468]

What to do next
After you have verified that the cluster is running, you can start guest VMs.
(Hyper-V only) If the Hyper-V failover cluster was stopped, start it by logging on to a Hyper-V host and
running the Start-Cluster PowerShell command.

Warning: By default, Nutanix clusters have redundancy factor 2, which means they can tolerate the failure of a
single node or drive. Nutanix clusters with a configured option of redundancy factor 3 allow the Nutanix cluster to
withstand the failure of two nodes or drives in different blocks.
• Never shut down or restart multiple Controller VMs or hosts simultaneously.
• Always run the cluster status command to verify that all Controller VMs are up before
performing a Controller VM or host shutdown or restart.

Stopping a Cluster
Before you begin
Shut down all guest virtual machines, including vCenter if it is running on the cluster. Do not shut down
Nutanix Controller VMs.

Note:
• If you are running Acropolis File Services (AFS), stop AFS before stopping your AOS cluster.
• If you are planning to stop your cluster that has metro availability configured, do not stop the
cluster before performing some remedial actions. For more information, see Data Protection
Guidelines (Metro Availability) topic in the Prism Web Console guide.

(Hyper-V only) Stop the Hyper-V failover cluster by logging on to a Hyper-V host and running the Stop-
Cluster PowerShell command.

Note: This procedure stops all services provided by guest virtual machines, the Nutanix cluster, and the
hypervisor host.

1. Log on to a running Controller VM in the cluster with SSH.

AOS | Cluster Management | 7


2. Stop the Nutanix cluster.
nutanix@cvm$ cluster stop

Wait to proceed until output similar to the following is displayed for every Controller VM in the cluster.
CVM: 172.16.8.191 Up, ZeusLeader
Zeus UP [3167, 3180, 3181, 3182, 3191, 3201]
Scavenger UP [3334, 3351, 3352, 3353]
ConnectionSplicer DOWN []
Hyperint DOWN []
Medusa DOWN []
DynamicRingChanger DOWN []
Pithos DOWN []
Stargate DOWN []
Cerebro DOWN []
Chronos DOWN []
Curator DOWN []
Prism DOWN []
AlertManager DOWN []
StatsAggregator DOWN []
SysStatCollector DOWN []

Destroying a Cluster
Before you begin
Reclaim licenses from the cluster to be destroyed by following Reclaiming Licenses When Destroying a
Cluster in the Web Console Guide.

Note: If you have destroyed the cluster and did not reclaim the existing licenses, contact Nutanix Support to
reclaim the licenses.

Destroying a cluster resets all nodes in the cluster to the factory configuration. All cluster configuration and
guest VM data is unrecoverable after destroying the cluster.

Note: If the cluster is registered with Prism Central (the multiple cluster manager VM), unregister the cluster
before destroying it. See Registering with Prism Central in the Web Console Guide for more information.

1. Log on to any Controller VM in the cluster with SSH.

2. Stop the Nutanix cluster.


nutanix@cvm$ cluster stop

Wait to proceed until output similar to the following is displayed for every Controller VM in the cluster.
CVM: 172.16.8.191 Up, ZeusLeader
Zeus UP [3167, 3180, 3181, 3182, 3191, 3201]
Scavenger UP [3334, 3351, 3352, 3353]
ConnectionSplicer DOWN []
Hyperint DOWN []
Medusa DOWN []
DynamicRingChanger DOWN []
Pithos DOWN []
Stargate DOWN []
Cerebro DOWN []
Chronos DOWN []
Curator DOWN []
Prism DOWN []

AOS | Cluster Management | 8


AlertManager DOWN []
StatsAggregator DOWN []
SysStatCollector DOWN []

3. Destroy the cluster.

Caution: Performing this operation deletes all cluster and guest VM data in the cluster.

nutanix@cvm$ cluster destroy

Follow the prompts to confirm destruction of the cluster.

Creating Clusters from a Multiblock Cluster


The minimum size for a cluster is three nodes.

1. Remove nodes from the existing cluster.


• If you want to preserve data on the existing cluster, remove nodes from the cluster using the
Hardware > Table > Host screen of the web console.
• If you want multiple new clusters, destroy the existing cluster by following Destroying a Cluster on
page 8.

2. Create one or more new clusters by following Configuring the Cluster on page 10.

Cluster IP Address Configuration

AOS includes a web-based configuration tool that automates assigning IP addresses to cluster
components and creates the cluster.
Requirements
The web-based configuration tool requires that IPv6 link-local be enabled on the subnet. If IPv6 link-local is
not available, you must configure the Controller VM IP addresses and the cluster manually. The web-based
configuration tool also requires that the Controller VMs be able to communicate with each other.
All Controller VMs and hypervisor hosts must be on the same subnet. The hypervisor can be multihomed
provided that one interface is on the same subnet as the Controller VM.
Guest VMs can be on a different subnet.

AOS | Cluster Management | 9


Configuring the Cluster
Before you begin
Check that the cluster is ready to be configured by following Preparing to Set IP Addresses on page 23.

Note: This procedure has been deprecated (superseded) in AOS 4.5 and later releases. Instead, use the
Foundation tool to configure a cluster. See the "Creating a Cluster" topics in the Field Installation Guide for more
information.

Video: Click here to see a video (MP4 format) demonstration of this procedure. (The video may not reflect the
latest features described in this section.)

Figure 1: Cluster IP Address Configuration Page

1. Open a web browser.


Nutanix recommends using Internet Explorer 9 for Windows and Safari for Mac OS.

Note: Internet Explorer requires protected mode to be disabled. Go to Tools > Internet Options >
Security, clear the Enable Protected Mode check box, and restart the browser.

2. In the browser, go to http://[cvm_ipv6_addr]:2100/cluster_init.html.


Replace [cvm_ipv6_addr] with the IPv6 address of any Controller VM that should be added to the
cluster.

AOS | Cluster Management | 10


Following is an example URL to access the cluster creation page on a Controller VM:
http://[fe80::5054:ff:fea8:8aae]:2100/cluster_init.html

If the cluster_init.html page is blank, then the Controller VM is already part of a cluster.
Connect to a Controller VM that is not part of a cluster.
You can obtain the IPv6 address of the Controller VM by using the ifconfig command.
Example
nutanix@cvm$ ifconfig
eth0 Link encap:Ethernet HWaddr 52:54:00:A8:8A:AE
inet addr:10.1.65.240 Bcast:10.1.67.255 Mask:255.255.252.0
inet6 addr: fe80::5054:ff:fea8:8aae/64 Scope:Link
...etc...

The value of the inet6 addr field up to the / character is the IPv6 address of the Controller VM.

3. Type a meaningful value in the Cluster Name field.


This value is appended to all automated communication between the cluster and Nutanix support. It
should include the customer's name and, if necessary, a modifier that differentiates this cluster from
any other clusters that the customer might have.

Note: This entity has the following naming restrictions:


• The maximum length is 75 characters (for vSphere and AHV) and 15 characters (for Hyper-
V).
• Allowed characters are uppercase and lowercase standard Latin letters (A-Z and a-z),
decimal digits (0-9), dots (.), hyphens (-), and underscores (_).

4. Type a virtual IP address for the cluster in the Cluster External IP field.
This parameter is required for Hyper-V clusters and is optional for vSphere and AHV clusters.
You can connect to the external cluster IP address with both the web console and nCLI. In the event
that a Controller VM is restarted or fails, the external cluster IP address is relocated to another
Controller VM in the cluster.

5. (Optional) If you want to enable redundancy factor 3, set Cluster Max Redundancy Factor to 3.
Redundancy factor 3 has the following requirements:
• A cluster must have at least five nodes, blocks, racks for redundancy factor 3 to be enabled.
• For guest VMs to tolerate the simultaneous failure of two nodes or drives in different blocks, the
data must be stored on storage containers with replication factor 3.
• Controller VM must be configured with enough memory to support redundancy factor 3. See the
Acropolis Advanced Administration Guide topic CVM Memory Configurations for Features.

6. Type the appropriate DNS and NTP addresses in the respective fields.

Note: You must enter NTP servers that the Controller VMs can reach in the CVM NTP Servers field. If
reachable NTP servers are not entered or if the time on the Controller VMs is ahead of the current time,
cluster services may fail to start.

For Hyper-V clusters, the CVM NTP Servers parameter must be set to the IP addresses of one or
more Active Directory domain controllers.
The Hypervisor NTP Servers parameter is not used in Hyper-V clusters.

7. Type the appropriate subnet masks in the Subnet Mask row.

AOS | Cluster Management | 11


8. Type the appropriate default gateway IP addresses in the Default Gateway row.

9. Select the check box next to each node that you want to add to the cluster.
All unconfigured nodes on the current network are presented on this web page. If you are going to
configure multiple clusters, be sure that you only select the nodes that should be part of the current
cluster.

10. Provide an IP address for all components in the cluster.


Ensure that all components satisfy the cluster subnet requirements. See Cluster IP Address
Configuration on page 9. The use of a DHCP server is not supported for Controller VMs, so make
sure to not use DHCP for Controller VMs.

Note: The unconfigured nodes are not listed according to their position in the block. Ensure that you
assign the intended IP address to each node.

11. Click Create.


Wait until the Log Messages section of the page reports that the cluster has been successfully
configured.
Output similar to the following indicates successful cluster configuration.
Configuring IP addresses on node 13SM71450003/A...
Configuring IP addresses on node 13SM71450003/A...
Configuring IP addresses on node 13SM71450003/A...
Configuring IP addresses on node 13SM71450003/A...
Configuring the Hypervisor DNS settings on node 13SM71450003/A...
Configuring the Hypervisor DNS settings on node 13SM71450003/A...
Configuring the Hypervisor DNS settings on node 13SM71450003/A...
Configuring the Hypervisor DNS settings on node 13SM71450003/A...
Configuring the Hypervisor NTP settings on node 13SM71450003/A...
Configuring the Hypervisor NTP settings on node 13SM71450003/A...
Configuring the Hypervisor NTP settings on node 13SM71450003/A...
Configuring the Hypervisor NTP settings on node 13SM71450003/A...
Configuring Zeus on node 13SM71450003/A...
Configuring Zeus on node 13SM71450003/A...
Configuring Zeus on node 13SM71450003/A...
Configuring Zeus on node 13SM71450003/A...
Initializing cluster...
Cluster successfully initialized!
Initializing the CVM DNS and NTP servers...
Successfully updated the CVM NTP and DNS server list

The cluster is started automatically after creation.

12. Log on to any Controller VM in the cluster with SSH.

13. Verify that all services are up on all Controller VMs.


nutanix@cvm$ cluster status

If the cluster is running properly, output similar to the following is displayed for each node in the
cluster:
CVM: 10.1.64.60 Up
Zeus UP [5362, 5391, 5392, 10848,
10977, 10992]
Scavenger UP [6174, 6215, 6216, 6217]
SSLTerminator UP [7705, 7742, 7743, 7744]
SecureFileSync UP [7710, 7761, 7762, 7763]
Medusa UP [8029, 8073, 8074, 8176,
8221]
DynamicRingChanger UP [8324, 8366, 8367, 8426]

AOS | Cluster Management | 12


Pithos UP [8328, 8399, 8400, 8418]
Hera UP [8347, 8408, 8409, 8410]
Stargate UP [8742, 8771, 8772, 9037,
9045]
InsightsDB UP [8774, 8805, 8806, 8939]
InsightsDataTransfer UP [8785, 8840, 8841, 8886,
8888, 8889, 8890]
Ergon UP [8814, 8862, 8863, 8864]
Cerebro UP [8850, 8914, 8915, 9288]
Chronos UP [8870, 8975, 8976, 9031]
Curator UP [8885, 8931, 8932, 9243]
Prism UP [3545, 3572, 3573, 3627,
4004, 4076]
CIM UP [8990, 9042, 9043, 9084]
AlertManager UP [9017, 9081, 9082, 9324]
Arithmos UP [9055, 9217, 9218, 9353]
Catalog UP [9110, 9178, 9179, 9180]
Acropolis UP [9201, 9321, 9322, 9323]
Atlas UP [9221, 9316, 9317, 9318]
Uhura UP [9390, 9447, 9448, 9449]
Snmp UP [9418, 9513, 9514, 9516]
SysStatCollector UP [9451, 9510, 9511, 9518]
Tunnel UP [9480, 9543, 9544]
ClusterHealth UP [9521, 9619, 9620, 9947,
9976, 9977, 10301]
Janus UP [9532, 9624, 9625]
NutanixGuestTools UP [9572, 9650, 9651, 9674]
MinervaCVM UP [10174, 10200, 10201, 10202,
10371]
ClusterConfig UP [10205, 10233, 10234, 10236]
APLOSEngine UP [10231, 10261, 10262, 10263]
APLOS UP [10343, 10368, 10369, 10370,
10502, 10503]
Lazan UP [10377, 10402, 10403, 10404]
Orion UP [10409, 10449, 10450, 10474]
Delphi UP [10418, 10466, 10467, 10468]

Verifying IPv6 Link-Local Connectivity

The automated IP address and cluster configuration utilities depend on IPv6 link-local addresses, which
are enabled on most networks. Use this procedure to verify that IPv6 link-local is enabled.

1. Connect two Windows, Linux, or Apple laptops to the switch to be used.

2. Disable any firewalls on the laptops.

3. Verify that each laptop has an IPv6 link-local address.


• Windows (Control Panel)
Start > Control Panel > View network status and tasks > Change adapter settings > Local
Area Connection > Details

AOS | Cluster Management | 13


• Windows (command-line interface)
> ipconfig

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . : corp.example.com


Link-local IPv6 Address . . . . . : fe80::ed67:9a32:7fc4:3be1%12
IPv4 Address. . . . . . . . . . . : 172.16.21.11
Subnet Mask . . . . . . . . . . . : 255.240.0.0
Default Gateway . . . . . . . . . : 172.16.0.1

• Linux
$ ifconfig eth0

eth0 Link encap:Ethernet HWaddr 00:0c:29:dd:e3:0b


inet addr:10.2.100.180 Bcast:10.2.103.255 Mask:255.255.252.0
inet6 addr: fe80::20c:29ff:fedd:e30b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2895385616 errors:0 dropped:0 overruns:0 frame:0
TX packets:3063794864 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2569454555254 (2.5 TB) TX bytes:2795005996728 (2.7 TB)

• Mac OS
$ ifconfig en0

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500


ether 70:56:81:ae:a7:47
inet6 fe80::7256:81ff:feae:a747 en0 prefixlen 64 scopeid 0x4
inet 172.16.21.208 netmask 0xfff00000 broadcast 172.31.255.255
media: autoselect
status: active

Note the IPv6 link-local addresses, which always begin with fe80. Omit the / character and anything
following.

AOS | Cluster Management | 14


4. From one of the laptops, ping the other laptop.
• Windows
> ping -6 ipv6_linklocal_addr%interface

• Linux/Mac OS
$ ping6 ipv6_linklocal_addr%interface

• Replace ipv6_linklocal_addr with the IPv6 link-local address of the other laptop.
• Replace interface with the interface identifier on the other laptop (for example, 12 for Windows, eth0
for Linux, or en0 for Mac OS).
If the ping packets are answered by the remote host, IPv6 link-local is enabled on the subnet. If the
ping packets are not answered, ensure that firewalls are disabled on both laptops and try again before
concluding that IPv6 link-local is not enabled.

5. Reenable the firewalls on the laptops and disconnect them from the network.
Results
• If IPv6 link-local is enabled on the subnet, you can use automated IP address and cluster configuration
utility.
• If IPv6 link-local is not enabled on the subnet, you have to manually set IP addresses and create the
cluster.

Note: IPv6 connectivity issue might occur if mismatch occurs because of VLAN tagging. This issue might occur
because ESXi that is shipped from the factory does not have VLAN tagging, hence it might have VLAN tag as
0. The workstation (laptop) that you have connected might be connected to access port, so it might use different
VLAN tag. Hence, ensure that ESXi port must be in the trunking mode.

Failing from one Site to Another


Disaster failover

Connect to the backup site and activate it.


ncli> pd activate name="pd_name"

This operation does the following:


1. Restores all VM files from last fully-replicated snapshot.
2. Registers VMs on recovery site.
1. All the VMs are registered on a single host in the cluster.
2. The VMs are not powered on automatically. You need to manually start the VMs.
3. It is recommended to enable DRS in the cluster so that the hypervisor migrates the VMs them once
they are powered on.

Caution: The VM registration might fail if the storage container is not mounted on the selected host.

3. Marks the failover site protection domain as active.

Planned failover

Connect to the primary site and specify the failover site to migrate to.
ncli> pd migrate name="pd_name" remote-site="remote_site_name2"

This operation does the following:

AOS | Cluster Management | 15


1. Creates and replicates a snapshot of the protection domain.
2. Shuts down VMs on the local site.
3. Creates and replicates another snapshot of the protection domain.
4. Unregisters all VMs and removes their associated files.
5. Marks the local site protection domain as inactive.
6. Restores all VM files from the last snapshot and registers them on the remote site.
7. Marks the remote site protection domain as active.

Fingerprinting Existing vDisks


The vDisk manipulator utility fingerprints vDisks that existed in the cluster before deduplication was
enabled.
Before you begin
The storage container must have fingerprint-on-write enabled.

Run the vDisk manipulator utility from any Controller VM in the cluster.
• To fingerprint a particular vDisk:
nutanix@cvm$ vdisk_manipulator --operation="add_fingerprints" \
--stats_only="false" --nfs_container_name="ctr_name" \
--nfs_relative_file_path="vdisk_path"

• Replace ctr_name with the name of the storage container where the vDisk to fingerprint resides.
• Replace vdisk_path with the path of the vDisk to fingerprint relative to the storage container path (for
example, Win7-desktop11/Win7-desktop11-flat.vmdk). You cannot specify multiple vDisks
in this parameter.
• To fingerprint all vDisks in the cluster:
nutanix@cvm$ ncli vdisk list | grep "Name.*NFS" | awk -F: \
'{print $4 ":" $5 ":" $6 ":" $7}' >> fingerprint.txt
nutanix@cvm$ for i in `cat fingerprint.txt`; do vdisk_manipulator --vdisk_name=
$i \
--operation="add_fingerprints" --stats_only=false; done

Note: You can run vdisk_manipulator in a loop to fingerprint multiple vDisks, but run only one instance
of vdisk_manipulator on each Controller VM at a time. Executing multiple instances on a Controller VM
concurrently would generate significant load on the cluster.
2
Changing Passwords
Changing User Passwords
You can change user passwords, including for the default admin user, in the web console or nCLI.
Changing the password through either interface changes it for both.

To change a user password, do one of the following:


Procedure

• (Web console) Log on to the web console as the user whose password
is to be changed and select Change Password from the user icon

pull-down list of the main menu.


For more information about changing properties of the current users, see the Web Console Guide.
• (nCLI) Specify the username and passwords.
$ ncli -u 'username' -p 'old_pw' user change-password current-
password="curr_pw" \
new-password="new_pw"

• Replace username with the name of the user whose password is to be changed.
• Replace curr_pw with the current password.
• Replace new_pw with the new password.

AOS | Changing Passwords | 17


Note: If you change the password of the admin user from the default, you must specify the password every
time you start an nCLI session from a remote system. A password is not required if you are starting an nCLI
session from a Controller VM where you are already logged on.

Changing the SMI-S Provider Password (Hyper-V)


If you change the password of the Prism admin user, you have to update the Prism run-as account in
SCVMM.

1. Log on to the system where the SCVMM console is installed and start the console.

2. Go to Settings > Security > Run As Account.

3. Right-click the account named cluster_name-Prism and select Properties.

Figure 2: Prism Run As Account in SCVMM

4. Update the username and password to include the new credentials and ensure that Validate domain
credentials is not checked.

5. Go to Fabric > Storage > Providers.

6. Right-click the provider with Name cluster_name and select Refresh.

Figure 3: Storage Provider

AOS | Changing Passwords | 18


Changing the Controller VM Password (Nutanix User)
Perform these steps on any one Controller VM in the cluster to change the password of the nutanix user.
After you have successfully changed the password, the new password is synchronized across all Controller
VMs in the cluster.

1. Log on to the Controller VM with SSH as the nutanix user.

2. Change the nutanix user password.


nutanix@cvm$ passwd

3. Respond to the prompts, providing the current and new nutanix user password.
Changing password for nutanix.
Old Password:
New password:
Retype new password:
Password changed.

Note: The password must meet the following complexity requirements:


• At least 8 characters long
• At least 1 lowercase letter
• At least 1 uppercase letter
• At least 1 number
• At least 1 special character
• At least 4 characters difference from the old password
• Must not be among the last 10 passwords
• Must not be longer than 199 characters

Changing the Controller VM Password (Admin User)


Perform these steps on any one Controller VM in the cluster to change the password of the admin user.
After you have successfully changed the password, the new password is synchronized across all Controller
VMs in the cluster.

1. Log on to the Controller VM with SSH as the admin user.

2. Change the admin user password.


admin@cvm$ passwd

3. Respond to the prompts, providing the current and new admin user password.
Changing password for admin.
Old Password:
New password:
Retype new password:
Password changed.

Note: The password must meet the following complexity requirements:


• At least 8 characters long
• At least 1 lowercase letter

AOS | Changing Passwords | 19


• At least 1 uppercase letter
• At least 1 number
• At least 1 special character
• At least 4 characters difference from the old password
• Must not be among the last 5 passwords
• Must not have more than 2 consecutive occurrences of a character
• Must not be longer than 199 characters
3
Cluster IP Address Configuration

AOS includes a web-based configuration tool that automates the modification of Controller VM IP
addresses and configures the cluster to use these new IP addresses. Other cluster components must be
modified manually.

Network Configuration (Virtual Interfaces, Virtual Switches, and IP


Addresses)
By default, Nutanix hosts have the following virtual switches:
Internal Virtual Switch
The internal virtual switch manages network communications between the Controller VM and the
hypervisor host. This switch is associated with a private network on the default VLAN and uses the
192.168.5.0/24 address space. The traffic on this subnet is typically restricted to the internal virtual switch,
but might be sent over the physical wire, through a host route, to implement storage high availability on
ESXi and Hyper-V clusters. This traffic is on the same VLAN as the Nutanix storage backplane.

Note: For guest VMs and other devices on the network, do not use a subnet that overlaps with the
192.168.5.0/24 subnet on the default VLAN. If you want to use an overlapping subnet for such devices, make
sure that you use a different VLAN.

The following tables list the interfaces and IP addresses on the internal virtual switch on different
hypervisors:

Table 3: Interfaces and IP Addresses on the Internal Virtual Switch virbr0 on an AHV Host

Device Interface Name IP Address

AHV Host virbr0 192.168.5.1

Controller VM eth1 192.168.5.2

eth1:1 192.168.5.254

AOS | Cluster IP Address Configuration | 21


Table 4: Interfaces and IP Addresses on the Internal Virtual Switch vSwitchNutanix on an ESXi Host

Device Interface Name IP Address

ESXi Host vmk1 192.168.5.1

Controller VM eth1 192.168.5.2

eth1:1 192.168.5.254

Table 5: Interfaces and IP Addresses on the Internal Virtual Switch InternalSwitch on a Hyper-V Host

Device Interface Name IP Address

Hyper-V Host vEthernet (InternalSwitch) 192.168.5.1

Controller VM eth1 192.168.5.2


eth1:1 192.168.5.254

External Virtual Switch


The external virtual switch manages communication between the virtual machines, between the virtual
machines and the host, and between the hosts in the cluster. The traffic on this virtual switch also includes
Controller VM–driven replication traffic for the purposes of maintaining the specified replication factor, as
well as any ADSF traffic that cannot be processed locally. The external switch is assigned a NIC team or
bond as the means to provide connectivity outside of the host.

Note: Make sure that the hypervisor and Controller VM interfaces on the external virtual switch are not assigned
IP addresses from the 192.168.5.0/24 subnet.

The following tables list the interfaces and IP addresses on the external virtual switch on different
hypervisors:

Table 6: Interfaces and IP Addresses on the External Virtual Switch br0 on an AHV Host

Device Interface Name IP Address

AHV Host br0 User-defined

Controller VM eth0 User-defined

Guest VM br0 or user-defined Open vSwitch User-defined


bridge)

Table 7: Interfaces and IP Addresses on the External Virtual Switch vSwitch0 on an ESXi Host

Device Interface Name IP Address

ESXi Host vmk0 User-defined

Controller VM eth0 User-defined

Guest VM vSwitch0 or user-defined switch User-defined

Table 8: Interfaces and IP Addresses on the External Virtual Switch ExternalSwitch on a Hyper-V Host

Device Interface Name IP Address

Hyper-V Host vEthernet (ExternalSwitch) User-defined

AOS | Cluster IP Address Configuration | 22


Device Interface Name IP Address

Controller VM eth0 User-defined

Guest VM vEthernet (ExternalSwitch) or user- User-defined


defined switch

Changing Controller VM IP Addresses

Warning: If you are reassigning a Controller VM IP address to another Controller VM, you must perform
this complete procedure twice: once to assign intermediate IP addresses and again to assign the desired IP
addresses.
For example, if Controller VM A has IP address 172.16.0.11 and Controller VM B has IP address
172.16.0.10 and you want to swap them, you would need to reconfigure them with different IP
addresses (such as 172.16.0.100 and 172.16.0.101) before changing them to the IP addresses in
use initially.

Perform one of the following to change the Controller VM IP address:


• Perform the following if IPv6 is enabled on the subnet:
1. Place the cluster in reconfiguration mode by following Preparing to Change Controller VM IP
Addresses on page 24.
2. Change the Controller VM IP addresses by following Changing Controller VM IP Addresses on
page 26.
3. Complete cluster reconfiguration by following Completing Controller VM IP Address Change on
page 27.
• If IPv6 is not enabled on the subnet, follow Changing a Controller VM IP Address (manual) on
page 28.

Preparing to Set IP Addresses


Before you configure the cluster, check that these requirements are met.
Procedure

• Confirm that IPv6 link-local is enabled on the subnet.


IPv6 link-local is required only for discovery of nodes. It is not required after cluster creation except to
add nodes to an existing cluster.
• Confirm that the system you are using to configure the cluster meets the following requirements:
• IPv6 link-local enabled.
• Windows 7, Vista, or MacOS.
• (Windows only) Bonjour installed (included with iTunes or downloadable from http://
support.apple.com/kb/DL999).
• (Hyper-V only) Confirm that the hosts have only one type of NIC (10 GbE or 1 GbE) connected during
cluster creation. If the nodes have multiple types of network interfaces connected, disconnect them until
after you join the hosts to the domain.
• Determine the IPv6 service name of any Controller VM in the cluster.
The service name depends on a unique identifier for the system.

AOS | Cluster IP Address Configuration | 23


Table 9: Nutanix Serial Number

IPv6 service names are uniquely


generated at the factory and have the
following form (note the final period):
NTNX-block_serial_number-node_location-
CVM.local.

On the right side of the block toward the front


is a label that has the block_serial_number (for
example, 12AM3K520060).
The node_location is A for one-node blocks, A-B
for two-node blocks, and A-D for four-node blocks.
If you do not have access to get the block serial
number, see the Nutanix support knowledge base
for alternative methods.

Table 10: Dell Service Tag

IPv6 service names are uniquely


generated at the factory and have the
following form (note the final period):
NTNX-system_service_tag-node_location-
CVM.local.

On the front left side of the system is a slide-out


label that contains the system_service_tag (for
example, B57PW12).
The node_location is A for one-node blocks.

Preparing to Change Controller VM IP Addresses


Before you begin
• Ensure that the cluster NTP and DNS servers are reachable from the new Controller VM addresses.
If different NTP and DNS servers are to be used, remove the existing NTP and DNS servers from the
cluster configuration and add the new ones. If the new addresses are not known, remove the existing
NTP and DNS servers before cluster reconfiguration and add the new ones afterwards.

AOS | Cluster IP Address Configuration | 24


Web Console
> Name Servers

> NTP Servers

nCLI ncli>cluster remove-from-name-servers servers="name_servers"


ncli>cluster add-to-name-servers servers="name_servers"

ncli>cluster remove-from-ntp-servers servers="ntp_servers"


ncli>cluster add-to-ntp-servers servers="ntp_servers"

• Log on to a Controller VM in the cluster and check that all hosts are part of the metadata store.
nutanix@cvm$ ncli host ls | grep "Metadata store status"

For every host in the cluster, Metadata store enabled on the node should be shown.

Warning: If Node marked to be removed from metadata store is displayed, do not proceed with
the IP address reconfiguration, and contact Nutanix support to resolve the issue.

1. Log on to any Controller VM in the cluster with SSH.

2. Stop the Nutanix cluster.


nutanix@cvm$ cluster stop

Wait to proceed until output similar to the following is displayed for every Controller VM in the cluster.
CVM: 172.16.8.191 Up, ZeusLeader
Zeus UP [3167, 3180, 3181, 3182, 3191, 3201]
Scavenger UP [3334, 3351, 3352, 3353]
ConnectionSplicer DOWN []
Hyperint DOWN []
Medusa DOWN []
DynamicRingChanger DOWN []
Pithos DOWN []
Stargate DOWN []
Cerebro DOWN []
Chronos DOWN []
Curator DOWN []
Prism DOWN []
AlertManager DOWN []
StatsAggregator DOWN []
SysStatCollector DOWN []

3. Put the cluster in reconfiguration mode.


nutanix@cvm$ cluster reconfig

Type y to confirm the reconfiguration.


Wait until the cluster successfully enters reconfiguration mode, as shown in the following example.
INFO cluster:185 Restarted Genesis on 172.16.8.189.
INFO cluster:185 Restarted Genesis on 172.16.8.188.
INFO cluster:185 Restarted Genesis on 172.16.8.191.
INFO cluster:185 Restarted Genesis on 172.16.8.190.
INFO cluster:864 Success!

AOS | Cluster IP Address Configuration | 25


Changing Controller VM IP Addresses
Before you begin
Check that the cluster is ready to be configured by following Preparing to Set IP Addresses on page 23.

Warning: If you are reassigning a Controller VM IP address to another Controller VM, you must perform
this complete procedure twice: once to assign intermediate IP addresses and again to assign the desired IP
addresses.
For example, if Controller VM A has IP address 172.16.0.11 and Controller VM B has IP address
172.16.0.10 and you want to swap them, you would need to reconfigure them with different IP
addresses (such as 172.16.0.100 and 172.16.0.101) before changing them to the IP addresses in
use initially.

The cluster must be stopped and in reconfiguration mode before changing the Controller VM IP addresses.

Note: If you are changing the Controller VM IP addresses to another subnet, contact Nutanix Support for
assistance.

1. Open a web browser.


Nutanix recommends using Internet Explorer 9 for Windows and Safari for Mac OS.

Note: Internet Explorer requires protected mode to be disabled. Go to Tools > Internet Options > Security,
clear the Enable Protected Mode check box, and restart the browser.

2. To access the web-based configuration tool, do one of the following in a web browser:
• If the network on the cluster is unsegmented, go to http://[cvm_eth0_ipv6_addr]:2100/
ip_reconfig.html.
Replace [cvm_eth0_ipv6_addr] with the IPv6 address of the eth0 interface of any Controller VM in
the cluster.
• If the network on the cluster is segmented, go to http://[cvm_eth2_ipv6_addr]:2100/
ip_reconfig.html.
Replace [cvm_eth2_ipv6_addr] with the IPv6 address of the eth2 interface of any Controller VM in
the cluster. The workstation must be connected to the same switch as the Nutanix node and must be
on the backplane LAN. If a VLAN has been specified for the backplane LAN, the VLAN ID must be
configured on the switch port to which the workstation is connected.
You can obtain the IPv6 address of a Controller VM interface by using the ifconfig command.
Example
nutanix@cvm$ ifconfig
eth0 Link encap:Ethernet HWaddr 52:54:00:A8:8A:AE
inet addr:10.1.65.240 Bcast:10.1.67.255 Mask:255.255.252.0
inet6 addr: fe80::5054:ff:fea8:8aae/64 Scope:Link
...etc...

The value of the inet6 addr field up to the / character is the IPv6 address of the Controller VM.
For information about network segmentation, see the Network Management chapter of the Prism Web
Console Guide.

3. Update one or more cells on the IP Reconfiguration page.


Ensure that all components satisfy the cluster subnet requirements. See Cluster IP Address
Configuration on page 21.

AOS | Cluster IP Address Configuration | 26


4. Click Reconfigure.

5. Wait until the Log Messages section of the page reports that the cluster has been successfully
reconfigured, as shown in the following example.
Configuring IP addresses on node S10264822116570/A...
Success!
Configuring IP addresses on node S10264822116570/C...
Success!
Configuring IP addresses on node S10264822116570/B...
Success!
Configuring IP addresses on node S10264822116570/D...
Success!
Configuring Zeus on node S10264822116570/A...
Configuring Zeus on node S10264822116570/C...
Configuring Zeus on node S10264822116570/B...
Configuring Zeus on node S10264822116570/D...
Reconfiguration successful!

The IP address reconfiguration disconnects any SSH sessions to cluster components. The cluster is
taken out of reconfiguration mode.

Completing Controller VM IP Address Change

1. If you changed the IP addresses by modifying the Controller VM configuration files directly rather than
using the Nutanix utility, take the cluster out of reconfiguration mode.
Perform these steps for every Controller VM in the cluster.

a. Log on to the Controller VM with SSH.


b. Take the Controller VM out of reconfiguration mode.
nutanix@cvm$ rm ~/.node_reconfigure

c. Restart genesis.
nutanix@cvm$ genesis restart

If the restart is successful, output similar to the following is displayed:


Stopping Genesis pids [1933, 30217, 30218, 30219, 30241]
Genesis started on pids [30378, 30379, 30380, 30381, 30403]

2. Log on to any Controller VM in the cluster with SSH.

3. Start the Nutanix cluster.


nutanix@cvm$ cluster start

If the cluster starts properly, output similar to the following is displayed for each node in the cluster:
CVM: 10.1.64.60 Up
Zeus UP [5362, 5391, 5392, 10848, 10977,
10992]
Scavenger UP [6174, 6215, 6216, 6217]
SSLTerminator UP [7705, 7742, 7743, 7744]
SecureFileSync UP [7710, 7761, 7762, 7763]
Medusa UP [8029, 8073, 8074, 8176, 8221]
DynamicRingChanger UP [8324, 8366, 8367, 8426]
Pithos UP [8328, 8399, 8400, 8418]
Hera UP [8347, 8408, 8409, 8410]

AOS | Cluster IP Address Configuration | 27


Stargate UP [8742, 8771, 8772, 9037, 9045]
InsightsDB UP [8774, 8805, 8806, 8939]
InsightsDataTransfer UP [8785, 8840, 8841, 8886, 8888,
8889, 8890]
Ergon UP [8814, 8862, 8863, 8864]
Cerebro UP [8850, 8914, 8915, 9288]
Chronos UP [8870, 8975, 8976, 9031]
Curator UP [8885, 8931, 8932, 9243]
Prism UP [3545, 3572, 3573, 3627, 4004,
4076]
CIM UP [8990, 9042, 9043, 9084]
AlertManager UP [9017, 9081, 9082, 9324]
Arithmos UP [9055, 9217, 9218, 9353]
Catalog UP [9110, 9178, 9179, 9180]
Acropolis UP [9201, 9321, 9322, 9323]
Atlas UP [9221, 9316, 9317, 9318]
Uhura UP [9390, 9447, 9448, 9449]
Snmp UP [9418, 9513, 9514, 9516]
SysStatCollector UP [9451, 9510, 9511, 9518]
Tunnel UP [9480, 9543, 9544]
ClusterHealth UP [9521, 9619, 9620, 9947, 9976,
9977, 10301]
Janus UP [9532, 9624, 9625]
NutanixGuestTools UP [9572, 9650, 9651, 9674]
MinervaCVM UP [10174, 10200, 10201, 10202,
10371]
ClusterConfig UP [10205, 10233, 10234, 10236]
APLOSEngine UP [10231, 10261, 10262, 10263]
APLOS UP [10343, 10368, 10369, 10370,
10502, 10503]
Lazan UP [10377, 10402, 10403, 10404]
Orion UP [10409, 10449, 10450, 10474]
Delphi UP [10418, 10466, 10467, 10468]

Changing a Controller VM IP Address (manual)

If IPv6 is not enabled on the subnet, manually change the Controller VM IP address for each Controller VM
in the cluster.
Contant Nutanix Support to manually change the Controller VM IP addresses.
If IPv6 is enabled on the subnet, you can change the Controller VM IP address by following Changing
Controller VM IP Addresses on page 26.

Warning: If you are changing the Controller VM IP addresses to another subnet, contact Nutanix Support for
assistance.
4
Creating a Windows Guest VM Failover Cluster
A guest VM failover cluster allows high availability for cluster-aware workloads within the VM and enables
an application to seamlessly failover to another VM on the same host or on a different host.

Note: Windows Guest VM Failover Clustering is for guest VMs only.

This feature is supported on the following environments:


• AHV
• Hyper-V
• ESXi
This feature supports Windows guest failover clustering for the following:
• Windows Server 2008
• Windows Server 2012 R2
• AHV clusters with Volumes

1. Create a volume group, then add a disk to the newly-created volume group. See see Creating a Volume
Group for Use with Volumes.

Note: For best results, Nutanix recommends that you configure 1 vDisk per volume group.

Next, log on to your Windows Server VM to perform the following steps.

2. From the Server Manager, add and enable the Multipath I/O feature in Tools > MPIO.

a. Add support for iSCSI devices by checking the box in the Discovered Multipaths tab.
b. Enable multipath for the targets by checking the box in the Microsoft iSCSI Initiator and selecting
the IP addresses for the Target Portal IP.

3. Set the default load balancing policy for all LUNS to Fail Over Only by running the following PowerShell
cmdlet on each Windows Server 2012 VM that is being used for Windows Failover Clustering:
> Set-MSDSMGlobalDefaultLoadBalancePolicy -Policy FOO

4. From Windows Server, get the iSCSI initiator name. Then, from the Acropolis CLI, attach the external
initiator to the volume group and verify the connection.

a. From the Windows Server Manager VM, in the Disk Management window, click Tools iSCSI
Initiator, then click the Configuration tab and copy the iSCSI initiator name from the text box.
b. Repeat this step for any remaining external initiators. Verify that the external initiators are connected.
<acropolis> vg.get vg_name

Note: You can also create a volume group and enable multiple initiators to access the volume group
by using Prism web console. For more information, see Creating a Volume Group section of Prism Web
Console Guide.

AOS | Creating a Windows Guest VM Failover Cluster | 29


5. Allow the cluster storage to be discoverable by the external initiators.

a. From the Windows Server Manager VM, in the iSCSI Initiator Properties window, click the
Discovery tab.
b. Click Discover Portal, add the data services IP address, and then click OK.
c. To verify that the IP address is connected, go to the Targets tab and click Refresh.
d. Click OK to exit.

6. From the Server Manager, place the target disks online and create a New Simple Volume.

a. In the Disk Management window, right-click each disk and choose the Online option.
Repeat for any remaining disks.
b. Click New Simple Volume Wizard and verify the information in the following windows until you
reach the Format Platform window.
c. Enter a name for the volume in Volume label and complete the remaining wizard steps.

Note: (Optional) If a formatting window appears, you can format the Simple Volume.

7. From the Server Manager, create a Windows Guest VM Failover Cluster and add the disks. For more
details, see Windows Create a failover cluster documentation.

a. In the Server Manager, click Tools > Failover Cluster Manager and click Create Cluster.
b. Click Browse and in the Select Computer window enter the names of the VMs you want to add,
then click OK. Click Yes to validate configuration tests.
c. Verify the information, then enter a name and IP address for the Windows Failover Cluster. Click OK.
d. In the Failover Cluster Manager, click the volume group and click Storage > Disks. Choose Add
disk.
e. Select the disks you want to add to the cluster and click OK.
The new cluster and disks have been created and configured.
5
Acropolis Dynamic Scheduling in AHV
In an Acropolis managed clusters, the Acropolis Dynamic Scheduling (ADS) feature proactively monitors
your cluster for any compute and/or storage I/O contention or hotspots over a period of time. If a problem is
detected, a migration plan is created and executed thereby eliminating hotspots in the cluster by migrating
VMs from one host to another. This feature only detects the contentions that are currently in progress. You
can monitor these tasks from the Task dashboard of the Prism Web console. You can click the VM link to
view the migration information, which includes the migration path (to the destination AHV host).
Some other advantages of the ADS feature are as follows.
• This feature also improves the initial placement of the VMs depending on the VM configuration.
• The Acropolis block services feature uses the ADS feature for balancing sessions of the externally
visible iSCSI targets.

Note: If you have configured any host or VM-host affinity or VM-VM anti-affinity policies, these policies are
honored.

By default, the feature is enabled and it is recommended you keep this feature enabled. However, you
can disable the feature by using aCLI. For disabling ADS feature, see Disabling Acropolis Dynamic
Scheduling on page 31. Even if you disable the feature, the checks for the contentions or hotspots run
in the background and if any anomalies are detected, an alert is raised in the Alerts dashboard after third
notification. However, no action is taken by the ADS feature to resolve these contentions. You need to
manually take the remedial actions or you can enable the feature. For more information about enabling
ADS feature, see Enabling Acropolis Dynamic Scheduling on page 32
Acropolis Dynamic Scheduling Requirements and Limitations
• Ensure that all the hosts are running AOS 5.0 or later releases.
• The iSCSI targets are displayed as an empty entity. However, if any action is taken on an iSCSI target,
the relevant message is displayed in the Tasks dashboard.
• If a problem is detected and the ADS cannot solve the issue (for example, because of limited CPU or
storage resources), the migration plan might fail. In these cases, an alert is generated. You need to
monitor these alerts from the Alerts dashboard of the Prism Web console and take necessary remedial
actions.
• If the host, firmware, or AOS upgrade is in progress and if any resource contention occurs, during the
period of upgrade no resource contention rebalancing is performed.

Disabling Acropolis Dynamic Scheduling


Perform the following procedure to disable the ADS feature. Note that it is not recommended to disable the
ADS feature.

1. Log into the Controller VM in your cluster through an SSH session and access the Acropolis command
line.

2. Disable the ADS feature.


acli> ads.update enable=false

AOS | Acropolis Dynamic Scheduling in AHV | 31


Even after you disable the feature, the checks for the contentions or hotspots run in the background and
if any anomalies are detected, an alert is raised in the Alerts dashboard. However, no action is taken by
the ADS to solve the contentions. You need to manually take the remedial actions or you can enable the
feature.

Enabling Acropolis Dynamic Scheduling


If you have disabled the ADS feature and want to enable the feature, perform the following procedure.

1. Log into the Controller VM in your cluster through an SSH session and access the Acropolis command
line.

2. Enable the ADS feature.


acli> ads.update enable=true
6
Logs
Sending Logs to a Remote Syslog Server
The Nutanix command-line interface (nCLI) command rsyslog-config enables you to send logs from
your Nutanix cluster to a remote syslog server.
• The Command Reference contains more information about rsyslog-config command syntax.
• The Acropolis Advanced Administration Guide troubleshooting topics have more detailed information
about common and AOS logs (such as Stargate and Cassandra logs).
Recommendations and considerations
• As the logs are forwarded from a Controller VM, the logs display the IP address of the Controller VM.
• You can only configure one rsyslog server; you cannot specify multiple servers.
• The remote syslog server is enabled by default.
• Supported transport protocols are TCP and UDP.
• You can also forward logs to a remote syslog server by using Reliable Event Logging Protocol (RELP).
To use RELP logging, ensure that you have installed rsyslog-relp on the remote syslog server.

Note: You can use RELP logging only if the transport protocol is TCP.

• rsyslog-config supports and can report messages from the following Nutanix modules:

Table 11: AOS Module Names for rsyslog-config

Logs are located in /home/nutanix/data/logs (except SYSLOG_MODULE).

Module name With monitor logs disabled, these With monitor logs enabled, these
logs are forwarded logs are also forwarded

SYSLOG_MODULE Controller VM: /home/log/messages Controller VM: /home/log/messages


AHV host: /var/log/messages AHV host: /var/log/messages

• Forwards all the Controller VM logs that are stored in /home/log/messages


to a remote syslog server.
• Forwards all the AHV host logs that are stored in /var/log/messages to a
remote syslog server.

CASSANDRA cassandra/system.log, cassandra_monitor.loglevel


dynamic_ring_changer.out,
dynamic_ring_changer.loglevel

CEREBRO cerebro.loglevel cerebro.out

CURATOR curator.loglevel curator.out

GENESIS genesis.out genesis.out

PRISM prism_gateway.log prism_monitor.loglevel, prism.out

AOS | Logs | 33
Module name With monitor logs disabled, these With monitor logs enabled, these
logs are forwarded logs are also forwarded
STARGATE stargate.loglevel stargate.out

ZOOKEEPER zookeeper.out zookeeper_monitor.loglevel

ACROPOLIS acropolis.out acropolis.out

Forwards all the logs related to the ACROPOLIS service to a remote syslog
server.

Table 12: AOS Log Level Mapping to syslog Log Levels

AOS log levels Contain information from these syslog log


levels

INFO DEBUG, INFO


WARNING NOTICE, WARNING

ERROR ERROR

FATAL CRITICAL, ALERT, EMERGENCY


• SYSLOG_MODULE logs Controller VM messages in /home/log/messages and the AHV host
messages in /var/log/messages. Adding the SYSLOG_MODULE module to the rsyslog
configuration configures rsyslog on compatible AHV hosts. A compatible host must be running an AHV
release later than AHV-20160217.2.
• Enable module logs at the ERROR level, unless you require more information. If you enable more
levels, the rsyslogd daemon sends more messages. For example, if you set the SYSLOG_MODULE
level to INFO, your remote syslog server might receive a large quantity of operating system messages.
• CPU usage might reach 10 percent when the rsyslogd daemon is initially enabled and starts
processing existing logs. This is an expected condition on first use of an rsyslog implementation.

Configuring the Remote Syslog Server Settings


Before you begin
Install the Nutanix command-line interface (nCLI) and connect to a Controller VM in your cluster. See the
Command Reference for details.

Note: As the logs are forwarded from a Controller VM, the logs display the IP address of the Controller VM.

1. As the remote syslog server is enabled by default, disable it while you configure settings.
ncli> rsyslog-config set-status enable=false

2. Create a syslog server (which adds it to the cluster) and confirm it has been created.
ncli> rsyslog-config add-server name=remote_server_name relp-enabled={true |
false} ip-address=remote_ip_address port=port_num network-protocol={tcp | udp}
ncli> rsyslog-config ls-servers

Name : remote_server_name
IP Address : remote_ip_address
Port : port_num
Protocol : TCP or UDP
Relp Enabled : true or false

AOS | Logs | 34
Option
remote_server_name A descriptive name for the remote server receiving the specified
messages
remote_ip_address The remote server's IP address
port_num Destination port number on the remote server.
tcp | udp Choose tcp or udp as the transport protocol
true | false Choose true to enable RELP and choose false to disable RELP

3. Choose a module to forward log information from and specify the level of information to collect.
ncli> rsyslog-config add-module server-name=remote_server_name module-
name=module level=loglevel include-monitor-logs={ false | true }

• Replace module with one of the following:


• ACROPOLIS
• CASSANDRA
• CEREBRO
• CURATOR
• GENESIS
• PRISM
• STARGATE
• SYSLOG_MODULE
• ZOOKEEPER
• Replace loglevel with one of the following:
• DEBUG
• INFO
• NOTICE
• WARNING
• ERROR
• CRITICAL
• ALERT
• EMERGENCY
Enable module logs at the ERROR level unless you require more information.
• (Optional) Set include-monitor-logs to specify whether the monitor logs are sent. It is enabled
(true) by default. If disabled (false), only certain logs are sent.

Note: If enabled, the include-monitor-logs option sends all monitor logs, regardless of the level set
by the level= parameter.

4. Configure additional modules if desired with rsyslog-config add-module.

5. Enable the server.


ncli> rsyslog-config set-status enable=true

Logs are now forwarded to the remote syslog server.

Common Log Files


Nutanix nodes store log files in different directories, depending on the type of information they contain.

AOS | Logs | 35
Nutanix Logs Root
The location for Nutanix logs is /home/nutanix/data/logs.
This location of the logs directory contains all the Nutanix process logs at the INFO, WARNING, ERROR
and FATAL levels. It also contains the directories for the system stats (sysstats), and Cassandra system
logs (cassandra).
The most recent FATAL log only contains the reason for the process to fail. More information can be found
in the other types of logs by analyzing the entries leading up to the failure.

Note: The symbolic link component_name.[INFO|WARNING|ERROR|FATAL] points to the most recent


component log. For example:
stargate.FATAL -> stargate.NTNX-12AM3K490006-2-
CVM.nutanix.log.FATAL.201701-141913.30286

.FATAL Logs
If a component fails, it creates a log file named according to the following convention:
component-name.cvm-name.log.FATAL.date-timestamp
• component-name identifies to the component that created the file, such as Curator or Stargate.
• cvm-name identifies to the Controller VM that created the file.
• date-timestamp identifies the date and time when the first failure within that file occurred.
Each failure creates a new .FATAL log file.
Log entries use the following format:
[IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
The first character indicates whether the log entry is an Info, Warning, Error, or Fatal. The next four
characters indicate the day on which the entry was made. For example, if an entry starts with F0820, it
means that at some time on August 20th, the component had a failure.

Tip: The cluster also creates .INFO and .WARNING log files for each component. Sometimes, the information
you need is stored in one of these files.

Self-Monitoring (sysstats) Logs


Self-monitoring logs are in /home/nutanix/data/logs/sysstats.
The node self-monitors itself by running several Linux tools every few minutes, including ping, iostat,
sar, and df.
This directory contains the output for each of these commands, along with the corresponding timestamp.

/home/nutanix/data/logs/cassandra
This is the directory where the Cassandra metadata database stores its logs. The Nutanix process that
starts the Cassandra database (cassandra_monitor) logs to the /home/nutanix/data/logs
directory. However, the most useful information relating to the Cassandra is found in the system.log* files
located in the /home/nutanix/data/logs/cassandra directory.
This directory contains the output for each of these commands, along with the corresponding timestamp.

Controller VM Log Files


These log files are present on Controller VMs.

AOS | Logs | 36
Table 13: Location: /home/nutanix/data/logs

Log Contents Frequency

alert_manager.[out, ERROR, FATAL, INFO, Alert manager process output


WARNING]

cassandra_monitor.[out, ERROR, FATAL, INFO] Cassandra database monitor


process output

catalina.out Catalina/Tomcat for Prism


process output

cerebro.[out, ERROR, FATAL] DR and replication activity

check-cores.log Core file processing every 1 min

check-fio fio-status output every 1 hour


check-hardware.log Power supply, fan speed, and every 1 min
DIMM temperature status

check_intel.log Intel PCIe-SSD status every 1 min

check-ip-connectivity.log Network connectivity status to every 1 min


IPMI, hypervisor, and Controller
VM of all nodes in the cluster

chronos_node_main.[INFO, ERROR, FATAL, Write-ahead log (WAL) status


WARNING]

connection_splicer.[out, ERROR, FATAL, INFO, Internal process connection status


WARNING]

cron_avahi_monitor.log Avahi process status

cron_time_check.log Check time difference across every 1 min


Controller VMs

curator.[out, ERROR, FATAL, INFO, WARNING] Metadata health and ILM activity

disk_usage.log Disk and inode usage of all every 1 min


partitions on the Controller VM

dynamic_ring_changer.[out, ERROR, FATAL] Metadata migration across nodes


activity

genesis.out Nutanix software start process


output

hyperint_monitor.[out, ERROR, FATAL, INFO, Hypervisor integration activity


WARNING]

pithos.[out, ERROR, FATAL, INFO, WARNING] vDisk configuration activity

prism_gateway.[out, ERROR, FATAL, INFO] Prism leader activity

prism_monitor.[out, ERROR, FATAL, INFO] Prism (Web console, nCLI, REST


API) monitor process output

scavenger.out Log and core file clean-up status

AOS | Logs | 37
Log Contents Frequency

send-email.log E-mail alerts sent from the every 1 min


Controller VM
snmp_manager.out SNMP service logs.

ssh_tunnel.log Connect status to


nsc.nutanix.com for the remote
support tunnel

stargate.[out, ERROR, FATAL, INFO, WARNING] NFS interface activity

stats_aggregator.[out, ERROR, FATAL, INFO] Statistics aggregator process


output

support-info.log Daily automated support (ASUP)


alerts
using-gflags.log gflags status

zeus_config_printer.INFO Contents of cluster configuration


database

zookeeper_monitor.[out, ERROR, INFO] Cluster configuration and cluster


state activity

Table 14: Location: /home/nutanix/data/logs/cassandra

Log Contents

system.log Cassandra system activity

Table 15: Location: /home/nutanix/data/logs/sysstats

Log Contents Frequency Command

df.info Mounted filesystems every 5 sec df -h

disk_usage.INFO Disk usage across disks every 1 hour du

interrupts.INFO CPU interrupts every 5 sec

iostat.INFO I/O activity for each physical disk every 5 sec sudo iostat

iotop.INFO Current I/O in realtime every 5 sec sudo iotop

lsof.INFO List of open files and processes every 1 min sudo lsof
that open them

meminfo.INFO Memory usage every 5 sec cat /proc/


meminfo

metadata_disk_usage.INFO Disk usage for metadata drives every 5 sec

mpstat.INFO CPU activities per CPU every 5 sec mpstat

ntpq.INFO NTP information every 1 min ntpq -pn

ping_gateway.INFO Pings to the default gateway every 5 sec ping

ping_hosts.INFO Pings to all other Controller VMs every 1 min ping

AOS | Logs | 38
Log Contents Frequency Command

sar.INFO Network bandwidth every 5 sec sar -n DEV, -n


EDEV
top.INFO Real-time CPU and memory every 5 sec top
activity

Table 16: Location: /home/nutanix/data/serviceability/alerts

Log Contents
num.processed Alerts that have been processed

Table 17: Location: /var/log

Log Contents
dmesg OS start messages

kernel OS kernel messages

messages OS messages after starting

Correlating the FATAL log to the INFO file


When a process fails, the reason for the failure is recorded in the corresponding FATAL log. There are two
ways to correlate this log with the INFO file to get more information:

1. Search for the timestamp of the FATAL event in the corresponding INFO files.

a. Determine the timestamp of the FATAL event.


b. Search for the timestamp in the corresponding INFO files.
c. Open the INFO file with vi and go to the bottom of the file (Shift+G).
d. Analyze the log entries immediately before the FATAL event, especially any errors or warnings.
In the following example, the latest stargate.FATAL determines the exact timestamp:
nutanix@cvm$ cat stargate.FATAL

Log file created at: 2013/09/07 01:22:23


Running on machine: NTNX-12AM3K490006-2-CVM
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0907 01:22:23.124495 10559 zeus.cc:1779] Timed out waiting for Zookeeper
session establishment

In the above example, the timestamp is F0907 01:22:23, or September 7 at 1:22:23 AM.
Next, grep for this timestamp in the stargate*INFO* files:
nutanix@cvm$ grep "^F0907 01:22:23" stargate*INFO* | cut -f1 -
d:stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130904-220129.7363

This tells us that the relevant file to look at is stargate.NTNX-12AM3K490006-2-


CVM.nutanix.log.INFO.20130904-220129.7363.

AOS | Logs | 39
2. If a process is repeatedly failing, it might be faster to do a long listing of the INFO files and select
the one immediately preceding the current one. The current one would be the one referenced by the
symbolic link.
For example, in the output below, the last failure would be recorded in the file
stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130904-220129.7363.
ls -ltr stargate*INFO*

-rw-------. 1 nutanix nutanix 104857622 Sep 3 11:22


stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130902-004519.7363
-rw-------. 1 nutanix nutanix 104857624 Sep 4 22:01
stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130903-112250.7363
-rw-------. 1 nutanix nutanix 56791366 Sep 5 15:12
stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130904-220129.7363
lrwxrwxrwx. 1 nutanix nutanix 71 Sep 7 01:22 stargate.INFO ->
stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130907-012223.11357
-rw-------. 1 nutanix nutanix 68761 Sep 7 01:33
stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130907-012223.11357

Tip: You can use the procedure above for the other types of files as well (WARNING and ERROR) in order
to narrow the window of information. The INFO file provides all messages, WARNING provides only warning,
error, and fatal-level messages, ERROR provides only error and fatal-level messages, and so on.

Stargate Logs
This section discusses common entries found in Stargate logs and what they mean.
The Stargate logs are located at /home/nutanix/data/logs/stargate.[INFO|WARNING|ERROR|
FATAL].
Log Entry: Watch dog fired
F1001 16:20:49.306397 6630 stargate.cc:507] Watch dog fired

This message is generic and can happen for a variety of reasons. While Stargate is initializing, a watch dog
process monitors it to ensure a successful startup process. If it has trouble connecting to other components
(such as Zeus or Pithos) the watch dog process stops Stargate.
If Stargate is running, this indicates that the alarm handler thread is stuck for longer than 30 seconds. The
stoppage could be due to a variety of reasons, such as problems connecting to Zeus or accessing the
Cassandra database.
To analyze why the watch dog fired, first locate the relevant INFO file, and review the entries leading up to
the failure.
Log Entry: HTTP request timed out
E0820 09:14:05.998002 15406 rpc_client.cc:559] Http request timed out

This message indicates that Stargate is unable to communicate with Medusa. This may be due to a
network issue.
Analyze the ping logs and the Cassandra logs.
Log Entry: CAS failure seen while updating metadata for egroup egroupid or Backend
returns error 'CAS Error' for extent group id: egroupid
W1001 16:22:34.496806 6938 vdisk_micro_egroup_fixer_op.cc:352] CAS failure seen
while updating metadata for egroup 1917333

This is a benign message and usually does not indicate a problem. This warning message means that
another Cassandra node has already updated the database for the same key.

AOS | Logs | 40
Log Entry: Fail-fast after detecting hung stargate ops: Operation with id <opid>
hung for 60secs
F0712 14:19:13.088392 30295 stargate.cc:912] Fail-fast after detecting hung
stargate ops: Operation with id 3859757 hung for 60secs

This message indicates that Stargate restarted because an I/O operation took more than 60 seconds to
complete.
To analyze why the I/O operation took more than 60 seconds, locate the relevant INFO file and review the
entries leading up to the failure.
Log Entry: Timed out waiting for Zookeeper session establishment
F0907 01:22:23.124495 10559 zeus.cc:1779] Timed out waiting for Zookeeper session
establishment

This message indicates that Stargate was unable to connect to Zookeeper.


Review the sysstats/ping_hosts.INFO log to determine if there were any network issues around that
time.
Log Entry: Too many attempts trying to access Medusa
F0601 10:14:47.101438 2888 medusa_write_op.cc:85] Check failed: num_retries_ < 5
(5 vs. 5) Too many attempts trying to access Medusa

This message indicates that Stargate had 5 failed attempts to connect to Medusa/Cassandra.
Review the Cassandra log (cassandra/system.log) to see why Cassandra was unavailable.
Log Entry:multiget_slice() failed with error: error_code while reading n rows
from cassandra_keyspace
E1002 18:51:43.223825 24634 basic_medusa_op.cc:1461] multiget_slice() failed with
error: 4 while reading 1 rows from 'medusa_nfsmap'. Retrying...

This message indicates that Stargate cannot connect to Medusa/Cassandra.


Review the Cassandra log (cassandra/system.log) to see why Cassandra was unavailable.
Log Entry: Forwarding of request to NFS master ip:2009 failed with error
kTimeout.
W1002 18:50:59.248074 26086 base_op.cc:752] Forwarding of request to NFS master
172.17.141.32:2009 failed with error kTimeout

This message indicates that Stargate cannot connect to the NFS master on the node specified.
Review the Stargate logs on the node specified in the error.

Cassandra Logs
After analyzing Stargate logs, if you suspect an issue with Cassandra/Medusa, analyze the Cassandra
logs. This topic discusses common entries found in system.log and what they mean.
The Cassandra logs are located at /home/nutanix/data/logs/cassandra. The most recent file is
named system.log. When the file reaches a certain size, it rolls over to a sequentially numbered file
(example, system.log.1, system.log.2, and so on).
Log Entry: batch_mutate 0 writes succeeded and 1 column writes failed for
keyspace:medusa_extentgroupidmap
INFO [RequestResponseStage:3] 2013-09-10 11:51:15,780 CassandraServer.java
(line 1290) batch_mutate 0 writes succeeded and 1 column writes failed for

AOS | Logs | 41
keyspace:medusa_extentgroupidmap cf:extentgroupidmap row:lr280000:1917645
Failure Details: Failure reason:AcceptSucceededForAReplicaReturnedValue : 1

This is a common log entry and can be ignored. It is equivalent to the CAS errors in the stargate.ERROR
log. It simply means that another Cassandra node updated the keyspace first.
Log Entry: InetAddress /x.x.x.x is now dead.
INFO [ScheduledTasks:1] 2013-06-01 10:14:29,767 Gossiper.java (line 258)
InetAddress /x.x.x.x is now dead.

This message indicates that the node could not communicate with the Cassandra instance at the specified
IP address.
Either the Cassandra process is down (or failing) on that node or there are network connectivity issues.
Check the node for connectivity issues and Cassandra process restarts.
Log Entry: Caught Timeout exception while waiting for paxos read response from
leader: x.x.x.x
ERROR [EXPIRING-MAP-TIMER-1] 2013-08-08 07:33:25,407 PaxosReadDoneHandler.java
(line 64) Caught Timeout exception while waiting for paxos read reponse from
leader: 172.16.73.85. Request Id: 116. Proto Rpc Id : 2119656292896210944. Row
no:1. Request start time: Thu Aug 08 07:33:18 PDT 2013. Message sent to leader
at: Thu Aug 08 07:33:18 PDT 2013 # commands:1 requestsSent: 1

This message indicates that the node encountered a timeout while waiting for the Paxos leader.
Either the Cassandra process is down (or failing) on that node or there are network connectivity issues.
Check the node for connectivity issues or for the Cassandra process restarts.

Prism Gateway Log


This section discusses common entries found in prism_gateway.log and what they mean. This log is
located on the Prism leader. The Prism leader is the node which is running the web server for the Nutanix
UI. This is the log to analyze if there are problems with the UI such as long loading times.
The Prism log is located at /home/nutanix/data/logs/prism_gateway.log on the Prism leader.
To identify the Prism leader, you can run cluster status | egrep "CVM|Prism" and determine
which node has the most processes. In the output below, 10.3.176.242 is the Prism leader.
nutanix@cvm$ cluster status | egrep "CVM|Prism"

2013-09-10 16:06:42 INFO cluster:946 Executing action status on CVMs


10.3.176.240,10.3.176.241,10.3.176.2422013-09-10
16:06:45 INFO cluster:987 Success!
CVM: 10.3.176.240 Up
Prism UP[32655, 32682, 32683, 32687]
CVM: 10.3.176.241 Up
Prism UP[11371, 25913, 25925, 25926]
CVM: 10.3.176.242 Up, ZeusLeader
Prism UP[4291, 4303, 4304, 19468, 20072, 20074,
20075, 20078, 20113]

Log Entry: Error sending request: java.net.NoRouteToHostException: Cannot assign


requested address
The stats_aggregator component periodically issues an RPC request for all Nutanix vdisks in the
cluster. It is possible that all the ephemeral ports are exhausted.

AOS | Logs | 42
The ss -s command shows you the number of open ports.
nutanix@cvm$ ss -s

Total: 277 (kernel 360)


TCP: 218 (estab 89, closed 82, orphaned 0, synrecv 0, timewait 78/0), ports
207

Transport Total IP IPv6


* 360 - -
RAW 1 1 0
UDP 23 13 10
TCP 136 84 52
INET 160 98 62
FRAG 0 0 0

If there are issues with connecting to the Nutanix UI, escalate the case and provide the output of the
ss -s command as well as the contents of prism_gateway.log.

Zookeeper Logs
The Zookeeper logs are located at /home/nutanix/data/logs/zookeeper.out.
This log contains the status of the Zookeeper service. More often than not, there is no need to look at this
log. However, if one of the other logs specifies that it is unable to contact Zookeeper and it is affecting
cluster operations, you may want to look at this log to find the error Zookeeper is reporting.

Genesis.out
When checking the status of the cluster services, if any of the services are down, or the Controller VM is
reporting Down with no process listing, review the log at /home/nutanix/data/logs/genesis.out to
determine why the service did not start, or why Genesis is not properly running.
Check the contents of genesis.out if a Controller VM reports multiple services as DOWN, or if the entire
Controller VM status is DOWN.
Like other component logs, genesis.out is a symbolic link to the latest genesis.out instance and has
the format genesis.out.date-timestamp.
An example of steady state output:
nutanix@cvm$ tail -F ~/data/logs/genesis.out

2017-03-23 19:24:00 INFO node_manager.py:2070 Certificate cache in sync

2017-03-23 19:24:00 INFO node_manager.py:4732 Checking if we need to sync the


local SVM and Hypervisor DNS configuration with Zookeeper

2017-03-23 19:24:38 ERROR lcm_zeus.py:96 Failed to read zknode /appliance/


logical/lcm/operation with error: no node

2017-03-23 19:24:39 INFO framework.py:637 No other LCM operation in progress

2017-03-23 19:26:00 INFO node_manager.py:1960 Certificate signing request data is


not available in Zeus configuration

2017-03-23 19:26:00 INFO node_manager.py:1874 No CA certificates found in the


Zeus configuration

2017-03-23 19:26:00 INFO node_manager.py:1877 No Svm certificates found in the


Zeus configuration

AOS | Logs | 43
2017-03-23 19:26:00 INFO node_manager.py:1880 No Svm certificate maps found in
the Zeus configuration

2017-03-23 19:26:00 INFO node_manager.py:2070 Certificate cache in sync

2017-03-23 19:26:00 INFO node_manager.py:4732 Checking if we need to sync the


local SVM and Hypervisor DNS configuration with Zookeeper

2017-03-23 19:28:00 INFO node_manager.py:1960 Certificate signing request data is


not available in Zeus configuration

2017-03-23 19:28:00 INFO node_manager.py:1874 No CA certificates found in the


Zeus configuration

2017-03-23 19:28:00 INFO node_manager.py:1877 No Svm certificates found in the


Zeus configuration

2017-03-23 19:28:00 INFO node_manager.py:1880 No Svm certificate maps found in


the Zeus configuration

2017-03-23 19:28:00 INFO node_manager.py:2070 Certificate cache in sync

Under normal conditions, the genesis.out file logs the following messages periodically:
Unpublishing service Nutanix Controller
Publishing service Nutanix Controller
Zookeeper is running as [leader|follower]
Prior to these occasional messages, you should see Starting [n]th service. This is an indicator that
all services were successfully started. As of 5.0, there are 34 services.

Tip: You can ignore any INFO messages logged by Genesis by running the command:
grep -v -w INFO /home/nutanix/data/logs/genesis.out

Possible Errors
2017-03-23 19:28:00 WARNING command.py:264 Timeout executing scp -q
-o CheckHostIp=no -o ConnectTimeout=15 -o StrictHostKeyChecking=no
-o TCPKeepAlive=yes -o UserKnownHostsFile=/dev/null -o
PreferredAuthentications=keyboard-interactive,password -o
BindAddress=192.168.5.254 'root@[192.168.5.1]:/etc/resolv.conf' /tmp/
resolv.conf.esx: 30 secs elapsed

2017-03-23 19:28:00 ERROR node_dns_ntp_config.py:287 Unable to download ESX DNS


configuration file, ret -1, stdout , stderr

2017-03-23 19:28:00 WARNING node_manager.py:2038 Could not load the local ESX
configuration

2017-03-23 19:28:00 ERROR node_dns_ntp_config.py:492 Unable to download the ESX


NTP configuration file, ret -1, stdout , stderr

Any of the above messages means that Genesis was unable to log on to the ESXi host using the
configured password.

Diagnosing a Genesis Failure

Determine the cause of a Genesis failure based on the information available in the log files.

AOS | Logs | 44
1. Examine the contents of the genesis.out file and locate the stack trace (indicated by the CRITICAL
message type).

2. Analyze the ERROR messages immediately preceding the stack trace.


...
2017-03-23 19:30:00 INFO node_manager.py:4170 No cached Zeus configuration
found.
2017-03-23 19:30:00 INFO hyperv.py:142 Using RemoteShell ...
2017-03-23 19:30:00 INFO hyperv.py:282 Updating NutanixUtils path
2017-03-23 19:30:00 ERROR hyperv.py:290 Failed to update the NutanixUtils path:
[Errno 104] Connection reset by peer
2017-03-23 19:30:00 CRITICAL node_manager.py:3559 File "/home/nutanix/cluster/
bin/genesis", line 207, in <module>
main(args)
File "/home/nutanix/cluster/bin/genesis", line 149, in main
Genesis().run()
File "/home/nutanix/jita/main/28102/builds/build-danube-4.1.3-stable-release/
python-tree/bdist.linux-x86_64/egg/util/misc/decorators.py", line 40, in
wrapper
File "/home/nutanix/jita/main/28102/builds/build-danube-4.1.3-stable-release/
python-tree/bdist.linux-x86_64/egg/cluster/genesis/server.py", line 132, in run
File "/home/nutanix/jita/main/28102/builds/build-danube-4.1.3-stable-release/
python-tree/bdist.linux-x86_64/egg/cluster/genesis/node_manager.py", line 502,
in initialize
File "/home/nutanix/jita/main/28102/builds/build-danube-4.1.3-stable-release/
python-tree/bdist.linux-x86_64/egg/cluster/genesis/node_manager.py", line 3559,
in discover
...

In the example above, the certificates in AuthorizedCerts.txt were not updated, which means that you
failed to connect to the NutanixHostAgent service on the host.

Note: NutanixHostAgent is Hyper-V specific.

ESXi Log Files


These log files are present on ESXi hosts.

Table 18: Location: /var/logs

Log Contents

hostd.log hostd (daemon to communicate with vmkernel) process output

vmkernel.log vmkernel activity

vpxa.log vpxa (daemon to commmunicate with vCenter) process output

Table 19: Location: /vmfs/volumes/

Log Contents

datastore/vm_name/vmware.log Virtual machine activity and health

Nutanix Calm Log Files


The following table provides Nutanix Calm logs related information.

AOS | Logs | 45
Table 20: Nutanix Calm Log Files

Log Description
/home/docker/nucalm/logs Logs of microservices from Nutanix Calm container.
/home/docker/epsilon/logs Logs of microservices from Epsilon Container.
/home/nutanix/data/logs/genesis.out Logs containing information about enabling
container service and starting Nutanix Calm and
epsilon containers.
/home/nutanix/data/logs/epsilon.out Logs containing information about starting epsilon
service.

Note: This log contains information about epsilon


container crashes.

/home/nutanix/data/logs/nucalm.out Logs containing information about starting Nutanix


Calm service.
/home/docker/docker-latest/plugins/*/ Logs containing the docker volume plug-in, used
rootfs/nvp.log to create or mount the Nutanix Calm volume group
when epsilon container starts. If volumes are not
listed by docker volume ls you can check here
to know why the volume group is not mounted.

/home/log/messages Logs containing information regarding the


communication with Prism web console cluster
for mounting volume group and network
communication issues with the Prism web console
cluster.
7
Troubleshooting Tools
Nutanix Cluster Check (NCC)
Nutanix Cluster Check (NCC) is a framework of scripts that can help diagnose cluster health. NCC can
be run provided that the individual nodes are up, regardless of cluster state. The scripts run standard
commands against the cluster or the nodes, depending on the type of information being retrieved.
When run from the Controller VM command line, NCC generates a log file with the output of the diagnostic
commands selected by the user.
NCC actions are grouped into plugins and modules.
• Plugins are objects that run the diagnostic commands.
• Modules are logical groups of plugins that can be run as a set.

Note: Some plugins run nCLI commands and might require the user to input the nCLI password. The password
is logged on as plain text. If you change the password of the admin user from the default, you must specify the
password every time you start an nCLI session from a remote system. A password is not required if you are
starting an nCLI session from a Controller VM where you are already logged on.

Comprehensive documentation of NCC is available in the Nutanix Command Reference.


NCC Output
Each NCC plugin is a test that completes independently of other plugins. Each test completes with one of
these status types.
PASS
The tested aspect of the cluster is healthy and no further action is required.
FAIL
The tested aspect of the cluster is not healthy and must be addressed.
WARN
The plugin returned an unexpected value and must be investigated.
INFO
The plugin returned an expected value that however cannot be evaluated as PASS/FAIL.
Running Health Checks
In addition to running all health checks, you can checks as follows:
Run all or some checks from the Prism Web Console
• From the Prism web console Health page, select Actions > Run Checks. Select All checks and click
Run.
• If you disable a check in the Prism web console, you cannot run it from the NCC command line unless
you enable it again from the web console.
• You can run NCC checks from the Prism web console for clusters where AOS 5.0 or later and NCC 3.0
or later are installed. You cannot run NCC checks from the Prism web console for clusters where AOS
4.7.x or previous and NCC 3.0 are installed.

AOS | Troubleshooting Tools | 47


• For AOS clusters where it is installed, running NCC 3.0 or later from the command line updates the
Cluster Health score, including the color of the score. For some NCC checks, you can clear the score
by disabling and then re-enabling the check.
Run two or more individual checks at a time
• You can specify two or more individual checks from the command line, with each check separated by a
comma. Ensure you do not use any spaces between checks, only a comma character. For example:
ncc health_checks system_checks \
--plugin_list=cluster_version_check,cvm_reboot_check

Re-run failing checks


• You can re-run any NCC checks or plug-ins that reported a FAIL status.
ncc --rerun_failing_plugins=True

Installing NCC from an Installer File


Before you begin

Note:
• NCC 3.0 is not supported on AOS 4.5.x and previous versions.
• If you are adding one or more nodes to expand your cluster, the latest version of NCC might not
be installed on each newly-added node. In this case, re-install NCC in the cluster after you have
finished adding the one or more nodes.
• This topic describes how to install NCC from the command line. To install NCC software from the
web console, see Upgrading NCC Software on page 50.

Note: To help ensure that Prism Central and each managed cluster are taking advantage of NCC features,
ensure that:
• Each node in your cluster is running the same NCC version.
• Prism Central and each cluster managed by Prism Central are all running the same NCC version.
To check the NCC version and optionally upgrade the NCC software version on Prism Central, see the Prism
Central Guide, Upgrading Nutanix Cluster Check (NCC) on Prism Central topic.

You can download the NCC installation file from the Nutanix support portal under Downloads > Tools &
Firmware. The file type to download depends on the NCC version:

Tip: Note the MD5 value of the file as published on the support portal.

• Some NCC versions include a single installer file (ncc_installer_filename.sh) that you can download and
run from any Controller VM.
• Some NCC versions include an installer file inside a compressed tar file (ncc_installer_filename.tar.gz)
that you must first extract, then run from any Controller VM.
• The directory to which you copy the installation package should exist on all nodes in the cluster (/
home/nutanix is suggested). Additionally, the folder should be owned by any accounts that uses
NCC.

1. Download the installation file to any controller VM in the cluster and copy the installation file to the /
home/nutanix directory.

AOS | Troubleshooting Tools | 48


2. Check the MD5 value of the file. It must match the MD5 value published on the support portal. If the
value does not match, delete the file and download it again from the support portal.
nutanix@cvm$ md5sum ./ncc_installer_filename.sh

3. Perform these steps for NCC versions that include a single installer file (ncc_installer_filename.sh)

a. Make the installation file executable.


nutanix@cvm$ chmod u+x ./ncc_installer_filename.sh

b. Install NCC.
nutanix@cvm$ ./ncc_installer_filename.sh

The installation script installs NCC on each node in the cluster.


NCC installation file logic tests the NCC tar file checksum and prevents installation if it detects file
corruption.
• If it verifies the file, the installation script installs NCC on each node in the cluster.
• If it detects file corruption, it prevents installation and deletes any extracted files. In this case,
download the file again from the Nutanix support portal.

4. Perform these steps for NCC versions that include an installer file inside a compressed tar file
(ncc_installer_filename.tar.gz).

a. Extract the installation package.


nutanix@cvm$ tar xvmf ncc_installer_filename.tar.gz --recursive-unlink

Replace ncc_installer_filename.tar.gz with the name of the compressed installation tar file.
The --recursive-unlink option is needed to ensure old installs are completely removed.
b. Run the install script. Provide the installation tar file name if it has been moved or renamed.
nutanix@cvm$ ./ncc/bin/install.sh [-f install_file.tar]

The installation script copies the install_file.tar tar file to each node and install NCC on each node in
the cluster.

5. Check the output of the installation command for any error messages.
• If installation is successful, a Finished Installation message is displayed. You can check any
NCC-related messages in /home/nutanix/data/logs/ncc-output-latest.log.
• In some cases, output similar to the following is displayed. Depending on the NCC version installed,
the installation file might log the output to /home/nutanix/data/logs/ or /home/nutanix/
data/serviceability/ncc.
Copying file to all nodes
[ DONE ]
-------------------------------------------------------------------------------
+
+---------------+
| State | Count |
+---------------+
| Total | 1 |
+---------------+
Plugin output written to /home/nutanix/data/logs/ncc-output-latest.log
[ info ] Installing ncc globally.
[ info ] Installing ncc on 10.130.45.72, 10.130.45.73

AOS | Troubleshooting Tools | 49


[ info ] Installation of ncc succeeded on nodes 10.130.45.72, 10.130.45.73.

What to do next
• As part of installation or upgrade, NCC automatically restarts the cluster health service on each node
in the cluster, so you might observe notifications or other slight anomalies as the service is being
restarted.

Upgrading NCC Software


Before you begin

Note:
• NCC 3.0 is not supported on AOS 4.5.x and previous versions.
• If you are adding one or more nodes to expand your cluster, the latest version of NCC might not
be installed on each newly-added node. In this case, re-install NCC in the cluster after you have
finished adding the one or more nodes.
• This topic describes how to install NCC software from the Prism web console. To install NCC from
the command line, see Installing NCC from an Installer File on page 48.

Note: To help ensure that Prism Central and each managed cluster are taking advantage of NCC features,
ensure that:
• Each node in your cluster is running the same NCC version.
• Prism Central and each cluster managed by Prism Central are all running the same NCC version.
To check the NCC version and optionally upgrade the NCC software version on Prism Central, see the Prism
Central Guide, Upgrading Nutanix Cluster Check (NCC) on Prism Central topic.

1. Run the Nutanix Cluster Checks (NCC) by using one of the following ways.
• From the Prism web console Health page, select Actions > Run Checks. Select All checks and
click Run.
• Log in to a Controller VM and use the ncc CLI.
nutanix@cvm$ ncc health_checks run_all

If the check reports a status other than PASS, resolve the reported issues before proceeding. If you are
unable to resolve the issues, contact Nutanix support for assistance.

2. Do this step to download and install the NCC upgrade files.

a. Log on to the Prism web console for any node in the cluster.
b. Click the gear icon in the main menu, select Upgrade Software in the Settings page, and then click
the NCC tab.
c. If an update is available, click Upgrade Available and then click Download.
d. When the download is complete, do one of the following:

• To run only the pre-upgrade installation checks on the controller VM where you are logged on, click
Upgrade > Pre-upgrade. These checks also run as part of the upgrade procedure.
• Click Upgrade > Upgrade Now, then click Yes to confirm.
The Upgrade Software dialog box shows the progress of your selection, including pre-installation
checks.

AOS | Troubleshooting Tools | 50


As part of installation or upgrade, NCC automatically restarts the cluster health service on each node
in the cluster, so you might observe notifications or other slight anomalies as the service is being
restarted.

Upgrading NCC by Uploading Binary and Metadata Files

• Do the following steps to download NCC binary and metadata .JSON files from the Nutanix Support
Portal, then upgrade NCC through Upgrade Software in the Prism web console.
• Typically you would need to perform this procedure if your cluster is not directly connected to the
Internet and you cannot download the binary and metadata .JSON files through the Prism web console.

1. Log on to the Nutanix support portal and select Downloads > Tools & Firmware.

2. Click the download link to save the binary gzipped TAR (.tar.gz) and metadata (.json) files on your local
media.

3. Log on to the Prism web console for any node in the cluster.

4. Click the gear icon in the main menu, select Upgrade Software in the Settings page, and then click the
NCC tab.

5. Click the upload the NCC binary link.

6. Click Choose File for the NCC metadata and binary files, respectively, browse to the file locations, and
click Upload Now.

7. When the upload process is completed, click Upgrade > Upgrade Now, then click Yes to confirm.
The Upgrade Software dialog box shows the progress of your selection, including pre-installation
checks.
As part of installation or upgrade, NCC automatically restarts the cluster health service on each node
in the cluster, so you might observe notifications or other slight anomalies as the service is being
restarted.

NCC Usage
The general usage of NCC is as follows:
nutanix@cvm$ ncc ncc-flags module sub-module [...] plugin plugin-flags

Alternately, you can run all or individual checks from the Prism web console. Select Actions > Run
Checks. Select All checks and click Run.
Typing ncc with no arguments yields a table listing the next modules that can be run. The Type column
distinguishes between modules (M) and plugins (P). The Impact tag identifies a plugin as intrusive or non-
intrusive. By default, only non-intrusive checks are used if a module is run with the run_all plugin.
nutanix@cvm$ ncc

+-----------------------------------------------------------------------------+
| Type | Name | Impact | Short help |
+-----------------------------------------------------------------------------+
| M | cassandra_tools | N/A | Plugins to help with Cassandra ring analysis |
| M | file_utils | N/A | Utilities for manipulating files on the |
| | | | cluster |
| M | health_checks | N/A | All health checks |
| M | info | N/A | Contains all info modules (legacy |
| | | | health_check.py) |
+-----------------------------------------------------------------------------+

AOS | Troubleshooting Tools | 51


The usage table is displayed for any module specified on the command line. Specifying a plugin runs its
associated checks.
nutanix@cvm$ ncc info

+------------------------------------------------------------------------------+
| Type | Name | Impact | Short help |
+------------------------------------------------------------------------------+
| M | cluster_info | N/A | Displays summary of info about this cluster. |
| M | cvm_info | N/A | Displays summary of info about this CVM. |
| M | esx_info | N/A | Displays summary of info about the local esx |
| | | | host. |
| M | ipmi_info | N/A | Displays summary of info about the local IPMI.|
| P | run_all | N/A | Run all the plugins in this module |
+------------------------------------------------------------------------------+

The file_utils module does not run any checks. It exists to help you manage files in the cluster.
nutanix@cvm$ ncc file_utils

+------------------------------------------------------------------------------+
| Type | Name | Impact | Short help |
+------------------------------------------------------------------------------+
| P | file_copy | Non-Intrusive | Copies a local file to all CVMs. |
| P | remove_old_cores | Non-Intrusive | Removing cores older than 30 days |
| P | remove_old_fatals | Non-Intrusive | Removing fatals older than 90 days|
| P | run_all | N/A | Run all the plugins in this module|
+------------------------------------------------------------------------------+

Usage Examples
Procedure

• Run all health checks.


nutanix@cvm$ ncc health_checks run_all

• Display default command flags.


nutanix@cvm$ ncc --ncc_interactive=false module sub-module [...] plugin \
--helpshort

• Run the NCC with a named output file and a non-standard path for ipmitool.
nutanix@cvm$ ncc --ncc_plugin_output_history_file=ncc.out health_checks \
hardware_checks ipmi_checks run_all --ipmitool_path /usr/bin/ipmitool

Note: The flags override the default configurations of the NCC modules and plugins. Do not run with these
flags unless your cluster configuration requires these modifications.

Diagnostics VMs
Nutanix provides a diagnostics capability to allow partners and customers to run performance tests on the
cluster. This is a useful tool in pre-sales demonstrations of the cluster and while identifying the source of
performance issues in a production cluster. Diagnostics should also be run as part of setup to ensure that
the cluster is running properly before the customer takes ownership of the cluster.
The diagnostic utility deploys a VM on each node in the cluster. The Controller VMs control the diagnostic
VM on their hosts and report back the results to a single system.
The diagnostics test provide the following data:

AOS | Troubleshooting Tools | 52


• Sequential write bandwidth
• Sequential read bandwidth
• Random read IOPS
• Random write IOPS
Because the test creates new cluster entities, it is necessary to run a cleanup script when you are finished.

Running a Test Using the Diagnostics VMs


Before you begin
• Ensure that 10 GbE ports are active on the ESXi hosts using esxtop or vCenter. The tests run very
slow if the nodes are not using the 10 GbE ports. For more information about this known issue with
ESXi 5.0 update 1, see VMware KB article 2030006.

1. Log on to any Controller VM in the cluster with SSH.

2. Set up the diagnostics test.


nutanix@cvm$ ~/diagnostics/diagnostics.py cleanup

(vSphere only) In vCenter, right-click any diagnostic VMs labeled as "orphaned", select Remove from
Inventory, and click Yes to confirm removal.

3. Start the diagnostics test.


nutanix@cvm$ ~/diagnostics/diagnostics.py run

If the command fails with ERROR:root:Zookeeper host port list is not set, refresh the
environment by running source /etc/profile or bash -l and run the command again.
The diagnostic may take up to 15 minutes to complete for a four-node cluster. Larger clusters take
longer time.
The script performs the following tasks:
1. Installs a diagnostic VM on each node.
2. Creates cluster entities to support the test, if necessary.
3. Runs four performance tests, using the Linux fio utility.
4. Reports the results.

4. Review the results.

5. Remove the entities from this diagnostic.


nutanix@cvm$ ~/diagnostics/diagnostics.py cleanup

(vSphere only) In vCenter, right-click any diagnostic VMs labeled as "orphaned", select Remove from
Inventory, and click Yes to confirm removal.

Diagnostics Output
System output similar to the following indicates a successful test.
Checking if an existing storage pool can be used ...
Using storage pool sp1 for the tests.
Checking if the diagnostics container exists ... does not exist.
Creating a new container NTNX-diagnostics-ctr for the runs ... done.
Mounting NFS datastore 'NTNX-diagnostics-ctr' on each host ... done.
Deploying the diagnostics UVM on host 172.16.8.170 ... done.
Preparing the UVM on host 172.16.8.170 ... done.
Deploying the diagnostics UVM on host 172.16.8.171 ... done.
Preparing the UVM on host 172.16.8.171 ... done.

AOS | Troubleshooting Tools | 53


Deploying the diagnostics UVM on host 172.16.8.172 ... done.
Preparing the UVM on host 172.16.8.172 ... done.
Deploying the diagnostics UVM on host 172.16.8.173 ... done.
Preparing the UVM on host 172.16.8.173 ... done.
VM on host 172.16.8.170 has booted. 3 remaining.
VM on host 172.16.8.171 has booted. 2 remaining.
VM on host 172.16.8.172 has booted. 1 remaining.
VM on host 172.16.8.173 has booted. 0 remaining.
Waiting for the hot cache to flush ... done.
Running test 'Prepare disks' ... done.
Waiting for the hot cache to flush ... done.
Running test 'Sequential write bandwidth (using fio)' ... bandwidth MBps
Waiting for the hot cache to flush ... done.
Running test 'Sequential read bandwidth (using fio)' ... bandwidth MBps
Waiting for the hot cache to flush ... done.
Running test 'Random read IOPS (using fio)' ... operations IOPS
Waiting for the hot cache to flush ... done.
Running test 'Random write IOPS (using fio)' ... operations IOPS
Tests done.

Note:
• Expected results vary based on the specific AOS version and hardware model used.
• The IOPS values reported by the diagnostics script is higher than the values reported by the
Nutanix management interfaces. This difference is because the diagnostics script reports physical
disk I/O, and the management interfaces show IOPS reported by the hypervisor.
• If the reported values are lower than expected, the 10 GbE ports may not be active. For more
information about this known issue with ESXi 5.0 update 1, see VMware KB article 2030006.

Syscheck Utility
Syscheck is a tool that runs load on a cluster and evaluate its performance characteristics. This tool
provides pass or fail feedback on all the checks. The current checks are network throughput and direct
disk random write performance. Syscheck tracks the tests on a per node basis and prints the result at the
conclusion of the test.

Using Syscheck Utility


Perform the following procedure to run the syscheck utility on AOS clusters.

Note:
• Run this test on a newly created cluster or a cluster that is idle or has minimal load.
• Do not run this test if systems are sharing the network as it may interfere with their operation.
• Do not run this test if the guest VMs have already been deployed.

1. Log into the Controller VM.

2. Run the syscheck utility.


nutanix@cvm$ /usr/local/nutanix/syscheck/bin/syscheck

After executing the command, a message that records all the considerations of running this test is
displayed. When prompted with the message, type yes to run the check.
The test returns either pass or fail result. The latest result is placed under /home/nutanix/data/
syscheck directory. An output tar file is also placed in /home/nutanix/data/ directory after every
time you run this utility.
8
Controller VM Memory Configurations
Controller VM memory allocation requirements differ depending on the models and the features that are
being used.

Note: G6/Skylake platforms do not have workload memory requirements for Controller VM and vCPU
configurations, unlike the G4/G5 platforms. G6/Skylake platforms do have Controller VM memory configuration
requirements and recommendations for features. See CVM Memory Configurations for Features on
page 58.

CVM Memory and vCPU Configurations (G5/Broadwell)


This topic lists the recommended Controller VM memory allocations for workload categories.
Controller VM Memory Configurations for Base Models

Note: If the AOS upgrade process detects that any node hypervisor host has total physical memory of 64 GB
or greater, it automatically upgrades any Controller VM in that node with less than 32 GB memory by 4 GB. The
Controller VM is upgraded to a maximum 32 GB.
If the AOS upgrade process detects any node with less than 64 GB memory size, no memory
changes occur.
For nodes with ESXi hypervisor hosts with total physical memory of 64 GB, the Controller VM
is upgraded to a maximum 28 GB. With total physical memory greater than 64 GB, the existing
Controller VM memory is increased by 4 GB.

• To calculate the number of vCPUs for your model, use the number of physical cores per socket in your
model. The minimum number of vCPUS your Controller VM can have is eight and the maximum number
is 12, unless otherwise noted.
• If your CPU has eight or fewer logical cores, allocate a maximum of 75 percent of the cores of a single
CPU to the Controller VM. For example, if your CPU has 6 cores, allocate 4 vCPUs.

Platform Recommended / vCPUs


Default Memory (GB)
Default configuration for all platforms 20 8

Nutanix Broadwell Models


The following table shows the minimum amount of memory required for the Controller VM on each node
for platforms that do not follow the default. For the workload translation into models, see Platform Workload
Translation (G5/Broadwell) on page 56.

Platform Default Memory (GB)


VDI, server virtualization 20
Storage Heavy 28
Storage Only 28
Large server, high-performance, all-flash 32

AOS | Controller VM Memory Configurations | 55


Platform Workload Translation (G5/Broadwell)
The following table maps workload types to the corresponding Nutanix and Lenovo models.
Workload Exceptions

Note: Upgrading to 5.1 requires a 4GB memory increase, unless the CVM memory already has 32 GB.

If all the data disks in a platform are SSDs, the node is assigned the High Performance workload except for
the following exceptions.
• Klas Voyager 2 uses SSDs but due to workload balance, this platform workload default is VDI.
• Cisco B-series is expected to have large remote storage and two SSDs as a local cache for the hot tier,
so this platform workload is VDI.

Workload Nutanix Nutanix Lenovo Cisco Dell Additional


Platforms
Features NX Model SX Model HX Model UCS XC

VDI NX-1065S- SX-1065- HX3310 B200-M4 XC430- Klas


G5 G5 Xpress Telecom
VOYAGER2

NX-1065- - HX3310-F C240-M4L - Crystal


G5
RS2616PS18

NX-3060- - HX2310-E C240-M4S - -


G5

NX-3155G- - HX3510-G C240- - -


G5 M4S2

NX-3175- - HX3710 C220-M4S - -


G5

- - HX1310 C220-M4L - -

- - HX2710-E Hyperflex - -
HX220C-
M4S

- - HX3510- - - -
FG

- - HX3710-F - - -

Storage Heavy NX-6155- - HX5510 - - -


G5

NX-8035- - HX5510-C - - -
G5

NX-6035- - - - - -
G5

Storage Node NX-6035C- - HX5510-C - XC730xd-12C -


G5

High Performance NX-8150- - HX7510 C240- XC630-10P -


and All-Flash G5 M4SX

AOS | Controller VM Memory Configurations | 56


Workload Nutanix Nutanix Lenovo Cisco Dell Additional
Platforms
Features NX Model SX Model HX Model UCS XC

NX-1155- - HX7510-F Hyperflex XC730xd-12R -


G5 HX240C-
M4SX

NX-6155- - - - - -
G5

NX-8150- - - - - -
G5

CVM Memory and vCPU Configurations (G4/Haswell/Ivy Bridge)


This topic lists the recommended Controller VM memory allocations for models and features.
Controller VM Memory Configurations for Base Models

Table 21: Platform Default

Platform Recommended/Default vCPUs


Memory (GB)
Default configuration for all platforms unless 20 8
otherwise noted

The following tables show the minimum amount of memory and vCPU requirements and recommendations
for the Controller VM on each node for platforms that do not follow the default.

Table 22: Nutanix Platforms

Platform Recommended Default Memory vCPUs


Memory (GB) (GB)
NX-1020 16 16 4
NX-6035C 28 28 8
NX-6035-G4 28 20 8
NX-8150 32 32 8
NX-8150-G4 32 32 8
NX-9040 32 20 8
NX-9060-G4 32 32 8

Table 23: Dell Platforms

Platform Recommended Default Memory vCPUs


Memory (GB) (GB)

XC730xd-24 32 20 8
XC6320-6AF
XC630-10AF

AOS | Controller VM Memory Configurations | 57


Table 24: Lenovo Platforms

Platform Recommended/Default vCPUs


Memory (GB)

HX-3500 28 8
HX-5500
HX-7500

CVM Memory Configurations for Features


Note: If the AOS upgrade process detects that any node hypervisor host has total physical memory of 64 GB
or greater, it automatically upgrades any Controller VM in that node with less than 32 GB memory by 4 GB. The
Controller VM is upgraded to a maximum 32 GB.
If the AOS upgrade process detects any node with less than 64 GB memory size, no memory
changes occur.
For nodes with ESXi hypervisor hosts with total physical memory of 64 GB, the Controller VM
is upgraded to a maximum 28 GB. With total physical memory greater than 64 GB, the existing
Controller VM memory is increased by 4 GB.

If each Controller VM in your cluster includes 32 GB of memory, you can enable and use all AOS features
listed here (deduplication, redundancy factor 3, and so on) for each platform type (high performance, all
flash, storage heavy, and so on).
The table shows the extra memory needed plus the minimum Controller VM memory if you are using
or enabling a listed feature. Controller VM memory required = (minimum CVM memory for the node +
memory required to enable features) or 32 GB CVM memory per node, whichever is less.
For example, to use capacity tier deduplication, each Controller VM would need at least 32 GB (20 GB
default + 12 GB for the feature).
To use performance tier deduplication and redundancy factor 3, each Controller VM would need a
minimum 28 GB (20 GB default + 8 GB for the features). However, 32 GB is recommended in this case.

Features Memory (GB)


Capacity tier deduplication (includes performance tier deduplication) 12
Redundancy factor 3 8
Performance tier deduplication 8
Cold-tier nodes + capacity tier deduplication 4
Capacity tier deduplication + redundancy factor 3 12