Académique Documents
Professionnel Documents
Culture Documents
Troubleshooting Guide
Abstract
This guide is for system administrators and experienced users who are familiar with HP 3PAR StoreServ 7000 Storage systems,
understand the operating systems they are using, and have a working knowledge of RAID. This guide provides information on
storage system alerts, components, LEDs, and power procedures.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall
not be liable for technical or editorial errors or omissions contained herein.
Acknowledgments
Microsoft, Windows, Windows XP, and Windows NT are U.S. registered trademarks of Microsoft Corporation.
http://www.hp.com/go/storagewarranty
Contents
1 Identifying Storage System Components........................................................6
Understanding Component Numbering.......................................................................................6
Drive Enclosures...................................................................................................................6
Disk Drive Numbering.....................................................................................................6
Controller Nodes.................................................................................................................7
PCIe Slots and Ports Numbering........................................................................................8
I/O Modules ......................................................................................................................9
Power Cooling Modules........................................................................................................9
Power Distribution Units......................................................................................................10
Service Processor Placement................................................................................................10
2 Understanding LED Indicator Status.............................................................11
Enclosure LEDs.......................................................................................................................11
Bezels LEDs.......................................................................................................................11
Disk Drive LEDs..................................................................................................................12
Storage System Component LEDs..............................................................................................13
Power Cooling Module LEDs................................................................................................13
I/O Modules LEDs.............................................................................................................14
Controller Node and Internal Component LEDs...........................................................................15
Ethernet LEDs....................................................................................................................15
FC Port LEDs......................................................................................................................16
SAS Port LEDs....................................................................................................................17
Interconnect Port LEDs.........................................................................................................17
Fibre Channel Adapter Port LEDs..........................................................................................18
Converged Network Adapter Port LEDs.................................................................................19
3 Powering Off/On the Storage System..........................................................20
Powering Off .........................................................................................................................20
Powering On the Storage System..............................................................................................20
4 Alerts......................................................................................................21
Getting Recommended Actions.................................................................................................21
5 Troubleshooting........................................................................................23
The checkhealth Command......................................................................................................23
Using the checkhealth Command.........................................................................................23
Troubleshooting Storage System Components.............................................................................24
Alert................................................................................................................................24
Format of Possible Alert Exception Messages.....................................................................24
Alert Example...............................................................................................................24
Alert Suggested Action..................................................................................................25
Cage...............................................................................................................................25
Format of Possible Cage Exception Messages...................................................................25
Cage Example 1...........................................................................................................25
Cage Suggested Action 1..............................................................................................25
Cage Example 2...........................................................................................................26
Cage Suggested Action 2..............................................................................................27
Cage Example 3...........................................................................................................27
Cage Suggested Action 3..............................................................................................27
Cage Example 4...........................................................................................................28
Cage Suggested Action 4..............................................................................................28
Cage Example 5...........................................................................................................29
Cage Suggested Action 5..............................................................................................29
Contents 3
Date................................................................................................................................30
Format of Possible Date Exception Messages.....................................................................30
Date Example...............................................................................................................30
Date Suggested Action..................................................................................................30
LD....................................................................................................................................31
Format of Possible LD Exception Messages........................................................................31
LD Example 1...............................................................................................................31
LD Suggested Action 1...................................................................................................31
LD Example 2...............................................................................................................32
LD Suggested Action 2...................................................................................................32
LD Example 3...............................................................................................................32
LD Suggested Action 3...................................................................................................32
LD Example 4...............................................................................................................33
LD Suggested Action 4...................................................................................................33
License.............................................................................................................................33
Format of Possible License Exception Messages.................................................................33
License Example............................................................................................................33
License Suggested Action...............................................................................................34
Network...........................................................................................................................34
Format of Possible Network Exception Messages...............................................................34
Network Example 1......................................................................................................34
Network Suggested Action 1..........................................................................................34
Network Example 2......................................................................................................34
Network Suggested Action 2..........................................................................................34
Node...............................................................................................................................35
Format of Possible Node Exception Messages...................................................................35
Suggested Node Action, General....................................................................................35
Node Example 1..........................................................................................................35
Node Suggested Action 1..............................................................................................36
Node Example 2..........................................................................................................36
Node Suggested Action 2..............................................................................................36
Node Example 3..........................................................................................................37
Node Suggested Action 3..............................................................................................37
PD...................................................................................................................................37
Format of Possible PD Exception Messages.......................................................................38
PD Example 1...............................................................................................................38
PD Suggested Action 1..................................................................................................38
PD Example 2...............................................................................................................39
PD Suggested Action 2..................................................................................................39
PD Example 3...............................................................................................................40
PD Suggested Action 3..................................................................................................40
PD Example 4...............................................................................................................41
PD Suggested Action 4..................................................................................................41
PD Example 5...............................................................................................................41
PD Suggested Action 5..................................................................................................41
PD Example 6...............................................................................................................42
PD Suggested Action 6..................................................................................................42
Port..................................................................................................................................42
Format of Possible Port Exception Messages......................................................................42
Port Suggested Actions, General.....................................................................................42
Port Example 1.............................................................................................................42
Port Suggested Action 1.................................................................................................42
Port Example 2.............................................................................................................44
Port Suggested Action 2.................................................................................................44
Port Example 3.............................................................................................................44
4 Contents
Port Suggested Action 3.................................................................................................44
Port Example 4.............................................................................................................44
Port Suggested Action 4.................................................................................................44
Port Example 5.............................................................................................................45
Port Suggested Action 5.................................................................................................45
RC...................................................................................................................................45
Format of Possible RC Exception Messages.......................................................................45
RC Example.................................................................................................................45
RC Suggested Action.....................................................................................................46
SNMP..............................................................................................................................46
Format of Possible SNMP Exception Messages..................................................................46
SNMP Example............................................................................................................46
SNMP Suggested Action................................................................................................46
Task.................................................................................................................................46
Format of Possible Task Exception Messages.....................................................................46
Task Example...............................................................................................................46
Task Suggested Action...................................................................................................46
VLUN...............................................................................................................................47
Format of Possible VLUN Exception Messages...................................................................47
VLUN Example.............................................................................................................47
VLUN Suggested Action.................................................................................................47
VV...................................................................................................................................48
Format of Possible VV Exception Messages.......................................................................48
VV Suggested Action.....................................................................................................48
Troubleshooting Storage System Setup.......................................................................................48
Storage System Setup Wizard Errors.....................................................................................48
Collecting SmartStart Log Files.............................................................................................54
Collecting Service Processor Log Files...................................................................................54
Contacting HP Support about System Setup...........................................................................54
6 Support and Other Resources.....................................................................56
Contacting HP........................................................................................................................56
HP 3PAR documentation..........................................................................................................56
Typographic conventions.........................................................................................................59
HP 3PAR branding information.................................................................................................59
7 Documentation feedback...........................................................................60
Contents 5
1 Identifying Storage System Components
NOTE: The illustrations in this chapter are used examples only and may not reflect your storage
system configuration.
NOTE: When the enclosures display in the HP 3PAR Management Console or CLI, they are DCS2
for 2U24 (M6710) , DCS1 (M6720) for 4U24, and DCN1 for a node enclosure.
Drive Enclosures
There are two types of drive enclosures. The 4U drive enclosure has 24 horizontal LFF drives with
four columns of six disk drives. The 2U drive enclosure has 24 SFF disk drives arranged vertically
in a single row. The maximum number of supported drive enclosures differs depending on the
model and the number of nodes.
Controller Nodes
The controller node is a component in the storage system that works to cache and manage data
in a system and to provide hosts with a coherent, virtualized view of the system. Controller nodes
are located in the rear of the node enclosure and are numbered 0 and 1 on the bottom and 2 and
3 on the top.
The HP 3PAR StoreServ 7200 Storage system includes two nodes. The HP 3PAR StoreServ 7400
Storage system can include two nodes or four nodes.
(2) 10Gb Converged Network (1) 10Gb CNA each No expansion card
Adapters (CNAs) only
(2) FC HBAs + (2) 10Gb CNAs (1) FC HBA each (1) 10Gb CNA each
1 Ethernet (2)
MGMT connects the public interface with a RJ45 serial cable.
RC connects to Remote Copy
2 Fibre channel
3 SAS (DP2 and DP-1) connects to the drive enclosures and I/O modules
using SAS cables
4 Cluster link Used with four directional interconnect cables that connect the
controller nodes in four node 7400 systems.
I/O Modules
The I/O modules connect the controller nodes to the drives using a SAS cable and help send data
back and forth between the nodes, the drives, PCMs, and enclosures. The I/O modules are located
in the rear of the drive enclosure and are numbered 01 from bottom to top. There are two I/O
modules per enclosure.
NOTE: In the HP M6720 Drive Enclosure (4U24), there are two PCMs that are diagonal from
each other; the remaining PCM slots are filled with blanks.
Enclosure LEDs
Bezels LEDs
The bezels are located at the front of the system on each side of the drive enclosure and include
three LEDs.
2 Module Fault Amber On System hardware fault to I/O modules or PCMs within the enclosure.
At the rear of the enclosure, identify if the PCM or I/O module LED is also
Amber.
Enclosure LEDs 11
Disk Drive LEDs
These LEDs are located on the front of the disk drives.
NOTE: Issue the locatenode command to flash the hotplug LED blue.
Ethernet LEDs
The controller node has two built-in Ethernet ports and each includes two LEDs:
FC Port LEDs
The controller node has two FC ports and each includes two LEDs. The arrow heads shaped LEDs
point to the port they are associated with. The illustration shows the LEDs associated with the FC-1
port.:
FP-1/FP-2 1 and 2 No light Wake up failure (dead device) or power is not applied
1 DP-1 Green Off SAS link is present or not, this LED does not remain lit
Flashing Activity on port
2 DP-2 Green Off SAS link is present or not, this LED does not remain lit
Flashing Activity on port
All ports No light Off Wake up failure (dead device) or power is not
applied
NOTE: The top port (on the far right in the illustration) corresponds with the first set of LEDs
(directly below the second port) and the second port corresponds with the second set of LEDs (the
far left in the illustration) on the bottom of the adapter.
Powering Off
NOTE: PDUs in any expansion cabinets connected to the storage system may need to be shut
off. Use the locatesys command to identify all connected cabinets before shutting down the
system. The command blinks all node and drive enclosure LEDs.
Before you begin, use either SPmaint or SPOCC to shut down and power off the system (see the
section Service Processor Onsite Customer Care in the HP 3PAR StoreServ 7000 Storage Service
Guide):
The system can be shutdown before powering off by any of the following three methods:
1. Using SPOCC:
a. Select InServ Product Maintenenace.
b. Select Halt an InServ cluster/node
c. Follow the prompts to shutdown an Inserv Cluster Do not shut down individual Nodes.
2. Using SPMAINT:
a. Select option 4 (Inserv Product Maintenance)
b. Select Halt an InServ cluster/node
c. Follow the prompts to shutdown an Inserv Cluster Do not shut down individual Nodes.
NOTE: The shutdown procedure must be repeated for each node in the system.
CAUTION: Failure to wait until all controller nodes are in a halted state could cause
the system to view the shutdown as uncontrolled and place the system in a checkld
state upon power up. This can seriously impact host access to data.
NOTE: Message codes ending in de indicate a degraded state alert. Message codes ending in
fa indicate a failed state alert.
See the HP 3PAR OS Command Line Interface Reference for complete information on the display
options on the event logs.
Table 14 Alert Severity Levels
Severity Description
Minor An event has occurred that requires action, but the situation
is not yet serious.
22 Alerts
5 Troubleshooting
The HP 3PAR OS CLI checkhealth command checks and displays the status of storage system
hardware and software components. For example, the checkhealth command can check for
unresolved system alerts, display issues with hardware components, or display information about
virtual volumes that are not optimal.
By default the checkhealth command checks most storage system components, but you can also
check the status of specific components. For a complete list of storage system components analyzed
by the checkhealth command, see Section (page 24).
If there are no faults or exception conditions, the checkhealth command indicates that the system
is healthy:
cli% checkhealth
Checking alert
Checking cage
Checking vlun
Checking vv
System is healthy
With the <component> specifier you can check the status of one or more specific storage system
components. For example:
Alert
Displays any unresolved alerts and shows any alerts that would be seen by showalert -n.
Alert Example
24 Troubleshooting
Alert Suggested Action
View the full Alert output using the IMC (GUI) or the showalert -d CLI command.
Cage
Displays drive cage conditions that are not optimal and reports exceptions if any of the following
do not have normal states:
Ports
Drive magazine states (DC1, DC2, & DC4)
Small Form-factor Pluggable (SFP) voltages (DC2 and DC4)
SFP signal levels (RX power low and TX failure)
Power supplies
Cage firmware (is not current)
Reports if a servicecage operation has been started and has not ended.
Cage Example 1
----------------------------------SFP Info-----------------------------------
FCAL SFP -State- --Manufacturer-- MaxSpeed(Gbps) TXDisable TXFault RXLoss DDM
0 0 OK FINISAR CORP. 4.1 No No Yes Yes
1 1 OK FINISAR CORP. 4.1 No No No Yes
-----------Midplane Info-----------
Firmware_status Current
Product_Rev 2.37
State Normal Op
Loop_Split 0
VendorId,ProductId 3PARdata,DC4
Unique_ID 1062030000098E00
...
Cage Example 2
26 Troubleshooting
Cage cage:1 Power supply 0's AC state is Failed
Cage cage:1 Power supply 2 is Off
-----------Midplane Info-----------
Firmware_status Current
Product_Rev 2.37
State Normal Op
Loop_Split 0
VendorId,ProductId 3PARdata,DC2
Unique_ID 10320300000AD000
Cage Example 3
NOTE: The primary path can be seen by an asterisk (*) in showpd's Ports columns.
-----------Midplane Info-----------
Firmware_status Current
Product_Rev 2.37
State Normal Op
Loop_Split 0
VendorId,ProductId 3PARdata,DC2
Unique_ID 10320300000AD000
cli% showpd -s
Id CagePos Type -State-- -----Detailed_State------
20 1:0:0 FC degraded disabled_B_port,servicing
21 1:0:1 FC degraded disabled_B_port,servicing
22 1:0:2 FC degraded disabled_B_port,servicing
23 1:0:3 FC degraded disabled_B_port,servicing
Cage Example 4
NOTE: DC1 and DC3 cages have firmware in the FCAL modules; DC2 and DC4 cages have
firmware on the cage midplane. Theupgradecage command may be used to upgrade the firmware.
cli% showcage
Id Name LoopA Pos.A LoopB Pos.B Drives Temp RevA RevB Model Side
2 cage2 2:0:3 0 3:0:3 0 24 29-43 2.37 2.37 DC2 n/a
3 cage3 2:0:4 0 3:0:4 0 32 29-41 2.36 2.36 DC2 n/a
28 Troubleshooting
Loop_Split 0
VendorId,ProductId 3PARdata,DC2
Unique_ID 10320300000AD100
cli% showfirmwaredb
Vendor Prod_rev Dev_Id Fw_status Cage_type Firmware_File
...
3PARDATA [2.37] DC2 Current DC2 /opt...dc2/lbod_fw.bin-2.37
Cage Example 5
----------------------------------SFP Info-----------------------------------
FCAL SFP -State- --Manufacturer-- MaxSpeed(Gbps) TXDisable TXFault RXLoss DDM
0 0 OK SIGMA-LINKS 2.1 No No No Yes
1 1 OK FINISAR CORP. 4.1 No No No Yes
Date
Checks the date and time on all nodes and reports an error if they are not the same.
Date Example
cli% showdate
Node Date
0 2010-09-08 10:56:41 PDT (America/Los_Angeles)
1 2010-09-08 10:56:39 PDT (America/Los_Angeles)
cli% shownet
IP Address Netmask/PrefixLen Nodes Active Speed
192.168.56.209 255.255.255.0 0123 0 100
Duplex AutoNeg Status
Full Yes Active
30 Troubleshooting
LD
Displays Logical Disks (LDs) that are not optimal:
Checks for preserved LDs
Checks that current and created availability are the same
Checks for owner and backup
Checks that preserved data space (pdsld's) is the same as total data cache
Checks size and number of logging LDs
LD Example 1
LD Suggested Action 1
Examine the identified LDs using CLI commands such as showld, showld d, showldmap,
showvvmap, and other such commands.
LDs are normally mapped to (used by) VVs but they can be disassociated with a VV if a VV is
deleted without the underlying LDs being deleted, or by an aborted tune operation. Normally, you
would remove the unmapped LD to return its chunklets to the free pool.
LD Suggested Action 2
Examine the identified LDs using CLI commands such as showld, showld d, showldch, and
showpd for any failed or missing disks. Write-through mode (WThru) indicates that host I/O
operations must be written through to the disk before the host I/O command is acknowledged.
This is usually due to a node-down condition, when node batteries are not working, or where disk
redundancy is not optimal.
LD Example 3
LD Suggested Action 3
LDs are created with certain high-availability characteristics, such as ha-cage. If chunklets in an
LD get moved to locations where the Current Availability (CAvail) is not at least as good as the
desired level of availability (Avail), this condition is reported. Chunklets may have been manually
moved with movech or by specifying it during a tune operation or during failure conditions such
as node, path, or cage failures. The HA levels from highest to lowest are port, cage, mag, and
ch (disk).
Examine the identified LDs using CLI commands such as showld, showld d, showldch, and
showpd for any failed or missing disks. In the example below, the LD should have cage-level
32 Troubleshooting
availability, but it currently has chunklet (disk) level availability (that is., the chunklets are on the
same disk).
LD Example 4
LD Suggested Action 4
Preserved data LDs (pdsld's) are created during system initialization (OOTB) and after some
hardware upgrades (via admithw). The total size of the pdsld's should match the total size of all
data-cache in the storage system (see below). This message appears if a node is offline because
the comparison of LD size to data cache size does not match. This message can be ignored unless
all nodes are online. If all nodes are online and the error condition persists, determine the cause
of the failure. Use the admithw command to correct the condition.
cli% shownode
Control Data Cache
Node --Name--- -State- Master InCluster ---LED--- Mem(MB) Mem(MB) Available(%)
0 1001335-0 OK Yes Yes GreenBlnk 2048 4096 100
1 1001335-1 OK No Yes GreenBlnk 2048 4096 100
License
Displays license violations. Returns information if a license is temporary or if it has expired.
License Example
Network
Displays Ethernet issues for the Administrative and Remote Copy over IP (RCIP) networks that have
been logged in the previous 24hour sampling window. Reports if the storage system has fewer
than two nodes with working admin Ethernet connections.
Check whether the number of collisions is greater than 5% of total packets in previous days
log.
Check for Ethernet errors and transmit (TX) or receive (RX) errors in previous days log.
Network Example 1
cli% shownet
IP Address Netmask/PrefixLen Nodes Active Speed Duplex AutoNeg Status
192.168.56.209 255.255.255.0 0123 0 100 Full Yes Changing
192.168.56.233 255.255.255.0 0123 0 100 Full Yes Unverified
Network Example 2
34 Troubleshooting
NOTE: The error counters shown by shownet and shownet -d cannot be cleared except by
rebooting a controller node. Because checkhealth is showing network counters from a history
log, checkhealth stops reporting the issue if there is no increase in error in the next log entry.
shownet -d
IP Address: 192.168.56.209 Netmask 255.255.255.0
Assigned to nodes: 0123
Connected through node 0
Status: Active
Node
Displays node conditions that are not optimal:
Checks if node batteries have been tested in the last 30 days.
Checks for offline nodes.
Checks for power supply and battery problems.
Node Example 1
NOTE: In the example below, the battery state is considered Degraded because the power supply
is Failed; this is normal.
cli% shownode
Control Data Cache
Node --Name--- -State-- Master InCluster ---LED--- Mem(MB) Mem(MB) Available(%)
0 1001356-0 Degraded Yes Yes AmberBlnk 2048 8192 100
1 1001356-1 Degraded No Yes AmberBlnk 2048 8192 100
cli% shownode -s
Node -State-- -Detailed_State-
0 Degraded PS 1 Failed
1 Degraded PS 0 Failed
Node Example 2
NOTE: The condition of the Degraded Power Supply (PS) is due to the battery failing.
cli% shownode
Control Data Cache
Node --Name--- -State-- Master InCluster ---LED--- Mem(MB) Mem(MB) Available(%)
2 1001356-2 OK No Yes GreenBlnk 2048 8192 100
3 1001356-3 Degraded No Yes AmberBlnk 2048 8192 100
cli% shownode -s
Node -State-- -Detailed_State-
2 OK OK
3 Degraded PS 1 Degraded
36 Troubleshooting
cli% showbattery
Node PS Bat Serial -State-- ChrgLvl(%) -ExpDate-- Expired Testing
3 0 0 100A300B OK 100 07/01/2011 No No
3 1 0 12345310 Failed 0 04/07/2011 No No
Node Example 3
showbattery -s
Node PS Bat -State-- -Detailed_State-
0 0 0 OK normal
0 1 0 Degraded Unknown
Examine the date of the last successful test of that battery. Assuming the comment date was
2009-10-14, the last battery test on Node 0, PS 1, Bat 0 was 2009-09-10, which is more
than 30 days ago.
showbattery -log
Node PS Bat Test Result Dur(mins) ---------Time----------
0 0 0 0 Passed 1 2009-10-14 14:34:50 PDT
0 0 0 1 Passed 1 2009-10-28 14:36:57 PDT
0 1 0 0 Passed 1 2009-08-27 06:17:44 PDT
0 1 0 1 Passed 1 2009-09-10 06:19:34 PDT
showbattery
Node PS Bat Serial -State-- ChrgLvl(%) -ExpDate-- Expired Testing
0 0 0 83205243 OK 100 04/07/2011 No No
0 1 0 83202356 Degraded 100 04/07/2011 No No
PD
Displays Physical Disks (PDs) with states or conditions that are not optimal:
Checks for failed and degraded PDs
Checks for an imbalance of PD ports, for example, if Port-A is used on more disks than Port-B
Checks for an Unknown Sparing Algorithm. For example, when it hasn't been set
Checks for disks experiencing a high number of IOPS
Reports if a servicemag operation is outstanding (servicemag status)
Reports if there are PDs that do not have entries in the firwmare DB file
The following checks are performed when the -svc option is used, or on 7400/7200 hardware:
PD Example 1
PD Suggested Action 1
Both degraded and failed disks show up in this report. When an FC path to a drive cage is not
working, all disks in the cage have a state of Degraded due to the nonredundant condition. Use
commands such as showpd, showpd -s, showcage, showcage -d, showport -sfp, and
the like to diagnose further.
38 Troubleshooting
Fibre Channel Info PortA0 PortB0 PortA1 PortB1
Link_Speed 2Gbps -- -- 0Gbps
----------------------------------SFP Info-----------------------------------
FCAL SFP -State- --Manufacturer-- MaxSpeed(Gbps) TXDisable TXFault RXLoss DDM
0 0 OK SIGMA-LINKS 2.1 No No No Yes
1 1 OK SIGMA-LINKS 2.1 No No Yes Yes
PD Example 2
PD Suggested Action 2
The primary and secondary I/O paths for disks (PDs) are balanced between nodes. The primary
path is indicated in the showpd -path output and by an asterisk in the showpd output. An
imbalance of active ports is usually caused by a non-functioning path/loop to a cage, or because
an odd number of drives is installed or detected. To diagnose further, use CLI commands such as
showpd, showpd path, showcage, and showcage -d.
cli% showpd
----Size(MB)----- ----Ports----
Id CagePos Type Speed(K) State Total Free A B
0 0:0:0 FC 10 normal 139520 119040 0:0:1* 1:0:1
1 0:0:1 FC 10 normal 139520 121600 0:0:1 1:0:1*
2 0:0:2 FC 10 normal 139520 119040 0:0:1* 1:0:1
3 0:0:3 FC 10 normal 139520 119552 0:0:1 1:0:1*
...
46 2:9:2 FC 10 normal 139520 112384 2:0:3* 3:0:3
47 2:9:3 FC 10 normal 139520 118528 2:0:3 3:0:3*
48 3:0:0 FC 10 degraded 139520 115200 2:0:4* -----
49 3:0:1 FC 10 degraded 139520 121344 2:0:4* -----
50 3:0:2 FC 10 degraded 139520 115200 2:0:4* -----
51 3:0:3 FC 10 degraded 139520 121344 2:0:4* -----
----------------------------------SFP Info-----------------------------------
FCAL SFP -State- --Manufacturer-- MaxSpeed(Gbps) TXDisable TXFault RXLoss DDM
0 0 OK SIGMA-LINKS 2.1 No No No Yes
1 1 OK SIGMA-LINKS 2.1 No No Yes Yes
PD Example 3
PD Suggested Action 3
This check samples the I/O per second (IOPS) information in statpd to see if any disks are being
overworked, and then it samples again after five seconds. This does not necessarily indicate a
problem, but it could negatively affect system performance. The IOPS thresholds currently set for
this condition are:
NL disks > 75
FC 10K RPM disks > 150
FC 15K RPM disks > 200
SSD > 1500
Operations such as servicemag and tunevv can cause this condition. If the IOPS rate is very
high and/or a large number of disks are experiencing very heavy I/O, examine the system further
40 Troubleshooting
using statistical monitoring commands/utilities such as statpd, the OS IMC (GUI) and System
Reporter. The following example reports disks whose total I/O is 150/sec or more.
PD Example 4
PD Suggested Action 4
The identified disk does not have firmware that the storage system considers current. When a disk
is replaced, the servicemag operation should upgrade the disk's firmware. When disks are
installed or added to a system, the admithw command can perform the firmware upgrade. Check
the state of the disk using CLI commands such as showpd -s, showpd -i, and
showfirmwaredb.
cli% showpd -s 3
Id CagePos Type -State-- -Detailed_State-
3 0:4:0 FC degraded old_firmware
cli% showpd -i 3
Id CagePos State ----Node_WWN---- --MFR-- ---Model--- -Serial- -FW_Rev-
3 0:4:0 degraded 200000186242DB35 SEAGATE ST3146356FC 3QN0290H XRHJ
cli% showfirmwaredb
Vendor Prod_rev Dev_Id Fw_status Cage_type
...
SEAGATE [XRHK] ST3146356FC Current DC2.DC3.DC4
PD Example 5
PD Suggested Action 5
Check the systems Sparing Algorithm value using the CLI command showsys -param. The value
is normally set during the initial installation (OOTB). If it must be set later, use the command setsys
SparingAlgorithm; valid values are Default, Minimal, Maximal, and Custom. After setting the
parameter, use the admithw command to programmatically create and distribute the spare
chunklets.
% showsys -param
System parameters from configured settings
----Parameter----- --Value--
RawSpaceAlertFC : 0
RawSpaceAlertNL : 0
RemoteSyslog : 0
RemoteSyslogHost : 0.0.0.0
SparingAlgorithm : Unknown
PD Suggested Action 6
Check the release notes for mandatory updates and patches to the HP 3PAR OS version that is
installed and install as needed to support this PD in this cage.
Port
Displays port connection issues:
Checks for ports in unacceptable states
Checks for mismatches in type and mode, such as hosts connected to initiator ports, or host
and Remote Copy over Fibre Channel (RCFC) ports configured on the same FC adapter
Checks for degraded SFPs and those with low power; perform this check only if this FC Adapter
type uses SFPs
Port Example 1
42 Troubleshooting
or contaminated FC connection, such as a cable. An alert should identify the condition, such as
the following:
Check SFP statistics using CLI commands such as showport -sfp, showport -sfp -ddm,
showcage, etc.
In the following example an RX power level of 361 microwatts (uW) for Port 0:0:1 DDM is a good
reading; and 98 uW for Port 0:0:2s is a weak reading ( < 100 uW). Normal RX power level
readings are 200-400 uW.
cli% showcage
Id Name LoopA Pos.A LoopB Pos.B Drives Temp RevA RevB Model Side
0 cage0 0:0:1 0 1:0:1 0 15 33-38 08 08 DC3 n/a
1 cage1 --- 0 1:0:2 0 15 30-38 08 08 DC3 n/a
cli% showpd -s
Id CagePos Type -State-- -Detailed_State-
1 0:2:0 FC normal normal
...
13 1:1:0 NL degraded missing_A_port
14 1:2:0 FC degraded missing_A_port
Port Example 3
Port Example 4
cli% showport
N:S:P Mode State ----Node_WWN---- -Port_WWN/HW_Addr- Type
3:5:1 target offline 2FF70002AC00054C 23510002AC00054C free
44 Troubleshooting
Port Example 5
cli% showport
N:S:P Mode State ----Node_WWN---- -Port_WWN/HW_Addr- Type
2:0:1 initiator ready 2FF70002AC000591 22010002AC000591 disk
2:0:2 initiator ready 2FF70002AC000591 22020002AC000591 disk
2:0:3 target ready 2FF70002AC000591 22030002AC000591 disk
2:0:4 target loss_sync 2FF70002AC000591 22040002AC000591 free
cli% showport
N:S:P Mode State ----Node_WWN---- -Port_WWN/HW_Addr- Type
0:1:1 initiator ready 2FF70002AC000190 20110002AC000190 rcfc
0:1:2 initiator loss_sync 2FF70002AC000190 20120002AC000190 free
0:1:3 initiator loss_sync 2FF70002AC000190 20130002AC000190 free
0:1:4 initiator loss_sync 2FF70002AC000190 20140002AC000190 free
RC
Displays Remote Copy issues.
Checks Remote Copy targets
Checks Remote Copy links
Checks Remote Copy Groups and VVs
RC rc:<name> "All links for target <name> are down but target not yet marked failed."
RC Example
SNMP
Displays issues with SNMP. Attempts the showsnmpmgr command and reports errors if the CLI
returns an error.
SNMP -- <err>
SNMP Example
Task
Displays failed tasks. Checks for any tasks that have failed within the past 24 hours. This is the
default time frame for the showtask -failed command.
Task Example
For this example, checkhealth also showed an Alert; this task failed because the command was
entered with a syntax error:
46 Troubleshooting
manually removed with the IMC (GUI) or CLI with removealert or setalert ack. To display
system-initiated tasks, use showtask -all.
VLUN
Displays inactive Virtual LUNs (VLUNs) and those which have not been reported by the host agent.
Reports VLUNs that have been configured but are not currently being exported to hosts or host-ports.
vlun vlun:(<vvID>, <lunID>, <hostname>)"Path to <wwn> is not is not seen by host" vlun
vlun:(<vvID>, <lunID>, <hostname>) "Path to <wwn> is failed"
vlun host:<hostname> "Host <ident>(<type>):<connection> is not connected to a port"
VLUN Example
VLUN Templates
Lun VVName HostName -Host_WWN/iSCSI_Name- Port Type
2 BigVV cs-wintec-test1 ---------------- --- host
VV
Displays Virtual Volumes (VV) that are not optimal. Checks for VVs and Common Provisioning
Groups (CPG) whose state is not normal.
VV Suggested Action
Check status with CLI commands such as showvv, showvv -d, showvv -cpg.
48 Troubleshooting
The error can occur for one of the following reasons:
Network connectivity is lost.
50 Troubleshooting
For information about contacting HP Support, see Contacting HP Support about System Setup
(page 54).
"The storage system found an error while checking port health. Details are listed below."
This error message displays in a dialog box with Retry and Cancel buttons. {0} is the location
of the port with the problem.
"The storage system found an error while checking cabling health. Details are listed below."
This error message displays in a dialog box with Retry and Cancel buttons. The message is
followed by a list of errors. The errors may include:
Cage {0} is connected to the same node twice through ports {1} and {2}. Recable this
cage.
This error displays if a cage is connected to the same node twice. {0} will be the name
of the cage and {1} and {2} will be the port locations where the cage is connected.
Recable the cage using best practices.
"Cage {0} is not connected to the same slot and port on the nodes it is connected to.
Recable this cage."
This message displays if a cage is connected to different slots, ports, and nodes. {0}
will be the name of the cage with the problem. Recable the cage using best practices.
"The storage system found an error while checking cabling health. Details are listed below."
This error message displays in a dialog box with Retry and Cancel buttons. Information listed
below the message is the CLI output for the chechhwconfig command, which occurs when
the SP does not recognize the command, allowing you to see the output.
For information about contacting HP Support, see Contacting HP Support about System Setup
(page 54).
The storage system found an error while checking cage health. The firmware upgrade
succeeded, but cage {0} has not come back. Contact HP support for help."
This error message displays in a dialog box with Retry and Cancel buttons. This error might
occur after the drive cages have had a firmware upgrade. {0} will be the name of the cage
with the problem. Although the firmware upgrade may have succeeded, this error might occur
if the cage does not boot back up. Contact HP Support.
For information about contacting HP Support, see Contacting HP Support about System Setup
(page 54).
"The storage system found an error while checking cage health. Details are listed below."
This error message displays in a dialog box with Retry and Cancel buttons. Information listed
below the message is the CLI output for the chechhwconfig command, which occurs when
the SP does not recognize the command, allowing you to see the output.
For information about contacting HP Support, see Contacting HP Support about System Setup
(page 54).
52 Troubleshooting
"Unable to set the storage system network configuration. The specified IPv4 gateway address
is not reachable by using the specified storage system IPv4 address."
This message displays in a dialog box. The error occurs if the storage system detects that the
defined IPv4 gateway address could not be reached.
Click Back and specify a valid IPv4 gateway address. If the error persists, contact HP Support.
For information about contacting HP Support, see Contacting HP Support about System Setup
(page 54).
"Unable to set the storage system network configuration. The storage system IPv4 address
cannot be the same as the IPv4 gateway."
This message displays in a dialog box. The error occurs if the storage system detects that the
defined IPv4 gateway address is the same as the configured IPv4 address.
Click Back and specify a different address for the IPv4 gateway address.
"Unable to set the storage system network configuration. The specified address is already in
use by another machine."
This message displays in a dialog box. The error occurs if the storage system detects that the
defined IPv4 address is already in use by another machine.
Click Back and specify a different IPv4 address.
"Unable to set the storage system network configuration. The storage system could not be
reached at the new IP address. Make sure your network settings are configured correctly."
This error message displays in a dialog box with Retry and Cancel buttons. This error displays
when the SP is unable to reach the storage system at the new IP address.
Click Cancel to close the wizard, and then begin the setup process again.
"Unable to set the storage system network configuration. The storage system did not recognize
its new IP address as being validated."
This error message displays in a dialog box with Retry and Cancel buttons. This error displays
when the SP reaches the storage system at the new IP but fails to recognize that the SP was
able to do this.
Click Back and specify a valid IP address. if the error persists, contact HP Support.
For information about contacting HP Support, see Contacting HP Support about System Setup
(page 54).
Errors strings for the time setup progress step
"Unable to set the storage system NTP server. An invalid address was specified."
This error message displays in a dialog box. This error displays if the storage system detects
that the NTP address is invalid.
Click Cancel to close the wizard, and then begin the setup process again.
"Unable to set the storage system NTP server. The storage system's admin volume has not
been created. This must be created before any networking information is created. Contact HP
support for help."
This error message displays in a dialog box with Retry and Cancel buttons. This error occurs
if a previous command failed and the wizard did not detect the error, or if the system was
rebooted for any reason during installation.
Click Cancel to close the wizard, and then begin the setup process again.
For information about contacting HP Support, see Contacting HP Support about System Setup
(page 54).
NOTE: You can continue to access the SmartStart log files in the Users folder after you have
removed SmartStart from your system.
54 Troubleshooting
Product model names and numbers
Technical support registration number (if applicable)
Product serial numbers
Error messages
Operating system type and revision level
Detailed questions
When you contact HP, specify that you are requesting support for your StoreServ 7000 Storage
product.
HP 3PAR StoreServ 7200 and 7400 Storage systems StoreServ 7000 Storage
HP 3PAR documentation
For information about: See:
Supported hardware and software platforms The Single Point of Connectivity Knowledge for HP
Storage Products (SPOCK) website:
http://www.hp.com/storage/spock
Using the HP 3PAR Management Console (GUI) to configure HP 3PAR Management Console User's Guide
and administer HP 3PAR storage systems
Using the HP 3PAR CLI to configure and administer storage HP 3PAR Command Line Interface Administrators
systems Manual
Installing and maintaining the Host Explorer agent in order HP 3PAR Host Explorer Users Guide
to manage host configuration and connectivity information
Creating applications compliant with the Common Information HP 3PAR CIM API Programming Reference
Model (CIM) to manage HP 3PAR storage systems
Migrating data from one HP 3PAR storage system to another HP 3PAR-to-3PAR Storage Peer Motion Guide
Configuring the Secure Service Custodian server in order to HP 3PAR Secure Service Custodian Configuration Utility
monitor and control HP 3PAR storage systems Reference
Using the CLI to configure and manage HP 3PAR Remote HP 3PAR Remote Copy Software Users Guide
Copy
Identifying storage system components, troubleshooting HP 3PAR F-Class, T-Class, and StoreServ 10000 Storage
information, and detailed alert information Troubleshooting Guide
Installing, configuring, and maintaining the Secure Service HP 3PAR Policy Manager Software User Guide
Policy Manager HP 3PAR Policy Manager Software Upgrade Guide
HP 3PAR documentation 57
For information about: See:
HP 3PAR 7200 and 7400 storage systems HP 3PAR StoreServ 7000 Storage Site Planning Manual
HP 3PAR 10000 storage systems HP 3PAR StoreServ 10000 Storage Physical Planning
Manual
HP 3PAR StoreServ 10000 Storage Third-Party Rack
Physical Planning Manual
Installing 7200 and 7400 storage systems and initializing HP 3PAR StoreServ 7000 Storage Installation Guide
the Service Processor HP 3PAR StoreServ 7000 Storage SmartStart Software
Users Guide
Maintaining, servicing, and upgrading 7200 and 7400 HP 3PAR StoreServ 7000 Storage Service Guide
storage systems
Troubleshooting 7200 and 7400 storage systems HP 3PAR StoreServ 7000 Storage Troubleshooting Guide
Maintaining the Service Processor HP 3PAR Service Processor Software User Guide
HP 3PAR Service Processor Onsite Customer Care
(SPOCC) User's Guide
Backing up Oracle databases and using backups for disaster HP 3PAR Recovery Manager Software for Oracle User's
recovery Guide
Backing up Exchange databases and using backups for HP 3PAR Recovery Manager Software for Microsoft
disaster recovery Exchange 2007 and 2010 User's Guide
Backing up SQL databases and using backups for disaster HP 3PAR Recovery Manager Software for Microsoft SQL
recovery Server Users Guide
Backing up VMware databases and using backups for HP 3PAR Management Plug-in and Recovery Manager
disaster recovery Software for VMware vSphere User's Guide
Installing and using the HP 3PAR VSS (Volume Shadow Copy HP 3PAR VSS Provider Software for Microsoft Windows
Service) Provider software for Microsoft Windows User's Guide
Best practices for setting up the Storage Replication Adapter HP 3PAR Storage Replication Adapter for VMware
for VMware vCenter vCenter Site Recovery Manager Implementation Guide
Troubleshooting the Storage Replication Adapter for VMware HP 3PAR Storage Replication Adapter for VMware
vCenter Site Recovery Manager vCenter Site Recovery Manager Troubleshooting Guide
Installing and using vSphere Storage APIs for Array HP 3PAR VAAI Plug-in Software for VMware vSphere
Integration (VAAI) plug-in software for VMware vSphere User's Guide
Bold monospace text Commands you enter into a command line interface
System output emphasized for scannability
WARNING! Indicates that failure to follow directions could result in bodily harm or death, or in
irreversible damage to data or to the operating system.
CAUTION: Indicates that failure to follow directions could result in damage to equipment or data.
Required
Indicates that a procedure must be followed as directed in order to achieve a functional and
supported implementation based on testing at HP.
Typographic conventions 59
7 Documentation feedback
HP is committed to providing documentation that meets your needs. To help us improve the
documentation, send any errors, suggestions, or comments to Documentation Feedback
(docsfeedback@hp.com). Include the document title and part number, version number, or the URL
when submitting your feedback.
60 Documentation feedback