Solution ID: kb30541 Last updated: 4 MAY 2010

What do FCP Partner Path Misconfigured messages mean?

View Environment section

Symptoms
How to troubleshoot FCP Partner Path Misconfigured errors. AutoSupport message: FCP PARTNER PATH MISCONFIGURED

Syslog and EMS messages

[hostname: scsitarget.partnerPath.misconfigured:error]: FCP Partner Path Misconfigured. [hostname: scsitarget.partnerPath.misconfigured:error]: FCP Partner Path Misconfigured - Host I/O access through a non -primary and non-optimal path was detected.

Terminology

Partner Path: Any path to LUNs that utilizes the partner node to access a LUN hosted by the local node. The LUNs are in an active-active cluster. Non-primary path: Synonymous with partner path, proxy path and secondary path. All are examples of a nonprimary path. FCP target port: The fibre channel interface that provides FCP service to hosts. Virtual Target Interconnect (VTIC): The virtual FCP target interface seen in the initiator group list. VTIC is used to indicate that the initiator has access to a secondary path.

y y

Problem Description

NetApp active-active clustered storage controllers allow access to logical units (LUNs) through FCP ports on both nodes of the cluster. Hosts should, under normal circumstances, only access LUNs through ports on the cluster node which hosts the LUN. I/O paths that utilize the ports of the cluster node that host the LUN are referred to as primary paths or optimized paths. I/O paths that utilize the partner cluster node are known as secondary paths, partner paths or non-optimized paths. A LUN should only be accessed through the partner cluster node when the primary ports are unavailable. I/O access to LUNs using a secondary path indicates one or both of the following conditions: the primary path(s) between host and storage controller have failed, or host MPIO software is not configured correctly. These conditions indicate that the redundancy and performance of the SAN has been compromised. Corrective action should be taken immediately to restore primary paths to the storage controllers.

In some circumstances, this error may also be triggered by non-I/O activity such as MPIO path management operations or host clustering software performing status checks to LUNs. If either of these situations is determined to be responsible for triggering the error, steps may be taken to decrease this activity or specify custom thresholds for the trigger conditions. Beginning with Data ONTAP 7.2.2, non-read and non-write operations will not trigger this error message.

FCP Partner Path Misconfigured errors may spuriously occur following storage controller boots, storage controller cluster takeover and giveback operations or host reboots. These instances of the error are normal and are usually corrected once the host's MPIO software detects the changed path status. These occurrences of the error may be ignored if they are not continuous. Data ONTAP 7.2.2 and later releases have enhancements to prevent spurious triggers of this error following storage controller boot or cluster takeover or giveback operation.

NetApp now recommends adjusting the thresholds that control the trigger for the FCP Partner Path Misconfigured message to avoid spurious and unnecessary occurrences. Please issue the following commands on both nodes of the storage controller cluster.

options lun.use_partner.cc.warn_limit 300 options lun.use_partner.cc.bytes 2457600

The first option increases the time interval from 10 seconds to 300 seconds and the second option increases the bytes transferred threshold from 512000 bytes to 2457600 bytes.

If the warning message continues after making these changes then please follow the steps outlined in the remainder of this document.

The following diagrams illustrate I/O access through a primary path and non-primary path.

Solution
Identifying Affected LUNs and Hosts The following steps provide a procedure that can be used to identify the I/O access that is responsible for triggering the error. The process of identifying the I/O access begins by identifying the LUNs receiving the I/O through the partner node's FCP target port. With the LUNs identified, the next step is to identify the initiators performing the I/O. Next, identify the FCP target ports on the storage controller that host the LUNs. The host should have access to at least one port on each node of the storage controller cluster. Once the host initiators and primary FCP target ports have been identified, the fabric can be inspected for correct and functioning connections and finally the hosts' multipath configurations can be inspected for correct operation. 1. Identify the LUNs being accessed through the partner node's FCP Target port. a. lun stats -o (LUN STATISTICS) Identify the host initiators that are performing the I/O through the partner path a. lun config_check -A (LUN CONFIG CHECK) b. lun show -v (LUN CONFIGURATION)

2.

c. igroup show -v (INITIATOR GROUPS) 3. Identify the primary storage controller FCP target ports available for access to the LUN. a. fcp show cfmode (FCP CFMODE) b. fcp show adapters (FCP TARGET ADAPTERS) Verify the host initiator connectivity to primary FCP target ports and the host MPIO software configuration. Verify use of the partner path has ceased from both cluster nodes. a. sysstat -b 1

4.

5.

Procedure and Example Data 1. Identify the LUNs being accessed through the partner node's FCP target port as well as the type of operation. The error threshold may be triggered by either the number of kilobytes read and written or the number of nonread and non-write operations performed. Examples of non-read and non-write SCSI operations are Inquiry, Persistent Reserve, Report LUNS, and Test Unit Ready. The default thresholds are 512,000 bytes per second or 300 FCP operations per second. The LUN STATISTICS section of the Autosupport will display the read and write operations for both local and partner paths per LUN. This output may also be obtained using the command lun stats -o. The counters may also be zeroed using the command lun stats -z which is useful for determining how often the counters are increasing. The output of lun stats might show that no counters have exceeded the threshold. In this case, the LUNs that are being accessed by a non-primary path may reside on the partner node. Once the effected LUNs have been identified, continue to Step 2 to locate the responsible host. Example of lun stats -o output from Autosupport: ===== LUN STATISTICS ===== /vol/esx_luns/guest001.lun (32 minutes, 39 seconds) Read (kbytes) Write (kbytes) Read Ops Write Ops Partner Ops Partner KBytes 13510011357 12648494826 707504214 932701251 263445977 2123914089

Other Ops 251651

QFulls 0

In the example above, both Partner Ops and Partner Kbytes have exceeded the threshold in the given time interval. ===== LUN STATISTICS ===== /vol/esx_luns/guest002.lun (10 minutes, 3 seconds) Read (kbytes) Write (kbytes) Read Ops Write Ops Partner Ops Partner KBytes 543897547 235563443 238273 324897 20000 100

Other Ops 251651

QFulls 0

In this example, only the Partner Ops have triggered the error threshold. This may happen when multiple hosts have access to the LUN. The hosts accessing the LUN in this way should be identified and the reasoning for the access evaluated. Possible solutions are to restrict access, or tune the host software so that it will not attempt access through the partner path. 2. Identify the host initiators that are performing the I/O through the partner path. Using the LUNs identified in Step 1, locate the suspect initiators by cross-referencing the LUN configuration with the initiator group mapping. This information can be found in the AutoSupport section titled LUN CONFIGURATION and by using the command lun show -v. Once the LUN's initiator groups have been identified, the initiator WWPNs that are members of the initiator group can be found in the Autosupport section INITIATOR GROUPS and with the command igroup show -v.

Example of lun show -v and igroup show -v from the AutoSupport: ===== LUN CONFIGURATION ===== /vol/esx_luns/guest001.lu n Comment: Serial#: XXXXXXXXX Share: none Space Reservation: enabled Multiprotocol Type: linux Maps: igroupA=0 igroupB=0

2.0t (2194459852800) (r/w, online, mapped)

===== INITIATOR GROUPS ===== igroupA (FCP) (ostype: vmware): 21:00:00:e0:8b:92:da:ef (logged in on: 0a, vtic) igroupB (FCP) (ostype: vmware): 21:00:00:e0:8b:82:d0:09 (logged in on: 0c, vtic) In this example, the LUN /vol/esx_luns/guest001.lun is mapped to two initiator groups, igroupA and igroupB. Each initiator group contains one WWPN. These WWPNs belong to different ports which may be on the same host or on different hosts, depending on the deployment. The WWPNs that were identified in the initiator groups comprise a list of suspect initiators that could be accessing the LUN through a non-primary path. This list of suspect initiators will need to be checked in Step 4. To reduce the suspect list, Data ONTAP 7.2.2 and later releases provide a list of initiators performing I/O access using the partner path. This information can be found in the LUN CONFIG CHECK section of the AutoSupport and the output of the command lun config_check -A. In addition to listing initiator access through the partner path, lun config_check -v will check and display a variety of other possible misconfigurations. Any issues presented in the output should be addressed before proceeding. Example of lun config_check -A from the AutoSupport: ===== LUN CONFIG CHECK ===== The following FCP Initiators are sending Read/Write i/o over the FCP Partner Paths during the last 15 seconds WWPN Partner's Port ops bytes 21:00:00:e0:8b:25:0c:10 0c 34 17408 21:00:00:e0:8b:25:03:66 0c 186 1117696 21:00:00:e0:8b:25:0c:18 0c 1618 10866688 21:00:00:e0:8b:25:0b:b0 0c 1693 13290496 In this example, four initiators are accessing LUNs through the partner path. 3. Identify the primary storage controller FCP target ports available for access to the LUN. In the active-active storage controller configuration, primary versus secondary ports will depend on the cluster failover mode (cfmode) employed. For example, the single_image cfmode allows a single port to be used for primary path access to LUNs hosted on the local node and partner path access for LUNs hosted on the partner node. For information about cfmode and port configurations, please see the Data ONTAP Block Access Management Guide. The cfmode may be identified using the command fcp show cfmode. It can also be located in the FCP CFMODE section of the AutoSupport. ===== FCP CFMODE ===== fcp show cfmode: single_image The FCP target ports can be displayed using the command fcp show adapter and in the AutoSupport section FCP TARGET ADAPTERS. The adapter type shows if the port can be used for I/O access to LUNs on the local node or the partner node. The highlighted lines in the output below will show the differences in the display between cfmode types.

This example shows the target adapters when using the cfmode single_image. When using single_image cfmode, either port may be used to access LUNs hosted on both the local and partner nodes. > fcp show adapters Slot: Description: 2322 (2362) rev. 3) Adapter Type: Status: FC Nodename: FC Portname: Standby: Slot: Description: 2322 (2362) rev. 3) Adapter Type: Status: FC Nodename: FC Portname: Standby:

0c Fibre Channel Target Adapter 0c (Dual -channel, QLogic Local ONLINE 50:0a:09:80:86:17:c3:ac (500a09808617c3ac) 50:0a:09:81:96:17:c3:ac (500a09819617c3ac) No 0d Fibre Channel Target Adapter 0d (Dual -channel, QLogic Local ONLINE 50:0a:09:80:86:17:c3:ac (500a09808617c3ac) 50:0a:09:82:96:17:c3:ac (500a09829617c3ac) No

This example shows the target adapters when using the cfmode partner. In this instance, the ports labeled Local provide access to LUNs hosts on the local node whereas ports labeled Partner allow access to LUNs on the partner node. > fcp show adapters Slot: Description: 2322 (2362) rev. 3) Adapter Type: Status: FC Nodename: FC Portname: Standby: Slot: Description: 2322 (2362) rev. 3) Adapter Type: Status: FC Nodename: FC Portname: Standby:

0c Fibre Channel Target Adapter 0c (Dual -channel, QLogic Local ONLINE 50:0a:09:80:86:67:c2:63 (500a09808667c263) 50:0a:09:81:86:67:c2:63 (500a09818667c263) No 0d Fibre Channel Target Adapter 0d (Dual -channel, QLogic Partner ONLINE 50:0a:09:80:86:17:c3:ac (500a09808617c3ac) 50:0a:09:8a:86:17:c3:ac (500a098a8617c3ac) No

This example shows the target adapters when using the cfmode dual_fabric. The FCP target port provides virtual ports which give access to LUNs hosted by each node. Dual_fabric mode provides three types of ports including Local, Partner and Standby. The Local port provides access to LUNs hosted by this node, the Partner port provides access to LUNs hosted by the partner node and the Standby port provides access to the partner node when in a takeover state. The dual_fabric mode applies only to FAS270 storage controllers. > fcp show adapters Slot: Description: rev. 2) Adapter Name: Adapter Type: Status: FC Nodename: FC Portname: Adapter Name:

0c Fibre Channel Target Adapter 0c (QLogic 2312 (2342) 0c_0 Local ONLINE 50:0a:09:80:85:14:70:80 (500a098085147080) 50:0a:09:81:85:14:70:80 (500a098185147080) 0c_1

Adapter Type: Status: FC Nodename: FC Portname: Adapter Name: Adapter Type: Status: FC Nodename: FC Portname: 4.

Standby OFFLINE 50:0a:09:80:85:e4:74:53 (500a098085e47453) 50:0a:09:81:85:e4:74:5 (500a098185e47453) 3 0c_2 Partner ONLINE 50:0a:09:80:85:e4:74:53 (500a098085e47453) 50:0a:09:89:85:e4:74:53 (500a098985e47453)

Verify the host initiator connectivity to primary FCP target ports and the host MPIO software configuration, using the suspect list generated in Step 2. Once the primary and secondary FCP target ports have been identified, confirm that the host initiators are logged in. Connectivity of host initiators to the fabric and FCP target ports can be checked from the host. While the storage controller is capable of determining which initiators have been logged in, this information is not always current. Only the host initiators are able to provide current login status information. Tools such as Emulex's HBAnywhere and QLogic's SANSurfer may be used to determine the state of the initiators. If the initiator is no longer logged into the primary FCP target port, then the fabric should be inspected for link failures or zone restrictions that would prevent connectivity to the FCP target ports. The following articles provide detailed procedures to verify host initiator connectivity and MPIO configuration for each host operating system. How to verify HP-UX fibre channel configurations with multipathing I/O (MPIO) How to verify VMWare ESX fibre channel configurations with multipathing I/O (MPIO) How to verify Windows fibre channel configurations with multipathing I/O (MPIO) How to verify Solaris fibre channel configurations with multipathing I/O (MPIO) How to verify Linux fibre channel configurations with multipathing I/O (MPIO) How to verify AIX fibre channel configurations with multipathing I/O (MPIO) How to verify NetWare fibre channel configurations with multipathing I/O (MPIO) How to verify VMware ESXi configurations with multipathing I/O

5.

Verify use of the partner path has ceased from both cluster nodes. The sysstat -b 1 command may be used to monitor SAN related summary performance counters. The "Partner" counter monitors I/O operations and data transfers that use the partner path. Monitoring these counters is an effective method to verify that the source of partner path traffic has been corrected. Be sure to check for partner path access from both cluster nodes as partner path traffic could be entering the cluster from both nodes. CPU Disk FCP iSCSI Partner Total kB/s CP CP Disk FCP in write time ty util 0% 0 0 0 8 24 0% 2% 0% 0 0 0 16 8 0% 1% 0% 0 1 0 0 0 0% 0% 0 0 1 0 0 0 kB/s out 0 0 0 iSCSI in 0 0 1 kB/s Partner out 0 0 0 in 0 0 0 kB/s out 0 0 0 read

Related links Fibre Channel and iSCSI Configuration Guide Data ONTAP Block Access Management Guide Emulex HBAnywhere QLogic SANSurfer and SANSurfer CLI