Configuring IP Network Multipathing on 

Solaris

May 8, 2009 in Solaris

Tags: configure IPMP on Solaris, IP Network Multipathing, IPMP, ipmp 2 interface 1
virtual ip, ipmp 4 interface 2 virtual ip, IPMP failover, ipmp with 2 interface, ipmp
with 4 interface, link base ipmp, probe base ipmp, solaris ipmp, solaris link base
ipmp, solaris probe base ipmp

IPMP has the following features:

1. It eliminates a single network adapter as a single point of failure in these cases
, ( a. Network adapter failure b. Network link failure)
2. It enables interfaces to fail over within approximately 10 seconds when using t
he default configuration.
3. It can be configured by adjusting the parameters in the ”/etc/default/mpathd” fi
le.
4. It can be configured for use with both IPv4 and IPv6.
5. It enables interfaces to be configured as standby interfaces.

{IPMP failure  detection:
Probe-based failure detection
-detects network error by sending ICMP ECHO_REQUEST messages
Link based failure detection
-detects network error by checking the IFF_RUNING

How to Configure IPMP on Solaris

*PROBE-BASED IPMP*

{Probe-Based IPMP Requirements:

 The Solaris 8 10/00 OS, as a minimum, must be installed.
 Unique MAC addresses must be configured on each network interface.
 Multiple network adapter interfaces must be connected on each subnet.
 An IPMP group name must be assigned to the group of interfaces.
 A test address is assigned to an interface.
 Additional hosts or devices must exist on the same subnet.

{Configuring Probe-Based IPMP:

Sample:

IPMP FAILOVER WITH 2 FLOATING/PUBLIC IP ADDRESS:

{2 NIC address:
e1000g0 ip : 192.168.1.1
e1000g1 ip : 192.168.1.2

1
{2 floating/public address:
192.168.1.3
192.168.1.4

{edit /etc/hosts with your 4 IP address:

192.168.1.3    server-1    server  loghost

192.168.1.4    server-2 loghost
192.168.1.1    server-e1000g0  loghost
192.168.1.2    server-e1000g1  loghost

Here are the step:

1. # cat /etc/release
2. # eeprom ”local-mac-address?=true”
3. # ifconfig e1000g0 192.168.1.1 netmask + broadcast + group ipmpserver
deprecated -failover up
4. # ifconfig e1000g0 addif 192.168.1.3 netmask + broadcast + failover up
5. # ifconfig e1000g1 192.168.1.2 netmask + broadcast + group ipmpserver
deprecated -failover standby up
6. # ifconfig e1000g1 addif 192.168.1.4 netmask + broadcast + failover up
7. # ifconfig -a

{Using  configuration  files:

1. # cat /etc/hostname.e1000g0
server-e1000g0 netmask + broadcast + group ipmpserver deprecated -failover up \
addif server-1 netmask + broadcast + failover up

2. # cat /etc/hostname.e1000g1
server-e1000g1 netmask + broadcast + group ipmpserver deprecated -failover
standby up \
addif server-2 netmask + broadcast + failover up

3. # init 6

IPMP FAILOVER WITH 1 FLOATING/PUBLIC IP ADDRESS:

{2 NIC address:
e1000g0 ip : 192.168.1.1
e1000g1 ip : 192.168.1.2

{1 floating/public address:
192.168.1.3

{edit /etc/hosts with your 3 IP address:

192.168.1.3    server   loghost

192.168.1.1    server-e1000g0
192.168.1.2    server-e1000g1

2
{Using  configuration  files:
1. # cat /etc/hostname.e1000g0
server-e1000g0 netmask + broadcast + group ipmpserver deprecated -failover up \
addif server netmask + broadcast + failover up

2. # cat /etc/hostname.e1000g1
server-e1000g1 netmask + broadcast + group ipmpserver deprecated -failover
standby up

3. # init 6

*LINK-BASED IPMP*

{Link-Based IPMP Requirements:

 Solaris 9 12/02 OS, at a minimum, must be installed.
 Network interfaces must use any of the following drivers: hme, eri, ce, ge, bge
, qfe, dmfe, e1000g, ixgb, nge, nxge, rge, xge
 Unique MAC addresses must be configured on each of the interfaces.
 An IPMP group name must be assigned to interfaces

{Configuring Link-Based IPMP

1. # cat /etc/release
2. # eeprom ”localmacaddress=true”
3. # ifconfig e1000g0 192.168.1.1 netmask + broadcast + group <groupname> up
4. # ifconfig e1000g1 192.168.1.2 netmask + broadcast + group <groupname>
standby up
5. # ifconfig a

{Configuring Link-IPMP – Administration Guide – IP

ServicesBased IPMP using configuration files
1. # cat /etc/hostname.e1000g0
server-e1000g0 netmask + broadcast + group <groupname> up

2. # cat /etc/hostname.e1000g1
server-e1000g1 netmask + broadcast + group <groupname> standby up

3. # init 6

{Edit file “/etc/default/mpathd” to adjust failover detection timeout.

find the line “FAILURE_DETECTION_TIME=10000″.

the default config is 10000, it mean the failover will take around 10 seconds.

To test the interface failover and failback, use these commands:

#if_mpadm -d e1000g0     >> Dettach interface and force failover

#if_mpadm -r e1000g0      >> Reattach interface and failback

3
NOTE:

deprecated =the interface will not used for data traffic

-failover = the test interface (physical NIC) will not failover
+failover = the floating/public ip address will failover.

Reference:

IPMP – Administration Guide – IP Services

man ifconfig
man in.mpathd
docs.sun.com

——————————————————
COMMON ERROR Messages and Diagnostics
——————————————————
Test address address is not unique; disabling probe based failure detection on
interface_name

Description:
For in.mpathd to perform probe-based failure detection, each test address in the group
must be unique. Since the IPv6 test address is a link-local address derived from the
MAC address, each IP interface in the group must have a unique MAC address.

——-
NIC interface_name of group group_name is not plumbed for IPv[4|6] and may affect
failover capability

Description:
All NICs in a multipathing group must be homogeneously plumbed. For example, if a
NIC is plumbed for IPv4, then all NICs in the group must be plumbed for IPv4. The
streams modules pushed on all NICs must be identical.

——-
No test address configured on interface interface_name disabling probe-based failure
detection on it

Description:
In order for in.mpathd to perform probe-based failure detection on a NIC, it must be
configured with a test address: IPv4, IPv6, or both.

——-
The link has come up on interface_name more than 2 times in the last minute;
disabling failback until it stabilizes.

Description:
In order to prevent interfaces with intermittent hardware, such as a bad cable, from

4
causing repeated failovers and failbacks, in.mpathd does not failback to interfaces
with frequently fluctuating link states.
——-

Invalid failure detection time assuming default 10000

Description:
An invalid value was encountered for FAILURE_DETECTION_TIME in the
/etc/default/mpathd file.

——-
Too small failure detection time of time assuming minimum 100

Description:
The minimum value that can be specified for FAILURE_DETECTION_TIME is
currently 100 milliseconds.
——-
Invalid value for FAILBACK value

Description:
Valid values for the boolean variable FAILBACK are yes or no.

——-
Invalid value for TRACK_INTERFACES_ONLY_WITH_GROUPS value

Description:
Valid values for the boolean variable
TRACK_INTERFACES_ONLY_WITH_GROUPS are yes or no.
——-

Cannot meet requested failure detection time of time ms on (inet[6] interface_name)

new failure detection time for group group_name is time ms

Description:
The round trip time for ICMP probes is higher than necessary to maintain the current
failure detection time. The network is probably congested or the probe targets are
loaded. in.mpathd automatically increases the failure detection time to whatever it can
achieve under these conditions.
——-

Improved failure detection time time ms on (inet[6] interface_name) for group

group_name

Description:
The round trip time for ICMP probes has now decreased and in.mpathd has lowered
the failure detection time correspondingly.
——-

NIC failure detected on interface_name

5
Description:
in.mpathd has detected NIC failure on interface_name, and has set the IFF_FAILED
flag on NIC interface_name.
——-

Successfully failed over from NIC interface_name1 to NIC interface_name2

Description:
in.mpathd has caused the network traffic to failover from NIC interface_name1 to
NIC interface_name2, which is part of the multipathing group.
——-
NIC repair detected on interface_name

Description:

in.mpathd has detected that NIC interface_name is repaired and operational. If the
IFF_FAILED flag on the NIC was previously set, it will be reset.

——-
Successfully failed back to NIC interface_name

Description:
in.mpathd has restored network traffic back to NIC interface_name, which is now
repaired and operational.

——-
The link has gone down on interface_name

Description:
in.mpathd has detected that the IFF_RUNNING flag for NIC interface_name has been
cleared, indicating the link has gone down.

——-
The link has come up on interface_name

Description:
in.mpathd has detected that the IFF_RUNNING flag for NIC interface_name has been
set, indicating the link has come up.

========================

UPDATE 4 FEBRUARY 2011 >> Other example, its more simple and just applied on
customer site:

=========================

CONFIGURE IPMP: 2 INTERFACE WITH 1 VIRTUAL IP ADDRESS:

——————————————————

6
setup >> local-mac-address?=true
——–
HOSTNAME: SOLARIS10
IP ADDRESS BGE0: 192.168.0.1
IP ADDRESS BGE1: 192.168.0.2

VIRTUAL IP ADDRESS: 192.168.0.3

GROUPNAME: IPMP
——–
[/etc/hostname.bge0:
SOLARIS10 netmask + broadcast + group ipmp \
deprecated -failover up addif 192.168.0.3 netmask + broadcast + failover up

[/etc/hostname.bge1:
192.168.0.2 netmask + broadcast + group ipmp deprecated -failover standby up

[/etc/host:
::1    localhost
127.0.0.1    localhost
192.168.0.1    SOLARIS10    loghost

REBOOT SERVER

RESULT:
[ifconfig -a:
bge0:
flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,
NOFAILOVER> mtu 1500 index 2
inet 192.168.0.1 netmask ffffff00 broadcast 192.168.0.255
groupname ipmp
ether 0:21:22:74:1f:5a
bge0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu
1500 index 2
inet 192.168.0.3 netmask ffffff00 broadcast 192.168.0.255
bge1:
flags=39040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4
,NOFAILOVER,FAILED,STANDBY> mtu 1500 index 3
inet 192.168.0.2 netmask ffffff00 broadcast 192.168.0.255
groupname ipmp
ether 0:21:22:74:1f:5b
sppp0:
flags=10010008d1<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST,IPv4,FI
XEDMTU> mtu 1500 index 4
inet 192.168.224.2 –> 192.168.224.1 netmask ffffff00
ether 0:0:0:0:0:0

##############################################
CONFIGURE IPMP: 4 INTERFACE WITH 2 VIRTUAL IP ADDRESS:
——————————————————

7
setup >> local-mac-address?=true
——–
HOSTNAME: SOLARIS11
IP ADDRESS NXGE0: 192.168.0.1
IP ADDRESS NXGE1: 192.168.0.2
IP ADDRESS NXGE2: 192.168.0.3
IP ADDRESS NXGE3: 192.168.0.4

VIRTUAL IP ADDRESS: 192.168.0.5

VIRTUAL IP ADDRESS: 192.168.0.6

GROUPNAME: IPMP AND IPMP1

———-

[/etc/hostname.nxge0:
SOLARIS11 netmask + broadcast + group ipmp \
deprecated -failover up addif 192.168.0.5 netmask + broadcast + failover up

[/etc/hostname.nxge1:
192.168.0.2 netmask + broadcast + group ipmp deprecated -failover standby up

[/etc/hostname.nxge2:
192.168.0.3 netmask + broadcast + group ipmp1 \
deprecated -failover up addif 192.168.0.6 netmask + broadcast + failover up

[/etc/hostname.nxge3:
192.168.0.4 netmask + broadcast + group ipmp1 deprecated -failover standby up

[etc/host:
::1    localhost
127.0.0.1    localhost
192.168.0.1    SOLARIS11    loghost

———-
REBOOT SERVER

RESULT

[ifconfig -a:
lo0:
flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL>
mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
nxge0:
flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,
NOFAILOVER> mtu 1500 index 2
inet 192.168.0.1 netmask ffffffe0 broadcast 192.168.0.255
groupname ipmp
ether 0:22:28:74:1c:ba
nxge0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu

8
1500 index 2
inet 192.168.0.5 netmask ffffffe0 broadcast 192.168.0.255
nxge1:
flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4
,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index

3
inet 192.168.0.2 netmask ffffffe0 broadcast 192.168.0.255
groupname ipmp
ether 0:22:28:74:1c:bb
nxge3:
flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,
NOFAILOVER> mtu 1500 index 4
inet 192.168.0.3 netmask ffffff00 broadcast 192.168.0.255
groupname ipmp1
ether 0:22:28:74:bb:5a
nxge3:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu
1500 index 4
inet 192.168.0.6 netmask ffffff00 broadcast 192.168.0.255
nxge4:
flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4
,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index

5
inet 192.168.0.4 netmask ffffff00 broadcast 192.168.0.255
groupname ipmp1
ether 0:22:28:74:bb:5b
sppp0:
flags=10010008d1<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST,IPv4,FI
XEDMTU> mtu 1500 index 6
inet 192.168.224.2 –> 192.168.224.1 netmask ffffff00
ether 0:0:0:0:0:0