Document 1010640.

1 of 7

https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...

Welcome, Usman

Home

Knowledge

Knowledge

Certifications

Browse

Patches & Updates

Service Request

Favorites

Create SR

Keyword

Communities

Favorites

Profile

Contact Us

Help

Site Map

ID Filter

Sign Out

Advanced

Document Display

Summary of Typical Solaris IP Multipathing (IPMP) Configurations (Doc ID 1010640.1)

Modified 15-JUN-2011

Type HOWTO

Migrated ID 214668

Status PUBLISHED(EXTERNAL)

Next Steps
Priority 3

To Bottom

Applies to:
Solaris SPARC Operating System - Version: 8 10/00 U2 to 10 9/10 - Release: 8.0 to 10.0
Solaris x64/x86 Operating System - Version: 8 10/00 U2 to 10 9/10 [Release: 8.0 to 10.0]
All Platforms

Rate Article Now

Rate this document

Goal

Excellent

Provide examples for different IPMP configuration scenarios.

Solution

Good
Poor
Did this resolve the problem?
Yes

Steps to Follow
Contents:
1. Production and test interfaces in the same IP subnet
1.1 With default router
1.2 Without default router
1.3 With dedicated hosts acting as test targets with "host-routes"
1.4 Configuration example for 1.1, 1.2 and 1.3

No
Just Browsing
Was this easily located?
Very Easy
Somewhat Easy
Not Easy
Feedback

2. Production and test interfaces in different IP subnets but the same

physical network
2.1 With default router in production subnet and test subnet
2.2 With default router in production subnet but without default router in test subnet
2.3 With dedicated hosts acting as test targets with "host-routes"
2.4 Configuration example for 2.1, 2.2 and 2.3
2.5 Configuration example for 2.2 if you use IPMP on the test subnet

Submit

A. Start script for adding static "host routes" on Solaris 8 and 9

B. Adding persistent "host routes" via 'route -p' option on Solaris 10
C. Summary
Note: This document does not cover the basic knowledge of IP Multipathing. If you would need it, please search in Oracle Documentation.
All IPMP patches mentioned below should be installed.
OS
Release
Sparc
Intel
Comments
----------------------------------------------------------------------------------7 or earlier
N/S
N/S
IPMP was introduced with Solaris[TM] 8 U2
----------------------------------------------------------------------------------8
108528
108529
kernel update patch (in.mpathd, ip, icmp, ifconfig)
116965
116966
ip/arp/tcp/udp/tun patch
109900
109901
/etc/init.d/network and /sbin/ifparse patch
109902
109903
in.ndpd patch
110378
110396
mipagent patch Mobile IP
116991
116992
rcm_daemon patch
----------------------------------------------------------------------------------9
112233
112234
Kernel Patch
114344
119435
arp, dlcosmk, ip, and ipgpc Patch
"
114348
/usr/sbin/in.routed patch
112912
N/P
libinetcfg Patch
113073
114733
ufs and fsck patch
----------------------------------------------------------------------------------10
118833-36 118855-36 kernel patch
120011-14 120012-14 kernel patch (min. required for IPMP patch 141532)
141532
141533
in.mpathd patch
10 U4 (8/07) 127111
127112
kernel patch
10 U5 (5/08) 137111
137112
kernel patch
10 U6 (10/08) 138888
138889
kernel patch
10 U7 (5/09) 141414
141415
kernel patch
10 U8 (10/09) 142900
142901
kernel patch
10 U9 (9/10) 144488
144489
kernel patch
----------------------------------------------------------------------------------Key
N/S - Product not supported on this OS
N/P - No patch applicable for this OS

Legend:
IPMP
T
p
t
--===

system with IPMP group

target host
network/host partof IP address of interface in production subnet
network/host part of IP address of interface in test subnet
data link subnet (i.e broadcast domain) with one IP subnet
data link subnet (i.e broadcast domain) with two (or more) IP subnets

Good to know:
The operation of IP Multipathing (in.mpathd) depends on the routing configuration. Therefore in.mpathd always refers to the routing-table (IRE-cache)
to distinguish which test partner(s) are going to be used. Test partners are required for deciding if the interface is working properly.
in.mpathd by default chooses the default router as single test-target for probing. If no default router exists for the test-interface ip address, arbitrary
hosts on the link are detected by sending out "all hosts" multicast packets (224.0.0.1) on the wire to detect its test-partners. An "all routers"
multicasts (224.0.0.2) will never be sent! The first five hosts that are responding to the echo packets are chosen as targets for probing. In such a

6/21/2011 3:21 PM

Document 1010640.1

2 of 7

https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...

non-default router environment, the in.mpathd always tries to find five probe targets via an "all hosts" multicast.
The in.mpathd detects failures and repairs by sending out 'icmp-echo' probes (like pinging) from all interfaces that are part of the IPMP group. If there
are no replies to five consecutive probes, the interface is considered to have failed and performs a failover of the network access to another interface
in the IPMP group. The probing rate depends on the failure detection time which is defined in /etc/default/mpathd. By default, failure detection time is
10 seconds. Thus the five probes will be sent within the failure detection time.

1. Production and test interfaces in the same IP subnet

1.1 With default router

+----------------+
| default router |
+-------o--------+
| p=t:172.20.20.1
|
|
----------+-+------------------------+------------------------| |
p=t:172.20.20/24
| |
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
|
IPMP |
+---------+

IPMP uses the default router as a single probe target if it is configured to the same IP subnet. Each test interface of the IPMP group will send ICMP
requests to the default router only. To get the configuration, IPMP looks to the routing table and is independent of the /etc/defaultrouter file. There will
be NO "all hosts" multicast (224.0.0.1) sent out.
Advantages:
Easiest configuration for IPMP.
Disadvantages:
If the default router is down then IPMP failover does not work anymore. The in.mpathd does NOT send out multicasts to get other probe
targets, therefore all interfaces in the IPMP group get the state "failed". You can ignore this bug/feature when you have a default router which
is 100% online! Please look to RFE 4431511 and 4489960 for further information.
If you have a lot of IPMP groups, the default router has to reply to many ICMP requests. Take care about the default router. Do not overload
the default router.
The default router has to reliably answer ICMP echo requests. (e.g. firewalls sometimes do not)

1.2 Without default router

+-----+
+-----+
| T 1 |
...
| T 5 |
+--o--+
+--o--+
| p=t:172.20.20.110
| p=t:172.20.20.150
|
|
|
|
-----------+-+--------------+---------------------+------| |
p=t:172.20.20.0/24
| |
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
|
IPMP |
+---------+

IPMP dynamically determines five arbitrary hosts on the link via an "all hosts" multicast (224.0.0.1). At least, you will need one probe target to get
IPMP working but be aware that one probe target is not reliable enough. As long as there are less than five targets available, the in.mpathd daemon
will continuously sent out "all hosts" multicasts to get a complete list of five probe targets.
Advantages:
easiest configuration for IPMP in a subnet without a default router.
very reliable due to the five targets.
Disadvantages:
a subnet without an default router is very rare.

1.3 With dedicated hosts acting as test targets with "host-routes"

+----------------+
+-----+
+-----+
| default router |
| T 1 |
...
| T 5 |
+-------o--------=
+--o--+
+--o--+
|
| p=t:172.20.20.110
| p=t:172.20.20.150
|
|
|
|
|
|
-----------+-+--------------+---------------------+------| |
p=t:172.20.20.0/24
| |
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
|
IPMP |
+---------+

6/21/2011 3:21 PM

Document 1010640.1

3 of 7

https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...

Solaris 8 and 9 - Some "host routes" will be defined with a start script in /etc/rc2.d/S70staticroutes (refer to appendix A). When IPMP refers to the
routing table it will choose the first five defined "host routes" as probe targets. This is due to the fact that normally the "host routes" are before the
default router in the routing table. If you have less than five "host routes", the default router (when available) will be used as a probe target too.
Examples:
1. Configuration with host1, host2 ... hostN (with N=5 or N>5), defaultrouter :
==> The first five hosts (host1 ... host5) will be defined as target, not the default router
2. Configuration with less than 5 hosts : for instance, host1, host2, defaultrouter :
==> The three systems (host1, host2, defaultrouter) will be defined as target.
The in.mpathd continuously tries to get five probe targets from the routing table but it will NOT send "all hosts" multicasts!
Solaris 10 - Use the "route -p" option to add persistent routes to the kernel (refer to appendix B). No additional script will be necessary. If there is a
default router defined in /etc/defaultrouter this will always be used as a probe target plus up to four additional probe targets defined via "host routes".
Advantages:
The default router is not a single point of failure anymore because you still have some other "host routes" as probe targets.
Higher availability.
Disadvantages:
More administrative work to do.
Due to static configuration you should regularly check that the probe targets are available.

1.4 Configuration examples for 1.1, 1.2 and 1.3

/etc/hosts
127.0.0.1
172.20.20.10
172.20.20.210
172.20.20.220

localhost
host10 loghost
host10-test-qfe0
host10-test-qfe4

/etc/hostname.qfe0
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 netmask + broadcast + deprecated -failover up
/etc/hostname.qfe4
host10-test-qfe4 netmask + broadcast + deprecated -failover group ipmp0 up

ifconfig output:

qfe0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0

ether 8:0:20:e8:88:dc
qfe0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 172.20.20.210 netmask ffffff00 broadcast 172.20.20.255
qfe4: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 172.20.20.220 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:89:34

Note: The example describes the setup of an active-active IPMP configuration which is best practices. But you can also configure an active-standby
configuration. You simple have to add the "standby" flag to the /etc/hostname.qfe4 file. More Details available in the following Technical
Instruction Document 1012450.1 "Solaris 8 and up: How to Disable Outbound Load Balancing in an IPMP group".

2. Production and test interfaces in different IP subnets

If you don't have enough additional IP addresses available, you can configure the IPMP test interfaces to a different IP subnet than your production
network (e.g. 192.168., 10. ..). But you have to make sure that there are always enough test partners in that test network to respond to the ICMP
probes. In such a configuration in.mpathd will use the test IP addresses only as source address for the outgoing probe packets.
Note: The in.mpathd only checks the test network. If there is no responding probe target in that IP subnet the IPMP group will fail even though the
production subnet is still available.

2.1 With default router in production subnet and test subnet

+----------------+
| default router |
+-------o--------+
| p:172.20.20.1
| t:192.168.1.1
|
----------+-+------------------------+------------------------| |
p:172.20.20.0/24
| |
t:192.168.1.0/24
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+

6/21/2011 3:21 PM

Document 1010640.1

4 of 7

https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...

|
IPMP |
+---------+

IPMP uses the default router as a single probe target if it is configured to the same IP subnet. Each test interface of the IPMP group will send ICMP
requests to the default router only. To get the configuration, IPMP looks to the routing table and is independent of the /etc/defaultrouter file. There will
be NO "all hosts" multicast (224.0.0.1) sent out.
Advantages:
No additional production IP addresses required for the test interfaces
Disadvantages:
The interface of the default router has to reside in both the production AND test subnet.
Exceptional configuration of default router.
All other disadvantages already mentioned in section 1.1

2.2 With default router in production subnet net but without default router in test subnet

+----------------+
+-----+
+-----+
| default router |
| T 1 |
...
| T 5 |
+-------o--------=
+--o--+
+--o--+
p:172.20.20.1 |
| p:172.20.20.110
| p:172.20.20.150
|
| t:192.168.1.110
| t:192.168.1.150
|
|
|
-----------+-+--------------+---------------------+------| |
p:172.20.20.0/24
| |
t:192.168.1.0/24
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
|
IPMP |
+---------+

IPMP dynamically determines five arbitrary hosts on the link via an "all hosts" multicast (224.0.0.1). At least, you will need one probe target to get
IPMP working but be aware that one probe target is not reliable enough. As long as there are less than five targets available, the in.mpathd daemon
will continuously sent out "all hosts" multicasts to get a complete list of five probe targets.
Advantages:
easiest IPMP configuration if you don't have enough additional IP addresses available in the production subnet.
very reliable due to the five targets.
Disadvantages:
the probe targets must be available before you can setup the IPMP host.
more administrative work because you have to setup the probe targets as an additional interface in the test subnet.
Recommendation: Setup an additional logical network interface on target host.
(e.g. add a new interface to /etc/hostname.qfe0 with the 'addif' option)
If your test partners are on systems which run also IPMP you must add an logical network interface NOT flagged deprecated and NOT flagged
nofailover. An address associated with this interface will then be used as source address by responding properly to "all hosts" multicast
packets which are used for the automatic IPMP test partner detection. Be aware that Solaris 8 does NOT require the additional logical network
interface in the test subnet on the target host. So, in case of an upgrade from Solaris 8 to Solaris[TM] 9 or higher you have to change your
configuration.
Please refer to 2.5 for a detailed example.

2.3 with dedicated hosts acting as test targets with "host-routes"

+----------------+
+-----+
+-----+
| default router |
| T 1 |
...
| T 5 |
+-------o--------=
+--o--+
+--o--+
p:172.20.20.1 |
| p:172.20.20.110
| p:172.20.20.150
|
| t:192.168.1.110
| t:192.168.1.150
|
|
|
-----------+-+--------------+---------------------+------| |
p:172.20.20.0/24
| |
t:192.168.1.0/24
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
|
IPMP |
+---------+

Solaris 8 and 9 - Some "host routes" will be defined with a start script in /etc/rc2.d/S70staticroutes (refer to appendix A). When IPMP refers to the
routing table it will choose the first five defined "host routes" as probe targets. This is due to the fact that normally the "host routes" are before the
default router in the routing table. If you have less than five "host routes", the default router (when available in the test subnet) will be used as a probe
target too.
Examples:
Please refer to the examples of section 1.3.
Solaris 10 - Use the "route -p" option to add persistent routes to the kernel (refer to appendix B). No additional script will be necessary. If there is a
default router defined in /etc/defaultrouter this will always be used as a probe target plus up to four additional probe targets defined via "host routes".

Advantages:

6/21/2011 3:21 PM

Document 1010640.1

5 of 7

https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...

test interfaces don't need IP addresses of the production subnet

all others already mentioned in section 1.3
Disadvantages:
all others already mentioned in section 1.3
all others already mentioned in section 2.2

2.4 Configuration examples for 2.1, 2.2 and 2.3

/etc/hosts
127.0.0.1
172.20.20.10
192.168.1.210
192.168.1.220

localhost
host10 loghost
host10-test-qfe0
host10-test-qfe4

/etc/hostname.qfe0
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 netmask + broadcast + deprecated -failover up
/etc/hostname.qfe4
host10-test-qfe4 netmask + broadcast + deprecated -failover group ipmp0 up

ifconfig output:

qfe0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0

ether 8:0:20:e8:88:dc
qfe0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
qfe4: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
ether 8:0:20:e8:89:34

Note: The example describes the setup of an active-active IPMP configuration which is best practices. But you can also configure an active-standby
configuration. You simple have to add the "standby" flag to the /etc/hostname.qfe4 file. More details available in the following Technical Instruction
Document 1012450.1 "Solaris 8 and up: How to Disable Outbound Load Balancing in an IPMP group".

2.5 Configuration example for 2.2 if you use IPMP on the test subnet
/etc/hosts
127.0.0.1
localhost
#
# IPMP active IP address:
#
172.20.20.10
host10 loghost
#
# IPMP test interfaces
#
192.168.1.210
host10-test-ce0
192.168.1.220
host10-test-ce1
#
# additional active interface in the 'test subnet'
# to be able to respond to probe packets from other
# IPMP setups (See 2.2). Will also failover.
#
192.168.1.211
host10-prod-ce0

/etc/hostname.ce0
host10 netmask + broadcast + group ipmp0 up \
addif host10-prod-ce0 netmask + broadcast + up \
addif host10-test-ce0 netmask + broadcast + deprecated -failover up

/etc/hostname.ce1
host10-test-ce1netmask + broadcast + deprecated -failover group ipmp0 up

ifconfig output:

ce0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e2:da:d5
ce0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.1.211 netmask ffffff00 broadcast 192.168.1.255
ce0:2: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
ce1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
ether 8:0:20:e2:da:d6

6/21/2011 3:21 PM

Document 1010640.1

6 of 7

https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...

A. Start script for adding static "host routes" on Solaris 8 and 9

----------- Begin of start script /etc/init.d/ipmp.targets -------------#!/sbin/sh
# /etc/rc2.d/S70ipmp.targets /etc/init.d/ipmp.targets
# Copyright (c) 2005 by Sun Microsystems, Inc.
# All rights reserved.
#
#ident "@(#)ipmp.targets
1.0.0
#
# Edit the following IPMP test TARGETS to suit your needs.
# To install:
# 1) cp ipmp.targets /etc/init.d
# 2) perform edits on the script as required (e.g: add TARGETS)
# 3) chmod 744 /etc/init.d/ipmp.targets
# 4) chown root:sys /etc/init.d/ipmp.targets
# 5) ln /etc/init.d/ipmp.targets /etc/rc2.d/S70ipmp.targets
#
TARGETS="172.20.20.117 172.20.20.127 172.20.20.137172.20.20.147172.20.20.157"
case "$1" in
'start')
/usr/bin/echo "Adding static routes for use as IPMP targets"
for target in $TARGETS; do
/usr/sbin/route add -host $target $target
done
;;
'stop')
/usr/bin/echo "Removing static routes for use as IPMP targets"
for target in $TARGETS; do
/usr/sbin/route delete -host $target $target
done
;;
esac
----------- End of start script /etc/init.d/ipmp.targets --------------

See also:
System Administration Guide: IP Services > Configuring IPMP Groups > Configuring Target Systems

B. Adding persistent "host routes" via 'route -p' option on Solaris 10

#
#
#
#
#

route
route
route
route
route

-p
-p
-p
-p
-p

add
add
add
add
add

-host
-host
-host
-host
-host

172.20.20.117 172.20.20.117 -static

172.20.20.127 172.20.20.127 -static
172.20.20.137 172.20.20.137 -static
172.20.20.147 172.20.20.147 -static
172.20.20.157172.20.20.157 -static

Note: If there is a default router defined in /etc/defaultrouter this will always be used as a probe target plus up to four additional probe targets defined
via "host routes" (the first four).

If you don't like the default router to be used as a probe target at all, you'll have to delete the /etc/defaultrouter file and to define the default router as
the last static route, e.g.
# rm /etc/defaultrouter
#
#
#
#
#
#

route
route
route
route
route
route

-p
-p
-p
-p
-p
-p

add
add
add
add
add
add

-host 172.20.20.117 172.20.20.117 -static

-host 172.20.20.127 172.20.20.127 -static
-host 172.20.20.137 172.20.20.137 -static
-host 172.20.20.147 172.20.20.147 -static
-host 172.20.20.157172.20.20.157 -static
default 172.20.20.1 -static

Note: Due to the missing /etc/defaultrouter file, Solaris will automatically start the routing daemon 'in.routed' at boot time, so you should configure the
"quiet mode" for RIP to not supply any routing information to other systems:
# routeadm -m route:default quiet_mode=true

C. Summary
It's not easy to give a general recommendation because it depends on the network infrastructure which the customers have. Therefore you have to
discuss the various possibilities with your customer. Maybe the most used setups are 1.1 and 2.2.
To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle
Support Community, Oracle Solaris Kernel Community.

References
NOTE:1001790.1 - The differences between Network Adapter Failover in Sun[TM] Cluster 3.0 and IP Multipathing in Sun[TM] Cluster 3.1 or
Solaris[TM] Cluster 3.2
NOTE:1008064.1 - Solaris 10 IP Multipathing (IPMP) Link-based Only Failure Detection
NOTE:1012450.1 - Solaris[TM] 8 and up: How to disable outbound load balancing in an IPMP group

6/21/2011 3:21 PM

Document 1010640.1

7 of 7

https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx...

NOTE:1017689.1 - How to Use Solaris IP Multipathing (IPMP) for Network Path Failover

Related
Products

Sun Microsystems > Operating Systems > Solaris Operating System > Solaris SPARC Operating System > Network Interface > ipmp, ipmpstat,
mpathd, IP Multipathing
Sun Microsystems > Operating Systems > Solaris Operating System > Solaris x64/x86 Operating System > Network Interface > ipmp, ipmpstat,
mpathd, IP Multipathing
Knowledge Categories

Sun Microsystems > Operating Systems > Solaris Network

Keywords

ACTIVE STANDBY
CONFIGURATION
DEPRECATED
FAILOVER
IPMP
Return To Top
Favorites Profile Contact Us Help Site Map Sign Out
Copyright 2007, 2011, Oracle and/or its affiliates. All rights reserved

Legal Notices and Terms of Use | Privacy Statement

6/21/2011 3:21 PM