Solaris 10 IPMP

This is version 1.0 of IPMP

1 Table of Contents

2 Environment

server # grep hme /etc/path_to_inst "/pci@1f,0/pci@1,1/network@1,1" 0 "hme" "/pci@1f,0/pci@1/pci@2/SUNW,hme@0,1" 1 "hme" server # ifconfig -a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 inet 10.156.1.160 netmask ffffff00 broadcast 10.156.1.255 ether 8:0:20:ff:d1:66

3 Configure IPMP

3.1 Setup hosts

Create a /etc/inet/hosts file that contains 4 address for the IPMP environment.

# cat /etc/inet/hosts # Internet host table # ::1 localhost 127.0.0.1 localhost 10.156.1.160 ext1 ses5 loghost 10.156.1.161 int1 10.156.1.170 ext2 10.156.1.171 int2

In this example we use a test address for each interface under IPMP control. The int1 and int2 addresses are these test addresses. The ext1 and ext2 addresses are the actual address that the server will use to communicate over the network.

N.B. To solve possible mismatches between /etc/inet/hosts and /etc/inet/ipnodes I just linked /etc/inet/ipnodes to /etc/inet/hosts. This is the reason for the ::1 localhost (== IPv6) line.

3.2 Configure interfaces

Interface : hme0 

server # vi /etc/hostname.hme0 int1 netmask + broadcast + group production \ deprecated -failover up addif ext1 \ netmask + broadcast + failover up server # ifconfig hme0 int1 netmask + broadcast + group production deprecated -failover up addif ext1 netmask + broadcast + failover up Setting netmask of hme0 to 255.255.255.0 Setting netmask of hme0 to 255.255.255.0 Setting netmask of hme0 to 255.255.255.0 Created new logical interface hme0:1 Setting netmask of hme0:1 to 255.255.255.0

Interface : hme1 

server # vi /etc/hostname.hme1 int2 netmask + broadcast + group production \ deprecated -failover up addif ext2 \ netmask + broadcast + failover up server # ifconfig hme1 plumb server # ifconfig hme1 int2 netmask + broadcast + group production deprecated -failover up addif ext2 netmask + broadcast + failover up Setting netmask of hme1 to 255.255.255.0 Created new logical interface hme1:1 Setting netmask of hme1:1 to 255.255.255.0

3.3 Check configuration

server # ifconfig -a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4 inet 10.156.1.161 netmask ffffff00 broadcast 10.156.1.255 groupname production ether 8:0:20:ff:d1:66 hme0:1: flags=1000843 mtu 1500 index 4 inet 10.156.1.160 netmask ffffff00 broadcast 10.156.1.255 hme1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5 inet 10.156.1.171 netmask ffffff00 broadcast 10.156.1.255 groupname production ether 8:0:20:ec:f3:92 hme1:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5 inet 10.156.1.170 netmask ffffff00 broadcast 10.156.1.255 server # cat /etc/default/mpathd # #pragma ident "@(#)mpathd.dfl 1.2 00/07/17 SMI" # # Time taken by mpathd to detect a NIC failure in ms. The minimum time # that can be specified is 100 ms. # FAILURE_DETECTION_TIME=2500 # # Failback is enabled by default. To disable failback turn off this option # FAILBACK=yes # # By default only interfaces configured as part of multipathing groups # are tracked. Turn off this option to track all network interfaces # on the system # TRACK_INTERFACES_ONLY_WITH_GROUPS=yes server # ps -ef | grep mpath root 3135 1 0 12:55:50 ? 0:00 /usr/lib/inet/in.mpathd

4 Test configuration

4.1 Fail a network link

Now we can test the configuration by unplugging one of the network cables. In the /var/adm/messages file you will see the following message after in.mpathd detects that one link in down.

Sep 6 13:03:02 ses5 in.mpathd[3135]: [ID 215189 daemon.error] The link has gone down on hme0 Sep 6 13:03:02 ses5 in.mpathd[3135]: [ID 594170 daemon.error] NIC failure detected on hme0 of group production Sep 6 13:03:02 ses5 in.mpathd[3135]: [ID 832587 daemon.error] Successfully failed over from NIC hme0 to NIC hme1

and the interfaces are configured like this

server # ifconfig -a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=19040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 4 inet 10.156.1.161 netmask ffffff00 broadcast 10.156.1.255 groupname production ether 8:0:20:ff:d1:66 hme1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5 inet 10.156.1.171 netmask ffffff00 broadcast 10.156.1.255 groupname production ether 8:0:20:ec:f3:92 hme1:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5 inet 10.156.1.170 netmask ffffff00 broadcast 10.156.1.255 hme1:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5 inet 10.156.1.160 netmask ffffff00 broadcast 10.156.1.255

Especially pay attention to the FAILED keyword on the hme0 interface.

4.2 Repair network link

After the failed network link is repaired you wil see the following in /var/adm/messages

Sep 6 13:09:26 ses5 genunix: [ID 408822 kern.info] NOTICE: hme0: no fault in device; service available Sep 6 13:09:26 ses5 genunix: [ID 611667 kern.info] NOTICE: hme0: Internal Transceiver Selected. Sep 6 13:09:26 ses5 genunix: [ID 408822 kern.info] NOTICE: hme0: no fault external to device; service available Sep 6 13:09:26 ses5 genunix: [ID 611667 kern.info] NOTICE: hme0: 100 Mbps half duplex link up Sep 6 13:09:26 ses5 in.mpathd[3135]: [ID 820239 daemon.error] The link has come up on hme0 Sep 6 13:09:29 ses5 in.mpathd[3135]: [ID 299542 daemon.error] NIC repair detected on hme0 of group production Sep 6 13:09:29 ses5 in.mpathd[3135]: [ID 620804 daemon.error] Successfully failed back to NIC hme0

And your interface configuration looks like this

server # ifconfig -a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4 inet 10.156.1.161 netmask ffffff00 broadcast 10.156.1.255 groupname production ether 8:0:20:ff:d1:66 hme0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 inet 10.156.1.160 netmask ffffff00 broadcast 10.156.1.255 hme1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5 inet 10.156.1.171 netmask ffffff00 broadcast 10.156.1.255 groupname production ether 8:0:20:ec:f3:92 hme1:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5 inet 10.156.1.170 netmask ffffff00 broadcast 10.156.1.255

5 Disclaimer

I'm in no way responsible for what you do. Not even when you do it after reading my documents.