Sun Cluster 3.1

How to install SC 3.1 10/03 (Update 1) on Solaris 8 2/04 or Solaris 9 12/03

with the EIS process. Send all comments to Raf LaPietra.

Version: 31u1_037

Last Modified Fri Apr 23 08:13:00 EDT 2004

If you need a version of this document for older versions of software, please

contact me and I'll see what I have archived.

Please do NOT open any bugs or RFEs against EIS for this document, as it is

individually maintained. Send them directly to me. Please include the version

of this document that you are referring to.

This doc can be found on the EIS cd2 under sun2/docs/MISC/

The latest revision of this document can be found at:

http://webhome.east/raf/cluster/index.html

Important note - Patches marked with a ## are not found on the 30MAR04 EIS

cd (either a different rev or missing) and will need to

be downloaded separately and noted in the install report

I. Planning Meetings -

A. Review the cluster installation deliverables with the customer. Get from

http://www.sun.com/service/support/install/suncluster-software.html

B. Fill out the EIS checklists from <eiscd2>/sun2/docs/EISchecklists/pdf

1. Fill out the appropriate server checklist for each node and the admin

work station. This can be used to complete the EISdoc Tool documentation

C. Get customer to order the license keys from the licensing center for

VxVM and VxFS if either will be used. It is UNSUPPORTED by engineering to

run a cluster on temporary licenses

II. Install admin station

A. Install per the appropriate EIS checklist

B. Add cluster console software and (optional) documentation

1. pkgadd -d . SUNWccon

2. if desired, pkgadd the documentation:

a. man pages : pkgadd -d . SUNWscman

b. Note, for the following packages, SUNWsdocs is required by all, the

SUNWpsc* files are in pdf format, and the SUNWsc* are in html format

c. The documentation from the Sun Cluster 3.1 U1 cdrom

<cdrom mnt pt>/SunCluster_3.1/Sol_<OS Num>/Packages

SUNWsdocs SUNWpscfab SUNWpschw SUNWpscref SUNWscfab SUNWschw SUNWscref

d. From the data service cdrom in

<cdrom mnt pt>/components/SunCluster_Docs_3.1/Sol_<OS Num>/Packages

SUNWpscdab SUNWscdab

C. Create a file /etc/clusters with one entry:

D. Create a file /etc/serialports with one entry for each node

1. The entry is - <host> <tc> <500x> , x=tc port number

2. If a SunFire - <host> <sc> <500x> , x=1-4 for domains A-D

3. If an E10K, entry is - <host> <ssp> 23

4. If a SF15K or SF12K, entry is - <host> <sc> 23

E. Add all entries (nodes+logical hosts) to /etc/hosts

III. (If used) Configuration of the Sun Cluster Terminal Concentrator (X1312A)

A. Add an entry to /etc/remote: (for serial port a)

tc:dv=/dev/term/a:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:

B. Connect port 1 to admin station serial port a and tip tc

1. Cable notes:

a. For DB25 pin serial ports, use cable 530-2151(short) or 530-2152(long)

b. For Netra style RJ-45 serial ports, you can use cable 180-2001

1. NOTE - The "P1" end goes to the tc, the "P2" end to the host

C. Get into monitor mode by holding down test button while you power on

the TC. When the green light blinks, release the test button and then

press it again.

1. set IP address - monitor:: addr

2. check boot image is oper.52.enet - monitor:: image

3. set to boot from self - monitor:: seq

4. Power cycle the tc

5. Quit your tip session (~.)

6. telnet into the tc and select cli

7. become su and go into admin mode

annex: su

passwd: <tc ip address>

annex# admin

8. configure serial ports :

admin: set port=1-8 type dial_in imask_7bits Y

admin: set port=2-8 mode slave ps_history_buffer 32767

admin: quit

9. (optional) - configure rotaries

a. annex# edit config.annex

b. Add the following lines

%rotary

...

<node#>>:<tc port>@<tc ip addr>

c. ctrl-w

10. (optional) - configure default router

a. #edit config.annex

b. Add the following lines

%gateway

net default gateway <router-ip> metric 1 hardwired

c. ctrl-w

d. annex# admin set annex routed n

11. Reboot the tc

annex# boot

bootfile: <return>

warning: <return>

12. exit your telnet session (ctrl + ] and quit)

IV. Install cluster nodes

A. If using a tc:

1. On admin station - ccp <clustname>.

2. Choose cconsole

B. from the ok prompt: setenv local-mac-address? true

C. If Multi Initiated SCSI, follow infodoc 20704

D. Install Solaris 8 2/04 or Solaris 9 12/03

1. You may want to select xterms or other and enter dtterm for your term

type if using a tc/cconsole

2. Software package selection

a. Entire Distribution + OEM is RECOMMENDED

b. At least the END USER software is REQUIRED

1. Entire Distribution + OEM is required for E10K

2. If you use END USER, you may need to also add -

a. Apache software packages (SUNWapchr and SUNWapchu) if you want to

use SunPlex Manager

b. RSMAPI software packages (SUNWrsm, SUNWrsmx, SUNWrsmo,SUNWrsmox

if you want PCI-SCI adapters for the interconnect transport or

use the Remote Shared Memory Application Programming Interface

3. Root Disk partitioning

a. swap must be >= 750mb. Calculate the normal amount of swap needed by

Solaris, add in any required by third party applications, and then add

512mb for SunCluster.

b. Make and mount a 512mb filesystem called /globaldevices

c. For SDS/SVM - assign 256MB to slice 7 for the metadbs

d. For VxVM if you plan on encapsulating the root disk

1. leave slice 3 and 4 unassigned

2. leave 2 cylinders unassigned

E. Add the logging option to the / filesystem (and any others that you want)

in the /etc/vfstab file

F. If you are using GBE as your private interconnect, you may have to create

a /kernel/drv/ge.conf file. Please read InfoDoc 23041 for instructions.

G. Add entries in /etc/hosts for each logical host, phys host, admin station,

tc and ipmp test addresses

1. If you will be using an active-active IPMP configuration you must add

the nodename as the first name on the line in /etc/hosts

e.g. for a two interface active-active config, the etc hots must have:

xxx.xxx.xxx.xxx <nodename> <if1-hostname>

xxx.xxx.xxx.xxx <nodename> <if2-hostname> (this ip is optional)

xxx.xxx.xxx.xxx <if1-test-addr-hostname>

xxx.xxx.xxx.xxx <if2-test-addr-hostname>

H. Populate your /etc/defaultrouter if needed

I. Configure ipmp and network parameters on public network adapters.

1. IPMP: Edit the /etc/hostname.<if#> files for each adapter

a. For a single adapter group, the files should look like

<nodename> -failover group <ipmp0>

b. For a multi adapter group,active-standby:

1. active:

<nodename> group ipmp0 up \

addif <if#-test-addr-hostname> -failover deprecated up

2. standby:

<if#-test-addr-hostname> group ipmp0 -failover deprecated standby up

c. For a multi adapter group,active-active, for each adaptor:

1. <if#-hostname> group <ipmp0> up \

addif <if#-test-addr-hostname> -failover deprecated up

2. the second interface can use the same syntax above if it has its

own public ip address

3. If you have only 1 public ip address, you should use:

<if#-test-addr-hostname> group ipmp1 -failover deprecated up

2. ip_strict_dst_multihoming :

a. cp <eiscd1>/sun/tools/network/S68net-tune /etc/rc2.d

b. edit /etc/rc2.d/S68net-tune and add a line in the start section

ndd -set /dev/ip ip_strict_dst_multihoming 0

c. chmod 744 /etc/rc2.d/S68net-tune

d. chown root:sys /etc/rc2.d/S68net-tune

e. ln /etc/rc2.d/S68net-tune /etc/init.d/net-tune

J. A domainname must be set up to work around bug 4913925

1. run: domainname

2. if nothing is returned, you must set up a dummy domainname. This can

be removed after scvxinstall is completed, or after scinstall is done

if not using VxVM

a. domainname dummy

b. domainname > /etc/defaultdomain

K. If you want to use rsh for the cluster framework installation (Sol 8 or 9)

1. create a /.rhosts file on all the nodes that contains a list of all the

nodes in the cluster.

2. Note: if you are using active-active ipmp, you will want to add the

node name and the host names of all the active adapters.

L. If you want to use ssh for the cluster framework installation (Sol 9 only)

1. On all nodes run : ssh-keygen

a. accept default for file to save the key ((//.ssh/id_rsa)

b. You can just hit enter for the passphrase

2. add the contents of /.ssh/id_rsa.pub on all of the nodes to

/.ssh/authorized_keys on all of the nodes

a. make sure that each key for each node is on a single line!

b. an example /.ssh/authorized_keys would look like:

ssh-rsa <long key>= root@<node1>

ssh-rsa <long key>= root@<node2>

...

ssh-rsa <long key>= root@<node#>

3. edit /etc/ssh/sshd_config on all nodes

a. change the line which reads "PermitRootLogin no" to read

"PermitRootLogin without-password"

4. stop and restart sshd

/etc/init.d/sshd stop

/etc/init.d/sshd start

5. test that ssh is working on all the nodes by doing a ssh <nodename>

to and from every node in the cluster

a. The first time you ssh , you will get a warning similar to:

"The authenticity of host 'node1' can't be established."

"RSA key fingerprint in md5 is: <something>"

"Are you sure you want to continue connecting(yes/no)?"

(answer yes here)

"Warning: Permanently added '<nodename>,<ip>' (RSA) to the list of

known hosts."

6. logout and ssh in to and from each node again to make sure it goes clean

M. Setup the OS according to the proper EIS checklist.

1. In the "Run setup-standard:" section:

Answer yes to the "Are remote root logins to be allowed?"

Answer yes to the "Do you wish to add the SFS packages?"

2. Modify the root .profile

a. at the end of .profile uncomment the "cluster node" display section

b. To set your term type when using cconsole, add the following lines:

1. if [ "`tty`" = "/dev/console" ]; then

TERM=xterms;export TERM

2. a TERM of "vt100" or "dtterm" may also work for you

3. Activate the EIS environment

a. log out of the console

b. go back to the ccp and choose ctelnet. This will allow faster access

for the rest of the install process. Use the cconsole windows to see

any error messages logged to the console

4. For the "Install patches" sections of the server checklist:

a. Install EIS patches

1. cd <eiscd1>/sun/patch/<OS Number>

2. unpack-patches

3. cd /tmp/<OS Number>

4. ./install_all_patches

b. Install SAN patches

1. cd <eiscd1>/sun/patch/SAN

2. unpack-patches

3. cd /tmp/SAN

4. ./install_patches

5. cd <eiscd1>/sun/patch/SAN/<OS Number>

6. unpack-patches

7. cd /tmp/<OS Number>

8. ./install_patches

c. Install the following additional patches (per Infodoc 54960 or 54961)

if applicable to your cluster

1. In <eiscd1>/sun/patch/firmware

109115-16 T3 Firmware

109400-03 Hardware/Fcode: FC100/S HBA

111853-02 Hardware/Fcode: PCI Single FC HBA

113723-07 SE3510 Firmware

112276-11## T3+ 2.01.06 Firmware

2. If using the qus driver, add patches from

<eiscd1>/sun/patch/SUNWqus/<OS Number>

5. Probably a good time to reboot all the nodes after all those patches...

N. If using A3500 or A3500FC, install and configure RM6.22.1:

1. pkgadd rm6.22.1 from <eiscd3>/sun-internal/progs/RM/RM-6.22.1.tar.gz

2. add patches from <eiscd2>/sun2/patch/RM/6.22.1/

3. edit /etc/osa/rmparams.

a. System_MaxLunsPerController=<# of LUNs needed>

Rdac_NoAltOffline=TRUE

Rdac_RetryCount=1

a. if you have more than one A3500FC on a loop

1. Change Rdac_HotAddDisabled=PARTIAL to Rdac_HotAddDisabled=FALSE

2. Add the other A3500FC targets to Rdac_HotAddIDs:4:5:x:y:z

4. /usr/lib/osa/bin/genscsiconf

5. edit /etc/raid/rdac_address. Distribute your LUNs over the controllers

6. init 6

6. upgrade firmware and create luns (thru the rm6 GUI)

7. disable paritychk on 1 node of each pair attached to the storage (use

the rm6 GUI)

8. /usr/lib/osa/bin/rdac_disks

V. Install cluster framework on the nodes. The software packages are in

<eiscd3>/sun-internal/progs/SunCluster/3.1_U1

A. If using PCI-SCI cards, install drivers before cards

1. pkgadd -d . SUNWscrif SUNWscrdt SUNWsci SUNWscid SUNWscidx

2. Add patches from <eiscd1>/sun/patch/SunCluster/3.1/<OS Number>

Sol 8 Sol 9

========= =========

115058-03 115059-04 Reliable Datagram Transport patch

116154-01 116155-01 SCI 3.1u1: scid fixes

3. Install your SCI cards

B. Choose a node to install the cluster from.

1. Suggestion: use the node that has the highest used number in the

/etc/name_to_major file to avoid any possible conflict when the did

number is chosen.

a. A quick way to find this is to run:

cat /etc/name_to_major | /usr/bin/sort +1 -2n

2. If all the nodes have the same high number in name_to_major, use the

node you want to have the highest node number, since it will be the

last node to be installed.

C. Install the cluster framework.

1. On the install node, make a patch directory and fill it with patches:

a. mkdir -p /var/cluster/patches

b. Put uncompressed copies of these patches in /var/cluster/patches

Patches are in <eiscd1>/sun/patch/SunCluster/3.1/<OS Number>

Sol 8 Sol 9

========= =========

113800-11## 113801-11## Sun Cluster 3.1: Core/Sys Admin Patch

111488-08 N/A Sun Cluster 3.0/3.1: mediator patch

115054-01 115055-01 Sun Cluster 3.1: OpenSSL security patch

2. run scinstall

select "Install a cluster or cluster node"

select "Install all nodes of a new cluster"

select "Typical" for the type of install

a. If you need to change the ip address range for the interconnect, you

will need to choose "Custom"

3. This will install the cluster framework on all of the nodes sequentially

without further user intervention

4. When all of the nodes have installed, run scstat -n to make sure all

the nodes have joined the cluster

5. Configure ntp.conf on all of the nodes if there is not an external ntp

server to be used by the cluster

a. cp /etc/inet/ntp.conf.cluster /etc/inet/ntp.conf

b. Edit /etc/inet/ntp.conf and comment out the lines which read

peer clusternode<#>-priv for each node # which doesn't exist

For example, if you have a 2 node cluster, comment out the lines from

peer clusternode3-priv to peer clusternode16-priv

E. Set up the quorum disk (required for 2 node, recommended for 3+ nodes)

1. Run devfsadm -C on all nodes to be sure all the global devices are OK

2. Run scdpm -p all:all to be sure that all the shared disks paths are

OK to all of the attached nodes

3. scdidadm -L : to id the quorum disks.

a. Pick a disk seen by all nodes if possible. otherwise you may want to

set up more than one quorum disk

4. On ONE node, run scsetup to assign the quorum disk

a. Answer "Yes" to: Is it okay to reset "installmode"

F. If you did NOT set up a quorum device, reset install mode

1. On ONE node, run scsetup.

a. Answer "no" to "Do you want to add any quorum disks?"

b. Answer "Yes" to: Is it okay to reset "installmode"

G. Check that your ipmp groups are configured properly on all nodes

1. scstat -i

H. Edit /etc/nsswitch.conf.

1. For any cluster:

hosts: cluster files ...

group: files ...

netmasks: cluster files ...

services: files ...

2. Customize for your data services

a. for HA-NFS:

hosts: cluster files [SUCCESS=return] nis

rpc: files nis

b. for HA-Oracle

passwd: files ...

group: files ...

publickey: files ...

project: files ...

c. for HA-SAP, HA-Siebel, and HA-Sybase:

group: files [NOTFOUND=return] nis

d. for HA-LiveCache:

protocols: files nis

passwd: files nis [TRYAGAIN=0]

group: files [NOTFOUND=return] nis

project: files [NOTFOUND=return] nis

publickey: files [NOTFOUND=return] nis

e. HA-Samba:

passwd: files winbind ...

group: files winbind ...

I. Add the Diagnostic ToolKit package SUNWscdtk from

<eiscd3>/sun-internal/progs/SunCluster/3.1_U1/SUNWscdtk_31u1_s<#>.tar.gz

J. Edit the /etc/system file and add any needed entries. Shared memory sample

entries for popular databases are below.

1. For all systems add the following:

exclude: lofs

set ip:ip_enable_group_ifs=0

2. The cluster software installation will have added a line:

rpcmod:svc_default_stksize=0x6000

change this to read

set rpcmod:svc_default_stksize=0x8000

3. Add a line which reads:

set lwp_default_stksize=0x6000

4. If you are using ce interfaces on the interconnect:

set ce:ce_taskq_disable=1

5. If you are using D1000 shared storage add:

set sd:sd_retry_count=0x2

a. Also add to the /kernel/drv/glm.conf file

scsi-selection-timeout=64;

scsi_reset_delay=500;

6. If you plan on using dynamic reconfiguration add:

set kernel_cage_enable=1

7. For Oracle HA or OPS/RAC:

set shmsys:shminfo_shmmax=0xffffffff (or 0xffffffffffffffff for 64 bit)

set shmsys:shminfo_shmmin=1

set shmsys:shminfo_shmmni=200

set shmsys:shminfo_shmseg=200

set semsys:seminfo_semmap=1024

set semsys:seminfo_semmni=2048

set semsys:seminfo_semmns=2048

set semsys:seminfo_semmnu=2048

set semsys:seminfo_semume=200

set semsys:seminfo_semmsl=2048

set semsys:seminfo_semopm=100

set semsys:seminfo_semvmx=32767

forceload: sys/shmsys

forceload: sys/semsys

forceload: sys/msgsys

8. For Sybase:

set shmsys:shminfo_shmmax=0xffffffff (or 0xffffffffffffffff for 64 bit)

set shmsys:shminfo_shmseg=200

set rlim_fd_cur=1024

9. For Informix:

set shmsys:shminfo_shmmax=4026531839 (3.86GB = max)

set shmsys:shminfo_shmmni=50

set shmsys:shminfo_shmseg=200

K. Because of bug 4897239, add the following line to /etc/syslog.conf

user.warning /var/adm/messages

L. Check that you can connect to the SunPlex Manager on each node:

https://<nodename>:3000/

VI. If using Veritas, Install VxVM and VxFS 3.5

A. Uncompress the software from:

<eiscd3>/sun-internal/progs/veritas-vm/3.5/VxFS_VxVM_3.5_for_Solaris.tar.gz

B. Add the packages and license keys

1. If you are going to use VxFS it must be installed first.

a. pkgadd -d . VRTSvlic VRTSvxfs

b. Edit /etc/system and comment out the entries that were added by VxFS

for rpcmod:svc_default_stksize and lwp_default_stksize (marked in 2

sections labeled vxfs_START) since they are duplicates of our entries

and may have too low a value set

c. If using VxFS on EMC, add:

set cl_comm:rm_thread_stacksize = 0x8000

2. On ALL nodes at once: scvxinstall -i -d <path to software>

3. (Optional) Add Documentation & GUI packages

a. Documentation: pkgadd -d . VRTSvmdoc VRTSfsdoc

b. GUIs: pkgadd -d . VRTSob VRTSobgui VRTSvmpro VRTSfspro

4. add licenses with /sbin/vxlicinst

C. Add patches

1. For VxVM from <eiscd1>/sun/patch/veritas-vm/3.5

2. For VxFS from <eiscd2>/sun2/patch/veritas-fs/3.5/<OS Number>

D. On ONE node: scshutdown -y -g0

Then boot -r the entire cluster

E. vxinstall

1. Choose a Custom Installation

2. Do NOT encapsulate the root drive (will do in a later step)

3. Initialize a small rootdg

a. If you are going to use VxVM to mirror root (later), only add the

root mirror disk as a new disk to rootdg. Name the disk rootmir<Node#>

b. If you are going to use SDS/SVM to mirror root (later), you will

need to initialize another disk or 2 for your rootdg on each node.

You can use these disks for swap or local storage, not shared.

4. Leave all other disks alone

5. Do NOT have vxinstall shutdown and reboot

6. Reminor the device in rootdg to be unique on each node. Set each node to

have a base minor number of 100*<nodenumber>. E.G. node1=100, node2=200

vxdg reminor <x>00

7. Optional - disable hot relocation.

edit /etc/init.d/vxvm-recover

comment out: #vxrelocd root &

uncomment : vxsparecheck root &

8. On ONE node: scshutdown -y -g0

9. boot all nodes

F. If you have multiple paths (controllers) from a single node to shared

disks, and are not using STMS (MPXIO), prevent dmp on those controllers

1. run vxdiskadm

2. choose "Prevent multipathing/Suppress devices from VxVM's view"

3. choose "Prevent multipathing of all disks on a controller by VxVM"

4. suppress each controller, i.e. c2

G. Create your disk groups. For each disk group:

1. vxdiskadd c#t#d# c#t#d# ...

a. When prompted, enter the new disk group name

2. Register the disk group

a. Do NOT register OPS/RAC CVM-shared (with cluster functionality) groups

b. Run scsetup

1. Select "Device groups and volumes"

2. Select "Register a VxVM disk group as a device group"

3. quit out

3. scstat -D to determine the primary

a. run an scswitch -z -D <disk group> -h <nodename> to test failover

4. create your volumes, register and newfs them on the PRIMARY node. Note,

vxfs operations may fail if not done on the primary.

a. vxassist -g <dg> -U fsgen make <vol> <size> layout=mirror-stripe,log

b. Run scsetup to synchronize your volumes

1. Select "Device groups and volumes"

2. Select "Synchronize volume information for a VxVM device group"

c. newfs your volumes.

1. ufs : newfs /dev/vx/rdsk/<dg>/<vol>

2. vxfs: mkfs -F vxfs -o largefiles /dev/vx/rdsk/<dg>/<vol>

H. Mount your volumes

1. mkdir /global/<mnt> ON ALL NODES

2. add entries to vfstab ON ALL NODES THAT ARE DIRECTLY ATTACHED

a. for ufs global:

/dev/vx/dsk/<dg>/<vol> /dev/vx/rdsk/<dg>/<vol> /global/<mnt> ufs 2 yes global,logging

b. for ufs with HAStoragePlus local file system:

/dev/vx/dsk/<dg>/<vol> /dev/vx/rdsk/<dg>/<vol> /global/<mnt> ufs 2 no logging

c. for vxfs global:

/dev/vx/dsk/<dg>/<vol> /dev/vx/rdsk/<dg>/<vol> /global/<mnt> vxfs 2 yes global,log

Note: the log option with vxfs may cause a negative performance impact,

but must me used. Per the release notes, it is unsupported to use the

qlog, delaylog, or tmplog vxfs mount options and logging is required.

d. for vxfs with HAStoragePlus local file system:

/dev/vx/dsk/<dg>/<vol> /dev/vx/rdsk/<dg>/<vol> /global/<mnt> vxfs 2 no log

3. mount the file system ON THE PRIMARY NODE. If you are using vxfs, the

mount may fail if you are not on the primary

a. scstat -D to determine the primary node of the disk group/set

b. mount /global/<mnt>

VII. Install and patch SDS or SVM

A. For Solaris 8, pkgadd -d . SUNWmdr SUNWmdu SUNWmdx (+ SUNWmdg for the gui)

B. Install Solaris 8 SDS patches from <eiscd1>/sun/patch/sds/4.2.1

108693-24 SDS 4.2.1 patch

C. If needed, change /kernel/drv/md.conf file:

nmd=<max number metadevices, up to 1024 for SDS, 8192 for SVM>

md_nsets=<max number of disksets, up to 31>

Do not set these higher than you need, as it will increase your boot time

D. ON ONE NODE shutdown all nodes with : scshutdown -y -g 0

E. boot -r all nodes

F. ON ONE NODE run /usr/cluster/bin/scgdevs

G. Create the local metadb(s)

1. for Sol 9 : metadb -a -f -c 3 c#t#d#s7

for Sol 8 : metadb -a -f -l 8192 -c 3 c#t#d#s7

2. metadb to check

3. If your local metadbs are on an even number of disks or controllers (ie

you could loose exactly 1/2) then please read and review InfoDoc 79046

about adding md:mirrored_root_flag=1 to the /etc/system file.

H. Create your disk sets ON ONE NODE

1. metaset -s <set> -a -h <node1> <node2> ... <node#>

2. Add the disks to your set

a. metaset -s <set> -a /dev/did/rdsk/d# ...

b. To check : metaset -s <set>

metadb -s <set>

3. Repartition the disk using format or prtvtoc and fmthard.

a. slice 7 starts at cylinder 0. DO NOT CHANGE! (contains the set metadb)

b. slice 0 should be the rest of the disk. You can change it

c. you should NOT change slice 2 to be the entire disk. Disks in a disk

set must not have their slice 7 overlap with any other slice.

I. Create your metadevices

1. edit /etc/lvm/md.tab and add your metadevices. Use the /dev/did/rdsk/d#s0

devices

a. example entry for md.tab:

<set>/d101 1 1 /dev/did/rdsk/d4s0

<set>/d102 1 1 /dev/did/rdsk/d5s0

<set>/d100 -m <set>/d101 <set>/d102

2. ON ONE NODE metainit -s <set> -a

3. ON ONE NODE newfs /dev/md/<set>/rdsk/d#

J. If you have any disksets on exactly 2 arrays connected to 2 nodes, you

must configure the dual string mediators on one node

1. metaset -s <set> -a -m <node1> <node2>

2. To check : medstat -s <set>

K. test the failover

1. scstat -D to determine the primary

2. run an scswitch -z -D <disk group> -h <nodename> to test failover

L. mount your file systems

1. mkdir /global/<mnt> ON ALL NODES

2. Add to the vfstab ON ALL NODES DIRECT ATTACHED TO THE STORAGE.

a. for ufs global (with or without HAStoragePlus global):

/dev/md/<set>/dsk/d# /dev/md/<set>/rdsk/d# /global/<mnt> ufs 2 yes global,logging

b. for ufs with HAStoragePlus local file system:

/dev/md/<set>/dsk/d# /dev/md/<set>/rdsk/d# /global/<mnt> ufs 2 no logging

3. mount /global/<mnt> ON ONE NODE

VIII. Configure ha-nfs (logical host/failover/HAStoragePlus example)

A. Install the data service software

1. run scinstall on ALL nodes

2. select "Add support for new data services to this cluster node"

3. select "nfs"

4. There is no need to add any patch at this time. The following patch is

integrated into the software:

Solaris 8 Solaris 9

--------- ---------

115070-01 115071-01 Sun Cluster 3.1: HA-NFS Patch

B. Add your failover hostname/ip to /etc/hosts on ALL nodes

C. make the admin file system ON ONE NODE

1. mkdir -p /global/<mnt>/<nfs-adm>/SUNW.nfs

2. cd /global/<mnt>/<nfs-adm>/SUNW.nfs

3. vi dfstab.<nfs-res>

a. add entries to share /global/<mnt>

D. Register the nfs resource on ONE node

scrgadm -a -t SUNW.nfs

E. Create the nfs resource group on ONE node

scrgadm -a -g <nfs-rg> -h <node1,..> -y PATHPREFIX=/global/<mnt>/<nfs-adm>

F. Add a failover hostname/ip resource to the nfs-rg on ONE node

scrgadm -a -L -g <nfs-rg> -l <failover hostname> -n <ipmp#>@<node1>,...

G. Set up HAStoragePlus (optional)

1. Register the HAStoragePlus resource type

scrgadm -a -t SUNW.HAStoragePlus

2. Create the SUNW.HAStoragePlus resource

scrgadm -a -g <nfs-rg> -j <nfs-hastp-res> -t SUNW.HAStoragePlus \

-x FilesystemMountPoints=/<mnt-point>,</<mnt-point> \

-x AffinityOn=True

a. Please note that the mount points listed must be in the same order

that they are listed in the vfstab file

3. Enable the HAStoragePlus resource with the nfs resource group.

scswitch -Z -g nfs-rg

H. Add the nfs resource to the nfs-rg on ONE node

scrgadm -a -g <nfs-rg> -j <nfs-res> -t SUNW.nfs

or if using HAStoragePlus use

scrgadm -a -g <nfs-rg> -j <nfs-res> -t SUNW.nfs \

-y Resource_Dependencies=<nfs-hastp-res>

I. re-enable the modified nfs-rg on ONE node

scswitch -Z -g <nfs-rg>

J. Check the status

scstat -g

K. Test a manual switch of the resource group

scswitch -z -h <node#> -g <nfs-rg>

scstat -g

IX. Configure ha-apache (shared address/scalable example)

A. Install the data service software using scinstall on ALL nodes

1. There is no need to add any patch at this time. The following patch is

integrated into the software:

Solaris 8 Solaris 9

--------- ---------

115064-01 115065-01 HA-Apache Patch

B. Add your shared hostname/ip to /etc/hosts on all nodes

C. Create your directories on one node

1. mkdir /global/<mnt>/conf

mkdir /global/<mnt>/htdocs

mkdir /global/<mnt>/cgi-bin

chgrp bin /global/<mnt>/cgi-bin

2. cp /etc/apache/httpd.conf-example /global/<mnt>/conf/httpd.conf

3. vi /global/<mnt>/conf/httpd.conf

a. Fix the ServerName line

1. For Solaris 8, uncomment ServerName line and change it to the shared

hostname

2. For Solaris 9, change 127.0.0.1 to the shared hostname

b. change the DocumentRoot to /global/<mnt>/htdocs

c. change the ScriptAlias to /global/<mnt>/cgi-bin/ <- trailing slash

D. Populate and configure the bin, cgi-bin and data directories on ONE NODE

1. populate the directories

a. cp -rp /usr/apache/bin /global/<mnt>

b. cp -r /var/apache/htdocs/* /global/<mnt>/htdocs/

cp /global/<mnt>/htdocs/index.html.en /global/<mnt>/htdocs/index.html

(you can substitute language of your choice)

c. Copy the test-cluster.cgi script to cgi-bin

cp test-cluster.cgi /global/<mnt>/cgi-bin/test-cluster.cgi

f. chmod ugo=rx /global/<mnt>/cgi-bin/test-cluster.cgi

2. Edit the apachectl file to work with a different httpd.conf file

a. cd /global/<mnt>/bin

b. cp apachectl apachectl.orig

c. vi apachectl

1. search for the line which reads:

HTTPD=/usr/apache/bin/httpd

2. change it to read :

HTTPD="/global/<mnt>/bin/httpd -f /global/<mnt>/conf/httpd.conf"

3. Test the configuration

/global/<mnt>bin/apachectl configtest

E. Register the Apache data service

scrgadm -a -t SUNW.apache

F. Create the shared address resource group

scrgadm -a -g <sa-rg> -h <node1,...>

G. Assign an ip address + ipmp group to the shared resource group

scrgadm -a -S -g <sa-rg> -l <shared hostname> -n <ipmp#><@node#>, ...

H. Create another resource group for apache dependent on <sa-rg>

scrgadm -a -g <apache-rg> \

-y Maximum_primaries=2 \

-y Desired_primaries=2 \

-y RG_dependencies=<sa-rg>

I. Add a SUNW.apache resource to the apache-rg

scrgadm -a -j <apache-res> -g <apache-rg> -t SUNW.apache \

-x Bin_Dir=/global/apache/bin \

-y Port_list="80/tcp" \

-y Scalable=True \

-y Network_resources_used=<shared hostname>

J. Bring the shares address and apache resource groups online

1. scswitch -Z -g <sa-rg>

2. scswitch -Z -g <apache-rg>

K. run scstat -g to check the state

The <shared hostname> SharedAddress resource should be online on one node

The <apache-res> resource should be online on both nodes

L. Test the shared connection by visiting

http://<shared hostname>/cgi-bin/test-cluster.cgi

Due to the load balancing feature, you may need to refresh/reload MANY

times to see a connection to a different node. You should try connecting

from multiple clients if possible

X. Configuring the cluster for Oracle RAC or OPS

A. create the dba group and the oracle user

B. add the appropriate packages from the SC31 cd (not ds cd):

1. If using VxVM with Cluster functionality raw devices for data:

pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWcvmr SUNWcvm

2. If using HW raid raw devices for data:

pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWschwr

3. If using a GFS for data

pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr

4. Install patch for the data service:

Solaris 8 Solaris 9

--------- ---------

115062-04## 115063-04## Sun Cluster 3.1: OPS Core Patch

C. pkgadd the oracle udlm from the oracle cdrom or EIS cd3 under

sun-internal/progs/SunCluster

1. pkgadd -d . ORCLudlm

D. On one node, scshutdown -y -g0 and then boot all nodes

E. If using VxVM with CVM functionality

1. Find which node is the Veritas master using : vxdctl -c mode

2. Deport the raw volume disk group : vxdg deport <dg>

3. Import the disk group shared on the master :

vxdg -s import <dg>

vxvol -g <dg> startall

4. Make sure the disk group is seen as shared on both nodes: vxdg list

F. If using GFS, make sure your data volumes are mounted using forcedirectio

G. Install Oracle RAC/OPS

H. Create the RAC framework

1. run scsetup

select Data Services

select Sun Cluster support for Oracle RAC

select Create the RAC framework resource group

Do you want to configure all nodes in the resource group (yes/no) [yes]?

yes to continue

Use the menus to quit out

2. run scstat -g

check that the rac-framework-rg resource group is online on all nodes

check that the rac_framework, rac_udlm, and rac_cvm resources are online

on all nodes

XI. To install other data services. Follow the documentation that you installed

on the admin workstation. You can access the documents using a browser by

using file:///opt/sun_docs/sundocs.html as the url and accessing the

data services collection in PDF or HTML.

A. There is no need to add any patch at this time. The following patches are

integrated into the software:

Solaris 8 Solaris 9

--------- ---------

115066-01 115067-01 Sun Cluster 3.1: HA-DNS

115072-01 115073-01 Sun Cluster 3.1: HA-Netbackup

115076-01 115077-01 Sun Cluster 3.1: HA-Oracle

115078-01 115079-01 Sun Cluster 3.1: HA-SAP

115080-01 115081-01 Sun Cluster 3.1: HA-Sun One Web Server

115082-01 115083-01 Sun Cluster 3.1: HA-BEA Web Logic

115084-01 115085-01 Sun Cluster 3.1: HA-Sun One Application Server

115086-01 115087-01 Sun Cluster 3.1: HA-Sun One Message Queue

115089-01 N/A Sun Cluster 3.1: HA-Sybase Patch

XII. If using VxVM to encapsulate & mirror root

A. Edit /etc/vfstab so that the /global/.devices/node@<nodeid> mounts from

the physical c#t#d#s# instead of the /dev/did device. If you don't know

the device, there should be a /globaldevices line commented out. DO NOT

uncomment this line! Just use the physical device that it originally

mounted as the device in the /global/.devices/node@<nodeid> line. E.G. :

/dev/dsk/c0t0d0s3 /dev/rdsk/c0t0d0s3 /global/.devices/node@1 ufs 2 no global

B. Shut down the cluster (on one node, will shut down both):

scshutdown -y -g 0

C. Boot all nodes outside of the cluster

boot -x

E. Use vxdiskadm option 2 "Encapsulate one or more disks" to encapsulate the

root drive.

1. Name each root disk rootdisk<node#>

2. init 0

3. boot -x

Systems will boot, encapsulate root disk, and reboot (into the cluster)

F. Mirror your root disk using vxdiskadm or use:

/etc/vx/bin/vxmirror rootdisk<node#> rootmir<node#>

XIII. If using SDS/SVM to mirror root

A. metainit -f d1 1 1 <c#t#d#s0 (rootdisk - don't use the did device)>

B. metainit d2 1 1 <c#t#d#s0 (rootmir)>

C. metainit d0 -m d1

D. metaroot d0

E. scshutdown -y -g 0 (on one node)

F. boot all nodes

G. metattach d0 d2

H. Note - you should also mirror swap and /global/.devices/node@<#> (and any

other partitions you used on the root disk). If you do, be sure to use

the physical device for the metainit. Also be sure that the metadevice

you choose for each /global/.devices/node@<#> is unique in the cluster,

since they will be mounted globally. And don't forget to make your changes

in the /etc/vfstab file and the /etc/lvm/md.tab file.

XIV. Complete installation

A. If using SDS/SVM, set up the metacheck script

1. copy from <eiscd1>/sun/tools/MISC/SDS/metacheck.sh

2. enter a crontab entry for it. e.g: 00 06 * * * /metacheck.sh

B. If desired, install SunMC

C. Follow the EIS Cluster Checklist for Install Verification and running

explorer and SunCheckup

D. Use the EISdoc tool to document the cluster.