1/21/13

unixwerk: setup a two-nodes cluster with hacmp

unix we rk > AIX > two-node cluste r

Fe bruary 25, 2012

unixwerk

Se tup a Tw o-Node Cluste r w ith HACMP

AIX BSD Linux Solaris Others Un*x Topics

Contents 1. Introduction 2. Setup and Preparation Storage setup Network setup 3. Installation Prerequisite filesets HACMP Filesets 4. Cluster Topology Configuration Define the Cluster Define the Cluster Nodes Define Cluster Sites Define a Cluster Network Add a Communication Interface for Heartbeat Add Persistent IP Addresses Storage Configuration Disk Heartbeat 5. Resource Group Configuration Application Volume Groups Application Server Cluster Service Address Define Resource Group(s) Create LVs and Filesystems for Applications A. B. C. D. E. Failover Test Disk Heartbeat Check Useful Commands clstat and snmp Related Information

Guestbook Contact Archive

1. Introduction
This article describes how to setup a two-nodes-cluster with IBM's standard cluster solution for AIX. Although the name has changed to Power HA with Version 5.5 and to Power HA System Mirror with version 7 IBM's cluster solution is still widely known as HACMP. This article refers to version 5.5.

2. Setup and Preparation

Storage setup
The reason why we create a cluster is to make an application high available. Therefore we need storage from two independent sites (read it as storage from two different datacenters). In this article we have to sites: Datacenter1 and Datacenter2. Each filesystem will be mirrored over the two sites. All storage has to be visible on both nodes. In addition we need two (very small) LUNs for disk heartbeat. 512MB to 1G LUN size is sufficient.

Network setup
In our setup we have two nodes: barney and shakira. We need a boot address only used for cluster intercommunication, a service address, and a persistent address which is equal to the hostnames of our nodes. All cluster addresses have to be present in the / t / o t file on both nodes: echss

nd1nd2 v /t/ot oe+oe# i echss ## HCP ## AM #Bo ades ot drs 121.. 7.814 brebo anyot 121.. 7.816 saiaot hkrbo #SrieCutrades evc/lse drs 1.1.1.0 hsrie 011117 aevc1 #Nd/essetades oePritn drs 1.1.1. 011114 bre any 1.1.1. 011116 www.unixwerk.eu/aix/setup_hacmp.html saia hkr

1/19

1/21/13
1.1.1. 011116 ## ##

unixwerk: setup a two-nodes cluster with hacmp

saia hkr

Don't use hyphens (-) and underscores (_) in IP labels here.

3. Installation
Installation of Prerequisite Filesets
There are some filesets needed in order to get HACMP to work which are typically not part of a standard AIX installation. Check for

bsntnssre o.e.f.evr bscv o.lm rc.optbschcp stcma.ai.am rc.optcinshcp stcma.let.am

If they are not installed you have to do it now:

nd1nd2 sit isallts oe+oe# mty ntl_aet | bsntns o.e.f | +6110 NtokFl Sse Sre ... ewr ie ytm evr |> +6140 NtokFl Sse Sre ... ewr ie ytm evr | bscv o.lm | +6111 Ehne Cnurn LgclVlm Mngr ... nacd ocret oia oue aae | +6140 Ehne Cnurn LgclVlm Mngr ... nacd ocret oia oue aae |> +6142 Ehne Cnurn LgclVlm Mngr ... nacd ocret oia oue aae

AL| L | | AL| L | | |

| rc.optbsc stcma.ai AL| L | +2540 RC EetMngmn BscFnto ... ST vn aaeet ai ucin | |> +2540 RC EetMngmn BscFnto (AM/SSpot ... ST vn aaeet ai ucin HCPE upr) | | +2540 RC EetMngmn BscFnto (SPSpot ... ST vn aaeet ai ucin PS upr) | | rc.optcins stcma.let AL| L | +2540 RC EetMngmn Cin Fnto ... ST vn aaeet let ucin | |> +2540 RC EetMngmn Cin Fnto (AM/SSpot ... ST vn aaeet let ucin HCPE upr) | | +2540 RC EetMngmn Cin Fnto (SPSpot ... ST vn aaeet let ucin PS upr) | Installation of HACMP Filesets
Put the HACMP filesets and the update filesets somewhere where you can access them from both nodes and run inutoc. Then install the filesets on both cluster nodes:

nd1nd2 c /aht/fs oe+oe# d pt/obf nd1nd2 sit isallts oe+oe# mty ntl_aet |>cutre.let lse.scin | +5500 E Cin Lbais ... S let irre | +5504 E Cin Lbais ... S let irre | +5500 E Cin Rnie ... S let utm | +5505 E Cin Rnie ... S let utm | +5500 E Cin Uiiis ... S let tlte | +5505 E Cin Uiiis ... S let tlte | +5500 E CmuiainIfatutr ... S omncto nrsrcue | +5505 E CmuiainIfatutr ... S omncto nrsrcue | +5500 WbbsdSi ... e ae mt | +5505 WbbsdSi ... e ae mt |>cutre.evr lse.ssre | +5500 E Bs Sre Rnie ... S ae evr utm | +5506 E Bs Sre Rnie ... S ae evr utm | +5500 E Sre Das ... S evr ig | +5505 E Sre Das ... S evr ig | +5500 E Sre Eet ... S evr vns | +5506 E Sre Eet ... S evr vns | +5500 E Sre Uiiis ... S evr tlte | +5506 E Sre Uiiis ... S evr tlte | +5500 E CutrSmltr ... S lse iuao | +5504 E CutrSmltr ... S lse iuao | +5500 E CutrTs To ... S lse et ol | +5503 E CutrTs To ... S lse et ol | +5500 E ToNd CniuainAssat ... S w-oe ofgrto sitn |>cutre.f lse.scs | +5500 E CutrFl Sse Spot ... S lse ie ytm upr | www.unixwerk.eu/aix/setup_hacmp.html +5504 E CutrFl Sse Spot ... S lse ie ytm upr

AL| L | | | | | | | | | | AL| L | | | | | | | | | | | | | AL| L | |

2/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

| +5504 E CutrFl Sse Spot ... S lse ie ytm upr |>cutre.f lse.sns | +5500 E NSSpot ... S F upr | +5501 E NSSpot ... S F upr |>cutre.so lse.scpc | +5500 E CPCCmad ... S SO omns | +5506 E CPCCmad ... S SO omns | +5500 E CPCRnieCmad ... S SO utm omns | +5505 E CPCRnieCmad ... S SO utm omns | +5500 E CPCdh ... S SO s |>cutrlcne lse.ies | +5500 HCPEetoi Lcne ... AM lcrnc ies |>cutrmne_Se lse.a.nU.s | +5500 E MnPgs-US Egih ... S a ae .. nls | +5501 E MnPgs-US Egih ... S a ae .. nls | AL| L | | AL| L | | | | | AL| L | AL| L | |

Note: In the above fileset list HACMP update filesets for SP6 are included. If you installed HACMP from a base CD it's strongly recommended to update HACMP with the latest fixes. Base versions of HACMP are not known to be excessively tested.

nd1nd2 c /aht/pae oe+oe# d pt/oudt nd1nd2 sit udt_l oe+oe# mty paeal
The nodes have to be rebooted now.

nd1nd2 sudw -r oe+oe# hton F

4. Cluster Topology Configuration

Basically the cluster configuration has to be done on only one of our nodes. Only the initial definition and startup has to be done on both nodes. Please mind the command prompt in the below commands. It indicates whether something has to be done on one node or on both nodes.

Define the Cluster

The first step is to define a cluster. This means nothing more then just define the name of our cluster

nd1nd2 sit hcp oe+oe# mty am - Etne Cniuain > xedd ofgrto - Etne Tplg Cniuain > xedd oooy ofgrto - Cniuea HCPCutr > ofgr n AM lse - AdCag/hwa HCPCutr > d/hneSo n AM lse AdCag/hwa HCPCutr d/hneSo n AM lse Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils [lse1 Cutr]

*CutrNm lse ae NT:HCPms b RSATD OE AM ut e ETRE o alndsi odrfrcag t tk efc n l oe n re o hne o ae fet
We follow the advice and restart all cluster related services:

nd1nd2 sosc- cutr oe+oe# tpr g lse 01-4 TecsrgE Sbytmwsrqetdt so. 5304 h ltmrS usse a euse o tp nd1nd2 sosc- ccmE oe+oe# tpr s lodS 01-4 TeccmE Sbytmwsrqetdt so. 5304 h lodS usse a euse o tp nd1nd2 satr - ccmE oe+oe# trsc s lodS 01-5 TeccmE Sbytmhsbe satd SbytmPDi 685. 5309 h lodS usse a en tre. usse I s 173 nd1nd2 satr - cutr oe+oe# trsc g lse 01-5 TecifE Sbytmhsbe satd SbytmPDi 683. 5309 h lnoS usse a en tre. usse I s 154 01-5 TecsrgE Sbytmhsbe satd SbytmPDi 572. 5309 h ltmrS usse a en tre. usse I s 760

Define the Cluster Nodes

All steps so far we did on both nodes. But from now on we only work on one of our nodes.

www.unixwerk.eu/aix/setup_hacmp.html

3/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

We add the first node to our cluster:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Tplg Cniuain > xedd oooy ofgrto - CniueHCPNds > ofgr AM oe - AdaNd t teHCPCutr > d oe o h AM lse AdaNd t teHCPCutr d oe o h AM lse Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils [any bre] [anyot brebo]

*Nd Nm oe ae CmuiainPt t Nd omncto ah o oe

and now the second one:

AdaNd t teHCPCutr d oe o h AM lse Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils *Nd Nm oe ae CmuiainPt t Nd omncto ah o oe [hkr] saia [hkrbo] saiaot

Define Cluster Sites

We don't really use cluster sites in this example setup. But it makes sense to define cluster sites anyway. It gives you the possibility to label your storage. We will use these labels later when we create the application filesystems. First site:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Tplg Cniuain > xedd oooy ofgrto - CniueHCPSts > ofgr AM ie - AdaSt > d ie AdaSt d ie Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils [aaetr] Dtcne1 bre any [e] Ys [oe nn]

*St Nm ie ae *St Nds ie oe *Dmnne oiac *Bcu Cmuiain akp omnctos

Second site:

+ + +

AdaSt d ie Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils [aaetr] Dtcne2 saia hkr [o N] [oe nn]

*St Nm ie ae *St Nds ie oe *Dmnne oiac *Bcu Cmuiain akp omnctos

+ + + +

The home node of our service shall be barney - that's why we set the Dominance to Yes for barney and to No for shakira.

www.unixwerk.eu/aix/setup_hacmp.html

4/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

Define a Cluster Network
Before we start with the network configuration we let HACMP try to discover the topology. Automatic discovery does not always work, but it's worth a try.

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Dsoe HCPrltdIfrainfo Cniue Nds > icvr AM-eae nomto rm ofgrd oe
The network topology is used by HACMP for the heartbeat. First we configure heartbeat over ethernet:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Tplg Cniuain > xedd oooy ofgrto - CniueHCPNtok > ofgr AM ewrs - AdaNtokt teHCPCutr > d ewr o h AM lse +-------------------------------------------------------------------------+ | Slc aNtokTp eet ewr ye | | | |Mv cro t dsrdie adpesEtr oe usr o eie tm n rs ne. | | | | #Dsoeyls promd (aur 3 1:2 icvr at efre: Jnay 0 00) | | #Dsoee I-ae NtokTps icvrd Pbsd ewr ye | | ehr te | | | | #Dsoee Sra Dvc Tps icvrd eil eie ye | | r22 s3 | | | | #PedfndI-ae NtokTps r-eie Pbsd ewr ye | | X_aa Ddt | | X_p Di | | am t | | ehr te | | fd di | | hs p | | i b | | tkn oe | | | | #PedfndSra Dvc Tps r-eie eil eie ye | | X_s3 Dr22 | | dsh ikb | | r22 s3 | | tss mci | | tsa ms | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=mg 8Iae F0Ei 1=xt EtrD ne=o | |/Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
If you trust the automatic discovery select ether under "Discovered IP-based Network Types" - if not select ether under "Pre-defined IP-based Network Types". The latter always work - so it might be the better choice. In the next screen put in the correct netmask and activate the use of IP aliases for IP takeover:

Ada I-ae Ntokt teHCPCutr d n PBsd ewr o h AM lse Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils *NtokNm ewr ae [e_te_1 ntehr0] *NtokTp ewr ye ehr te *Ntak ems [5.5.5.] 2525250 *Eal I AdesTkoe vaI Aiss nbe P drs aevr i P lae [e] Ys I AdesOfe frHatetn oe I Aiss[ P drs fst o erbaig vr P lae ]

+ +

Add a Communication Interface for Heartbeat

Based on the network definition we define the boot addresses as communication interface:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto

www.unixwerk.eu/aix/setup_hacmp.html

5/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

- Etne Tplg Cniuain > xedd oooy ofgrto - CniueHCPCmuiainItrae/eie > ofgr AM omncto nefcsDvcs - AdCmuiainItrae/eie > d omncto nefcsDvcs - AdPedfndCmuiainItrae adDvcs > d r-eie omncto nefcs n eie - CmuiainItrae > omncto nefcs +-------------------------------------------------------------------------+ | Slc aNtokNm eet ewr ae | | | |Mv cro t dsrdie adpesEtr oe usr o eie tm n rs ne. | | | | AL L | | ntehr0 e_te_1 | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=mg 8Iae F0Ei 1=xt EtrD ne=o | |/Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
Select net_ether_01 and fill the empty fields:

Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils [anyot brebo] ehr te ntehr0 e_te_1 [any bre] [n] e8

*I LblAdes P ae/drs *NtokTp ewr ye *NtokNm ewr ae *Nd Nm oe ae NtokItrae ewr nefc
Do the same for the second node:

Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils [hkrbo] saiaot ehr te ntehr0 e_te_1 [hkr] saia [n] e8

*I LblAdes P ae/drs *NtokTp ewr ye *NtokNm ewr ae *Nd Nm oe ae NtokItrae ewr nefc

The network topology is setup now - time to synchronize the cluster:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Vrfcto adSnhoiain > xedd eiiain n ycrnzto HCPVrfcto adSnhoiain AM eiiain n ycrnzto Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. *Vrf,Snhoieo Bt eiy ycrnz r oh *Atmtclycreterr fuddrn uoaial orc ros on uig vrfcto? eiiain *Fresnhoiaini vrfcto fis oc ycrnzto f eiiain al? *Vrf cagsol? eiy hne ny *Lgig ogn [nr Fed] Ety ils [oh Bt] [o N] + +

[o N] [o N] [tnad Sadr]

+ + +

Add Persistent IP Addresses

We want to have the IPs belonging to the hostnames of our two nodes to be persistent:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Tplg Cniuain > xedd oooy ofgrto - CniueHCPPritn Nd I Lbl/drse > ofgr AM esset oe P aesAdess

www.unixwerk.eu/aix/setup_hacmp.html

- AdaPritn Nd I Lbl > d esset oe P ae

6/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

- AdaPritn Nd I Lbl > d esset oe P ae AdaPritn Nd I LblAdes d esset oe P ae/drs Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils bre any [e_te_1 ntehr0] [any bre] [ ]

*Nd Nm oe ae *NtokNm ewr ae *Nd I LblAdes oe P ae/drs Pei Lnt rfx egh
We do the same for shakira.

+ + #

We miss a default route here. Since the persistent IP is defined within HACMP there is no default route defined in the ODM. However, after a reboot the system comes up with boot and persistent address. So we define a default route on both nodes:

nd1nd2 cdv- ie0- ruent-ocut0,,011111oe+oe# he l nt a ot=e,hpon,,01.1.1. P

This sets 10.111.111.1 as the default gateway. We will activate the route later with the cluster start. In normal operation you don't have to touch the default route anymore.

Storage Configuration
First we set PVIDs on every LUN we want to use for HACMP and run cfgmgr on the other node.

nd1 cdv- hik - p=e oe# he l ds1 a vys hik cagd ds1 hne nd1 cdv- hik - p=e oe# he l ds2 a vys hik cagd ds2 hne : :
On node2 we have to remove the hdisks first an run cfgmgr again. Now we see the same PVIDs as on node1:

nd2 rdv-lhik oe# me d ds1 hik dltd ds1 eee : : nd2 cgg oe# fmr hik ds0 0c2b3910 072c8f7f hik ds1 0f488f4d 06134350 hik ds2 0f488f42 06134361 hik ds3 0f488f46 0613436c hik ds4 0f488f4b 06134360 hik ds5 0f488f4f 06134362 hik ds6 0f488f4c 061344fa hik ds7 0f488f51 06134405 hik ds8 0f488f55 06134404 hik ds9 0f488f58 0613440f hik0 ds1 0f488f5c 0613440a hik1 ds1 0f488f43 06134379 hik2 ds1 0f488f5f 0613440f
and we run the automatic discovery again:

rov otg Nn oe Nn oe Nn oe Nn oe Nn oe Nn oe Nn oe Nn oe Nn oe Nn oe Nn oe Nn oe

atv cie

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Dsoe HCPrltdIfrainfo Cniue Nds > icvr AM-eae nomto rm ofgrd oe
Now we connect the LUNs to our cluster sites. For every LUN do the following:

nd1 sit hcp oe# mty am - Sse Mngmn (-PC > ytm aaeet CSO) - HCPPyia Vlm Mngmn > AM hscl oue aaeet - CniueDs/ieLctosfrCosSt LMMroig > ofgr ikSt oain o rs-ie V irrn - AdDs/ieDfnto frCosSt LMMroig > d ikSt eiiin o rs-ie V irrn +-------------------------------------------------------------------------+ | www.unixwerk.eu/aix/setup_hacmp.html St Nms ie ae |

7/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

| St Nms ie ae | | | |Mv cro t dsrdie adpesEtr oe usr o eie tm n rs ne. | | | | Dtcne1 aaetr | | Dtcne2 aaetr | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=mg 8Iae F0Ei 1=xt EtrD ne=o | |/Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
Select site Datacenter1 and select all LUNs located there in the next screen:

AdDs/ieDfnto frCosSt LMMroig d ikSt eiiin o rs-ie V irrn Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils Dtcne1 aaetr +

*St Nm ie ae *DssPI ik VD

<F4> gives you a list of all LUNs configured for HACMP - select the ones for site Datacenter1:

+-------------------------------------------------------------------------+ | DssPI ik VD | | | |Mv cro t dsrdie adpesF. oe usr o eie tm n rs 7 | | OEO MR iescnb slce. N R OE tm a e eetd | |PesEtrATRmkn alslcin. rs ne FE aig l eetos | | | |>0f488f4d (hik o alslce nds) 06134350 ds1 n l eetd oe | |>0f488f42 (hik o alslce nds) 06134361 ds2 n l eetd oe | |>0f488f46 (hik o alslce nds) 0613436c ds3 n l eetd oe | |>0f488f4b (hik o alslce nds) 06134360 ds4 n l eetd oe | |>0f488f4f (hik o alslce nds) 06134362 ds5 n l eetd oe | |>0f488f43 (hik1o alslce nds) 06134379 ds1 n l eetd oe | | 0f488f4c (hik o alslce nds) 061344fa ds6 n l eetd oe | | 0f488f51 (hik o alslce nds) 06134405 ds7 n l eetd oe | | 0f488f55 (hik o alslce nds) 06134404 ds8 n l eetd oe | | 0f488f58 (hik o alslce nds) 0613440f ds9 n l eetd oe | | 0f488f5c (hik0o alslce nds) 0613440a ds1 n l eetd oe | | 0f488f5f (hik2o alslce nds) 0613440f ds1 n l eetd oe | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=eet 7Slc F=mg 8Iae F0Ei 1=xt | |EtrD ne=o /Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
We repeat the procedure for the LUNs located in site Datacenter2.

Disk Heartbeat
Two of our LUNs are dedicated to disk heartbeat. Typically you use small LUN sizes here. If you're not sure which LUNs are the heartbeat LUNs check with " b o i f - h i k X ". otno s ds<> To protect the LUNs for disk heartbeat we create volume groups for them - a separate VG for each LUN:

nd1 sit hcp oe# mty am - Sse Mngmn (-PC > ytm aaeet CSO) - HCPCnurn LgclVlm Mngmn > AM ocret oia oue aaeet - Cnurn Vlm Gop > ocret oue rus - Cet aCnurn Vlm Gop > rae ocret oue ru +-------------------------------------------------------------------------+ | Nd Nms oe ae | | | |Mv cro t dsrdie adpesF. oe usr o eie tm n rs 7 | | OEO MR iescnb slce. N R OE tm a e eetd | |PesEtrATRmkn alslcin. rs ne FE aig l eetos | | www.unixwerk.eu/aix/setup_hacmp.html |

8/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

| | |>bre any | |>saia hkr | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=eet 7Slc F=mg 8Iae F0Ei 1=xt | |EtrD ne=o /Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
Select both nodes.

+-------------------------------------------------------------------------+ | Pyia Vlm Nms hscl oue ae | | | |Mv cro t dsrdie adpesF. oe usr o eie tm n rs 7 | | OEO MR iescnb slce. N R OE tm a e eetd | |PesEtrATRmkn alslcin. rs ne FE aig l eetos | | | | 0f488f4d (hik o alslce nds) 06134350 ds1 n l eetd oe | | 0f488f42 (hik o alslce nds) 06134361 ds2 n l eetd oe | | 0f488f46 (hik o alslce nds) 0613436c ds3 n l eetd oe | | 0f488f4b (hik o alslce nds) 06134360 ds4 n l eetd oe | | 0f488f4f (hik o alslce nds) 06134362 ds5 n l eetd oe | | 0f488f43 (hik1o alslce nds) 06134379 ds1 n l eetd oe | | 0f488f4c (hik o alslce nds) 061344fa ds6 n l eetd oe | | 0f488f51 (hik o alslce nds) 06134405 ds7 n l eetd oe | | 0f488f55 (hik o alslce nds) 06134404 ds8 n l eetd oe | | 0f488f58 (hik o alslce nds) 0613440f ds9 n l eetd oe | | 0f488f5c (hik0o alslce nds) 0613440a ds1 n l eetd oe | | 0f488f5f (hik2o alslce nds) 0613440f ds1 n l eetd oe | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=eet 7Slc F=mg 8Iae F0Ei 1=xt | |EtrD ne=o /Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
We select the small LUN from Datacenter1 and fill the empty fields in the next screen:

Cet aCnurn Vlm Gopwt Dt Pt Dvcs rae ocret oue ru ih aa ah eie Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils bre,saia any hkr 0f488f43 06134379 [am_b] hcph1 4 [8 3] tu re fle as

Nd Nms oe ae PI VD VLM GOPnm OUE RU ae Pyia priinSZ i mgbts hscl atto IE n eaye Vlm gopMJRNME oue ru AO UBR Ehne Cnurn Md nacd ocret oe Eal CosSt LMMroigVrfcto nbe rs-ie V irrn eiiain

+ # + +

Wrig ann: Cagn tevlm gopmjrnme myrsl hnig h oue ru ao ubr a eut i tecmadbigual t eeue n h omn en nbe o xct scesul o and ta de nthv te ucsfly n oe ht os o ae h mjrnme cretyaalbe Pes cek ao ubr urnl vial. lae hc fracmol aalbemjrnme o alnds o omny vial ao ubr n l oe| bfr cagn ti stig eoe hnig hs etn.
The same procedure has to be done for the second disk heartbeat LUN. We call the second volume group " h c p h 2". am_b Before we go on with the disk heartbeat configuration we let HACMP discover first...

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Dsoe HCPrltdIfrainfo Cniue Nds > icvr AM-eae nomto rm ofgrd oe
Now we are ready to configure the disk heartbeat:

nd1 sit hcp oe# mty am

www.unixwerk.eu/aix/setup_hacmp.html

9/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

- Etne Tplg Cniuain > xedd oooy ofgrto - CniueHCPCmuiainItrae/eie > ofgr AM omncto nefcsDvcs - AdCmuiainItrae/eie > d omncto nefcsDvcs - AdDsoee CmuiainItraeadDvcs > d icvrd omncto nefc n eie - CmuiainDvcs > omncto eie +-------------------------------------------------------------------------+ | Slc Pitt-on Pi o Dsoee CmuiainDvcst Ad | eet on-oPit ar f icvrd omncto eie o d | | |Mv cro t dsrdie adpesF.Uearwky t srl. oe usr o eie tm n rs 7 s ro es o col | | OEO MR iescnb slce. N R OE tm a e eetd | |PesEtrATRmkn alslcin. rs ne FE aig l eetos | | | | #Nd oe Dvc eie Dvc Pt eie ah Pi vd | | bre any ty t0 /e/t0 dvty | | saia hkr ty t0 /e/t0 dvty | |> bre any hik1 /e/ds1 ds1 dvhik1 0f488f43 06134379 | | bre any hik2 /e/ds1 ds1 dvhik2 0f488f5f 0613440f | |> saia hkr hik1 /e/ds1 ds1 dvhik1 0f488f43 06134379 | | saia hkr hik2 /e/ds1 ds1 dvhik2 0f488f5f 0613440f | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=eet 7Slc F=mg 8Iae F0Ei 1=xt | |EtrD ne=o /Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
We choose the first pair of disks and repeat the procedure for the second pair.

5. Resource Group Configuration

Before we actually define a resource group we prepare all the resources we need:

Application Volume Groups

The first resource we need is a high available application volume group:

nd1 sit hcp oe# mty am - Sse Mngmn (-PC > ytm aaeet CSO) - HCPLgclVlm Mngmn > AM oia oue aaeet - Sae Vlm Gop > hrd oue rus - Cet aSae Vlm Gopwt Dt Pt Dvcs > rae hrd oue ru ih aa ah eie +-------------------------------------------------------------------------+ | Nd Nms oe ae | | | |Mv cro t dsrdie adpesF. oe usr o eie tm n rs 7 | | OEO MR iescnb slce. N R OE tm a e eetd | |PesEtrATRmkn alslcin. rs ne FE aig l eetos | | | |>bre any | |>saia hkr | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=eet 7Slc F=mg 8Iae F0Ei 1=xt | |EtrD ne=o /Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
Select both nodes as shown in the screen above and select the hdisks you need in the next screen. Choose one set of disks from Datacenter1 and one set of disks from Datacenter2. Unfortunately in this screen the location is not indicated. In this example we just select all available disks:

+-------------------------------------------------------------------------+ | Pyia Vlm Nms hscl oue ae | | | |Mv cro t dsrdie adpesF. oe usr o eie tm n rs 7 | | OEO MR iescnb slce. N R OE tm a e eetd | |PesEtrATRmkn alslcin. rs ne FE aig l eetos | | | |>0f488f4d (hik o alslce nds) 06134350 ds1 n l eetd oe | |>0f488f42 (hik o alslce nds) 06134361 ds2 n l eetd oe | |>0f488f46 (hik o alslce nds) 0613436c ds3 n l eetd oe | |>0f488f4b (hik o alslce nds) 06134360 ds4 n l eetd oe www.unixwerk.eu/aix/setup_hacmp.html |

10/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

|>0f488f4b (hik o alslce nds) 06134360 ds4 n l eetd oe | |>0f488f4f (hik o alslce nds) 06134362 ds5 n l eetd oe | |>0f488f4c (hik o alslce nds) 061344fa ds6 n l eetd oe | |>0f488f51 (hik o alslce nds) 06134405 ds7 n l eetd oe | |>0f488f55 (hik o alslce nds) 06134404 ds8 n l eetd oe | |>0f488f58 (hik o alslce nds) 0613440f ds9 n l eetd oe | |>0f488f5c (hik0o alslce nds) 0613440a ds1 n l eetd oe | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=eet 7Slc F=mg 8Iae F0Ei 1=xt | |EtrD ne=o /Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
The next screen asks for the type of volume group. These days scalable VGs seem to be the best choice:

+-------------------------------------------------------------------------+ | Vlm GopTp oue ru ye | | | |Mv cro t dsrdie adpesEtr oe usr o eie tm n rs ne. | | | | Lgc eay | | Oiia rgnl | | Bg i | | Saal clbe | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=mg 8Iae F0Ei 1=xt EtrD ne=o | |/Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
After we selected disks and VG type we choose a name for the volume group:

Cet aSae Vlm Gopwt Dt Pt Dvcs rae hrd oue ru ih aa ah eie Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils bre,saia any hkr 0f488f4d 0f> 06134350 06 [pl1g ap0v] 18 2 + [2 4] # tu re +

Nd Nms oe ae PI VD VLM GOPnm OUE RU ae Pyia priinSZ i mgbts hscl atto IE n eaye Vlm gopMJRNME oue ru AO UBR Eal CosSt LMMroigVrfcto nbe rs-ie V irrn eiiain Wrig ann: Cagn tevlm gopmjrnme myrsl hnig h oue ru ao ubr a eut i tecmadbigual t eeue n h omn en nbe o xct scesul o and ta de nthv te ucsfly n oe ht os o ae h mjrnme cretyaalbe Pes cek ao ubr urnl vial. lae hc fracmol aalbemjrnme o alnds o omny vial ao ubr n l oe bfr cagn ti stig eoe hnig hs etn.

After confirming with <ENTER> we are done with the VG and can go on with the

Application Server
For the application servers we first need application start and stop scripts. The scripts are usually provided by the application owners and should match at least two conditions: it should be no problem to run these scripts multiple times in succession. particularly the stop script should be robust, i.e. it should really be able to stop the application. If HACMP cannot unmount filesystems a manual takeover (aka resource group move) will fail. Once the scripts are in place we can configure the application server:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Rsuc Cniuain > xedd eore ofgrto - HCPEtne RsucsCniuain > AM xedd eore ofgrto - CniueHCPApiain > ofgr AM plctos - CniueHCPApiainSres > ofgr AM plcto evr - Ada ApiainSre > d n plcto evr

www.unixwerk.eu/aix/setup_hacmp.html

11/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

AdApiainSre d plcto evr Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. *Sre Nm evr ae *SatSrp tr cit *So Srp tp cit ApiainMntrNm() plcto oio aes [nr Fed] Ety ils [p_r0] apsv1 [echcpsatsv1 /t/am/tr_r0] [echcpso_r0] /t/am/tpsv1 +

In the above example the start/stop scripts are stored in a folder / t / a m . But you can place them echcp anywhere in the local filesystem tree. Don't place them on shared filesystems! Since the scripts are local we have to copy them over to the other node:

nd1 sp-p/t/am nd2/t/ oe# c r echcp oe:ec

Cluster Service Address(es)

The cluster service address is the IP address that clients use to connect to the application. Therefore a service address moves with the resource group. You can define more than one service address per resource group. In this example we define only one service address. Remember: we already defined the service address in / t / o t with the initial network setup. echss

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Rsuc Cniuain > xedd eore ofgrto - HCPEtne RsucsCniuain > AM xedd eore ofgrto - CniueHCPSrieI Lbl/drse > ofgr AM evc P aesAdess - AdaSrieI LblAdes > d evc P ae/drs +-------------------------------------------------------------------------+ | Slc aSrieI LblAdestp eet evc P ae/drs ye | | | |Mv cro t dsrdie adpesEtr oe usr o eie tm n rs ne. | | | | Cniual o Mlil Nds ofgrbe n utpe oe | | Budt aSnl Nd on o ige oe | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=mg 8Iae F0Ei 1=xt EtrD ne=o | |/Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
As the said before, the service address needs to move with the application - so we select "Configurable on Multiple Nodes" here.

AdaSrieI LblAdescniual o Mlil Nds(xedd d evc P ae/drs ofgrbe n utpe oe etne) Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils *I LblAdes P ae/drs hsrie aevc1 *NtokNm ewr ae ntehr0 e_te_1 AtraeH Adest acmayI LblAdes[ lent W drs o copn P ae/drs ]
Now we have all resources in place we finally can

Define Resource Group(s) nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Rsuc Cniuain > xedd eore ofgrto - HCPEtne Rsuc GopCniuain > AM xedd eore ru ofgrto - AdaRsuc Gop > d eore ru AdaRsuc Gop(xedd d eore ru etne)

www.unixwerk.eu/aix/setup_hacmp.html

12/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils [G0] R_1 [goe inr] [any bre] [hkr] saia + + +

*Rsuc GopNm eore ru ae ItrSt Mngmn Plc ne-ie aaeet oiy *PriiaigNdsfo PiaySt atcptn oe rm rmr ie PriiaigNdsfo ScnaySt atcptn oe rm eodr ie SatpPlc tru oiy Floe Plc alvr oiy Flbc Plc alak oiy

Oln O Hm Nd O + nie n oe oe > Floe T Nx Pi>+ alvr o et ro NvrFlbc ee alak +

In this panel we initially define name of the resource group (RG_01 here). The policy definitions on the bottom are typical to two-node clusters. But you could choose different values here. For HACMP insiders: The above setup is the classic cascading setup. Time again to let HACMP collect information:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Dsoe HCPrltdIfrainfo Cniue Nds > icvr AM-eae nomto rm ofgrd oe
Now we want to adjust some parameters of our resource group:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Rsuc Cniuain > xedd eore ofgrto - HCPEtne Rsuc GopCniuain > AM xedd eore ru ofgrto - Cag/hwRsucsadAtiue fraRsuc Gop > hneSo eore n trbts o eore ru +-------------------------------------------------------------------------+ | Cag/hwRsucsadAtiue fraRsuc Gop hneSo eore n trbts o eore ru | | | |Mv cro t dsrdie adpesEtr oe usr o eie tm n rs ne. | | | | R_1 G0 | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=mg 8Iae F0Ei 1=xt EtrD ne=o | |/Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+

Cag/hwAlRsucsadAtiue fraCso Rsuc Gop hneSo l eore n trbts o utm eore ru Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [O] TP Rsuc GopNm eore ru ae Itrst Mngmn Plc ne-ie aaeet oiy PriiaigNdsfo PiaySt atcptn oe rm rmr ie PriiaigNdsfo ScnaySt atcptn oe rm eodr ie SatpPlc tru oiy Floe Plc alvr oiy Flbc Plc alak oiy SrieI Lbl/drse evc P aesAdess ApiainSres plcto evr Vlm Gop oue rus Uefre vro o vlm gop,i ncsay s ocd ayn f oue rus f eesr AtmtclyIpr Vlm Gop uoaial mot oue rus Flsses(mt i ALfrVsseiid ieytm epy s L o G pcfe) FlssesCnitnyCek ieytm ossec hc FlssesRcvr Mto ieytm eoey ehd Flssesmutdbfr I cniue ieytm one eoe P ofgrd FlssesDrcoist Epr ieytm/ietre o xot [nr Fed] Ety ils R_1 G0 inr goe bre any saia hkr Oln O Hm Nd O nie n oe oe > Floe T Nx Pi> alvr o et ro NvrFlbc ee alak [aevc1 hsrie] [p_r0] apsv1 [pl1g ap0v] tu re fle as [ ] lgeo ord sqeta eunil fle as [ ] + + + + + + + + + + +

www.unixwerk.eu/aix/setup_hacmp.html

13/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

FlssesDrcoist NSMut ieytm/ietre o F on NtokFrNSMut ewr o F on [OE.1] MR..0 [ ] [ ] +

In the above smit panel we assign our service address and the application server we just created ( Application Server) and set the varyon policy to forced. Finally we synchronize the cluster to the other node:

nd1 sit hcp oe# mty am - Etne Cniuain > xedd ofgrto - Etne Vrfcto adSnhoiain > xedd eiiain n ycrnzto HCPVrfcto adSnhoiain AM eiiain n ycrnzto Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils [oh Bt] [o N]

*Vrf,Snhoieo Bt eiy ycrnz r oh *Atmtclycreterr fuddrn uoaial orc ros on uig vrfcto? eiiain *Fresnhoiaini vrfcto fis oc ycrnzto f eiiain al? *Vrf cagsol? eiy hne ny *Lgig ogn

+ +

[o N] [o N] [tnad Sadr]

+ + +

At this point the cluster is synchronized and in a consistent state. Both nodes have the same information about the cluster setup.

Create LVs and Filesystems for Applications

We want to use CSPOC to create the application filesystems. In order to make use of CSPOC we first start hacmp on both nodes:

nd1nd2 sit csat oe+oe# mty ltr Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils nw o [any bre] tu re Tu re fle as fle as Itrciey neatvl

*Satnw o sse rsato bt tr o, n ytm etr r oh SatCutrSrie o teends tr lse evcs n hs oe BODATmsaea satp RACS esg t tru? SatpCutrIfrainDeo? tru lse nomto amn Raqiersucsatrfre dw ? ecur eore fe ocd on Inr vrfcto err? goe eiiain ros Atmtclycreterr fuddrn uoaial orc ros on uig cutrsat lse tr?

+ + + + + + +

To activate the route we defined earlier ( Add Persistent IP Addresses) we issue the command

nd1nd2 mdv- ie0 oe+oe# ke l nt

s a i aon the other node. hkr

Once the cluster is up we go on with creating LVs and filesystems. If you don't want to use inline jfs2 logs, first a log device has to be created (if you don't do this a log LV called l g v 0will be automatically created ol0 with the first filesystem). The procedure to create a log LV is the same as for a regular filesystem with two exceptions: Use jfs2log as Logical volume TYPE Don't forget to format the jfs2log:

nd1 lgom/e/plg1jslg oe# ofr dvapv0_f2o lgom dsry/e/apv0jslg()y ofr: eto dvrplg_f2o y?
Refer to the next section on how to create the LV a p v 0 _ f 2 o and remember to set the right Logical plg1jslg volume TYPE. Now we are ready to create the application filesystems. The below example shows how to create one filesystem. Repeat the steps until all filesystems are setup. Remember to create a jfs2log for each volume group first (if you don't use inline logs).

www.unixwerk.eu/aix/setup_hacmp.html

14/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

nd1 sit hcp oe# mty am - Sse Mngmn (-PC > ytm aaeet CSO) - HCPLgclVlm Mngmn > AM oia oue aaeet - Sae LgclVlms > hrd oia oue - AdaSae LgclVlm > d hrd oia oue +-------------------------------------------------------------------------+ | Sae Vlm GopNms hrd oue ru ae | | | |Mv cro t dsrdie adpesEtr Uearwky t srl. | oe usr o eie tm n rs ne. s ro es o col | | | #eoreGop Rsuc ru Vlm Gop oue ru | | R_1 G0 ap0v pl1g | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=mg 8Iae F0Ei 1=xt EtrD ne=o | |/Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+
Select one pair of disks from both sites - mark with <F7>:

+-------------------------------------------------------------------------+ | Pyia Vlm Nms hscl oue ae | | | |Mv cro t dsrdie adpesF. oe usr o eie tm n rs 7 | | OEO MR iescnb slce. N R OE tm a e eetd | |PesEtrATRmkn alslcin. rs ne FE aig l eetos | | | | At-eet uoslc | |>bre hik any ds1 Dtcne1 aaetr | | bre hik any ds2 Dtcne1 aaetr | | bre hik any ds3 Dtcne1 aaetr | | bre hik any ds4 Dtcne1 aaetr | | bre hik any ds5 Dtcne1 aaetr | |>bre hik any ds6 Dtcne2 aaetr | | bre hik any ds7 Dtcne2 aaetr | | bre hik any ds8 Dtcne2 aaetr | | bre hik any ds9 Dtcne2 aaetr | | bre hik0 any ds1 Dtcne2 aaetr | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=eet 7Slc F=mg 8Iae F0Ei 1=xt | | | +-------------------------------------------------------------------------+
Warning: Don't use Auto-select here - although we assigned LUNs to sites it's not guaranteed that CSPOC selects LUNs from different sites!

AdaSae LgclVlm d hrd oia oue Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils R_1 G0 ap0v pl1g bre any [0 8] hik hik ds1 ds6 [pl0] apv1 [f2 js] mdl ide mnmm iiu [ ] 2 atv cie src tit

Rsuc GopNm eore ru ae VLM GOPnm OUE RU ae Rfrnend eeec oe *Nme o LGCLPRIIN ubr f OIA ATTOS PYIA VLM nms HSCL OUE ae Lgclvlm NM oia oue AE Lgclvlm TP oia oue YE PSTO o pyia vlm OIIN n hscl oue RNEo pyia vlms AG f hscl oue MXMMNME o PYIA VLMS AIU UBR f HSCL OUE t uefralcto o s o loain Nme o CPE o ec lgcl ubr f OIS f ah oia priin atto Mro WieCnitny irr rt ossec? Alct ec lgclpriincp loae ah oia atto oy o aSPRT pyia vlm? n EAAE hscl oue

+ + + # + + +

RLCT telgclvlm drn rognzto? ys EOAE h oia oue uig eraiain e

www.unixwerk.eu/aix/setup_hacmp.html

15/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

Lgclvlm LBL oia oue AE MXMMNME o LGCLPRIIN AIU UBR f OIA ATTOS Eal BDBOKrlcto? nbe A LC eoain SHDLN PLC frraigwiig CEUIG OIY o edn/rtn lgclpriincpe oia atto ois Eal WIEVRF? nbe RT EIY Srp Sz? tie ie
On the just created LV we create a filesystem:

[ ] [1] 52 n o prle aall n o [o Srpd Nt tie]

+ + + +

nd1 sit hcp oe# mty am - Sse Mngmn (-PC > ytm aaeet CSO) - HCPLgclVlm Mngmn > AM oia oue aaeet - Sae Fl Sses > hrd ie ytm - Ehne JunldFl Sses > nacd orae ie ytm - Ada Ehne JunldFl Sse o aPeiul DfndLgclVlm > d n nacd orae ie ytm n rvosy eie oia oue +-------------------------------------------------------------------------+ | LgclVlm Nms oia oue ae | | | |Mv cro t dsrdie adpesEtr oe usr o eie tm n rs ne. | | | | apv1 bre,hkr pl0 anysaia | | | |F=ep 1Hl F=ers 2Rfeh F=acl 3Cne | |F=mg 8Iae F0Ei 1=xt EtrD ne=o | |/Fn =id nFn Nx =id et | +-------------------------------------------------------------------------+

Ada Ehne JunldFl Sse o aPeiul DfndLgclVlm d n nacd orae ie ytm n rvosy eie oia oue Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils bre,hkr anysaia apv1 pl0 [ap0/s1 /pl1f0] ra/rt edwie [ ] 49 06 n o [ ]

Nd Nms oe ae LGCLVLM nm OIA OUE ae *MUTPIT ON ON PRISOS EMSIN MutOTOS on PIN BokSz (ye) lc ie bts Iln Lg nie o? Iln Lgsz (Bts nie o ie Mye)

+ + + + #

Repeat the steps until all filesystems are setup. Our cluster is ready for use now.

Appendix

A. Failover Test
A cluster failover test is typically done in three or four phases:

1. Manual Failover
The manual failover is the most important test for a cluster configuration. This test can be invoked on one node by

nd1 sit cso oe# mty ltp So CutrSrie tp lse evcs Tp o slc vle i etyfed. ye r eet aus n nr ils PesEtrATRmkn aldsrdcags rs ne FE aig l eie hne. [nr Fed] Ety ils nw o [any bre] tu re Mv Rsuc Gop oe eore rus

*So nw o sse rsato bt tp o, n ytm etr r oh So CutrSrie o teends tp lse evcs n hs oe BODATcutrsudw? RACS lse hton *Slc a Ato o Rsuc Gop eet n cin n eore rus www.unixwerk.eu/aix/setup_hacmp.html

+ + + +

16/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

*Slc a Ato o Rsuc Gop eet n cin n eore rus Mv Rsuc Gop oe eore rus +

When stopping the cluster on node 1 the first thing executed is the cluster stop script. It brings down the applications and unmounts all application filesystems. If your application stop script is not able to stop all application processes some filesystems can't be unmounted and the failover fails. When all resources are down on node 1 HACMP starts to bring up all resources on node 2. The application start script is the last thing hacmp does. Check that your application is working properly and that all clients can connect. If so the first phase of the failover test is completed.

2. Manual Failback
Switch the resources back to the home node. Again check if everything is fine.

3. Automatic Failover
This test simulates a hardware failure on the active node. The easiest way to simulate is to issue the command

nd1 hl oe# at q
on the active node. Check that everything will be brought up on node 2.

4. Partial Hardware Failure

Sometimes only a component fails. Maybe a network switch fails or a storage system becomes unavailable. Test these scenarios to make sure that HACMP is correctly setup - and only starts a failover if needed. These tests also check if your VGs are correctly mirrored over two sites.

B. Disk Heartbeat Check

This is an example on how to check the disk heartbeat. On the first node we set the heartbeat disk to receive mode

nd1 /s/bnrc/i/h_ed- /e/ds1 oe# ursi/stbndbra p dvhik1 r DBCASCMD H LSI OE Frtnd bt ofe:640 is oe ye fst 14 Scn nd bt ofe:696 eod oe ye fst 27 Hnsaigbt ofe:604 adhkn ye fst 52 Ts bt ofe:652 et ye fst 41 RcieMd: eev oe Wiigfrrsos ... atn o epne Mgcnme =08642 ai ubr x7531 Mgcnme =08642 ai ubr x7531 Mgcnme =08642 ai ubr x7531 Mgcnme =08642 ai ubr x7531 Mgcnme =08642 ai ubr x7531 Mgcnme =08642 ai ubr x7531 Mgcnme =08642 ai ubr x7531 Mgcnme =08642 ai ubr x7531 Ln oeaignral ik prtn omly
and on the other node we set the same disk to transmit mode...

nd2 /s/bnrc/i/h_ed- /e/ds1 oe# ursi/stbndbra p dvhik1 t DBCASCMD H LSI OE Frtnd bt ofe:640 is oe ye fst 14 Scn nd bt ofe:696 eod oe ye fst 27 Hnsaigbt ofe:604 adhkn ye fst 52 Ts bt ofe:652 et ye fst 41 Tasi Md: rnmt oe Mgcnme =08642 ai ubr x7531 Dtce rmt uiiyi rciemd. Wiigfrrsos ... eetd eoe tlt n eev oe atn o epne Mgcnme =08642 ai ubr x7531 Mgcnme =08642 ai ubr x7531 Ln oeaignral ik prtn omly
The last line in the above output indicates that the disk heartbeat is working properly.

www.unixwerk.eu/aix/setup_hacmp.html

17/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

C. Useful Commands
This is only a brief and selective list of commands that might be useful when working with HACMP Which node is owning a resource group?

#/s/bncutruiiiscRif ursi/lse/tlte/lGno ---------------------------------------------------------------------------GopNm Sae ru ae tt Nd oe ---------------------------------------------------------------------------R_1 G0 OLN NIE bre any OFIE saia FLN hkr
Move a resource group to another node

#/s/bncutruiiiscRmv - <G - <OE ursi/lse/tlte/lGoe g R> n ND> m

Stop cluster service (on current node)

#sit cso mty ltp

Start cluster service (on current node)

#sit csat mty ltr

Overview cluster state

#/s/bncutrcsa ursi/lse/ltt a
Overview cluster state

#/s/bncutruiiisclslg ursi/lse/tlte/litos /a/am/o/am.u vrhcplghcpot /a/am/o/am.u. vrhcplghcpot1 /a/am/o/am.u. vrhcplghcpot2

D. clstat and snmp

clstat and cldump rely on SNMP to be configured properly. If cldump fails with a message like this:

cdm:WiigfrteCutrSU pe (ltmrS lup atn o h lse MX er csrgE) t saiie...... o tblz....... Ual t cmuiaewt teCutrSU Pe Deo nbe o omnct ih h lse MX er amn

then / t / n p v . o fhas to be fixed by adding a line ecsmd3cn

VC_IWdfutiw AMVE ealVe

136141231215 ...........

-icue nldd

snmpd has to be restarted:

#sosc- smd tpr s np #satr - smd trsc s np

E. Related Information
AIX 6.1 Information Center > PowerHA SystemMirror > HACMP Version 6.1 IBM Redbook: PowerHA for AIX Cookbook Certification Study Guide: HACMP for AIX

Also on

unixwerk

HACMP: Cluster Commandline How to Add a New VG to an Active HACMP Resource Group How to Add a New Resource Group to an Active Cluster

www.unixwerk.eu/aix/setup_hacmp.html

18/19

1/21/13

unixwerk: setup a two-nodes cluster with hacmp

How to Add a Node to an HACMP Cluster How to Remove a Node from an HACMP Cluster

(c) unix we rk

C onte nt is available unde r the BSD Docum e ntation Lice nse .

www.unixwerk.eu/aix/setup_hacmp.html

19/19