Chapter 5. Examples and best practices
5.1. Testing the environment
Perform the following steps to check if everything is working as expected.
Procedure
Execute a takeover
Change the score of the master nodes to do a failover. In this example, the
SAPHanaclone resource isrsc_SAPHana_HDB_HDB00-clone, andsaphdb3is one node in the second availability zone:pcs constraint location rsc_SAPHana_HDB_HDB00-clone rule role=master score=100 \#uname eq saphdb3
This constraint should be removed again with:
pcs constraint remove rsc_SAPHana_HDB_HDB00Otherwise, pacemaker tries to start HANA on
SAPHDB1.Fence a node
You can fence a node with the command:
pcs stonith fence <nodename>Depending on the other fencing options and the infrastructure used, this node will stay down or come back.
killHANAYou can also
killthe database to check if the SAP resource agent is working. Assidadm, you can call:sidadm% HDB killPacemaker detects this issue and resolves it with a solution.
5.2. Useful aliases
5.2.1. Aliases for user root
These aliases are added to −/.bashrc:
export ListInstances=$(/usr/sap/hostctrl/exe/saphostctrl -function ListInstances| head -1 ) export sid=$(echo "$ListInstances" |cut -d " " -f 5| tr [A-Z] [a-z]) export SID=$(echo $sid | tr [a-z] [A-Z]) export Instance=$(echo "$ListInstances" |cut -d " " -f 7 ) alias crmm='watch -n 1 crm_mon -1Arf' alias crmv='watch -n 1 /usr/local/bin/crmmv' alias clean=/usr/local/bin/cleanup alias cglo='su - ${sid}adm -c cglo' alias cdh='cd /usr/lib/ocf/resource.d/heartbeat' alias vhdbinfo="vim /usr/sap/${SID}/home/hdbinfo;dcp /usr/sap/${SID}/home/hdbinfo" alias gtr='su - ${sid}adm -c gtr' alias hdb='su - ${sid}adm -c hdb' alias hdbi='su - ${sid}adm -c hdbi' alias hgrep='history | grep $1' alias hri='su - ${sid}adm -c hri' alias hris='su - ${sid}adm -c hris' alias killnode="echo 'b' > /proc/sysrq-trigger" alias lhc='su - ${sid}adm -c lhc' alias python='/usr/sap/${SID}/HDB${Instance}/exe/Python/bin/python' alias pss="watch 'pcs status --full | egrep -e Node\|master\|clone_state\|roles'" alias srstate='su - ${sid}adm -c srstate' alias shr='watch -n 5 "SAPHanaSR-monitor --sid=${SID}"' alias sgsi='su - ${sid}adm -c sgsi' alias spl='su - ${sid}adm -c spl' alias srs='su - ${sid}adm -c srs' alias sapstart='su - ${sid}adm -c sapstart' alias sapstop='su - ${sid}adm -c sapstop' alias sapmode='df -h /;su - ${sid}adm -c sapmode' alias smm='pcs property set maintenance-mode=true' alias usmm='pcs property set maintenance-mode=false' alias tma='tmux attach -t 0:' alias tmkill='tmux killw -a' alias tm='tail -100f /var/log/messages |grep -v systemd' alias tms='tail -1000f /var/log/messages | egrep -s\ "Setting master-rsc_SAPHana_${SID}_HDB${Instance}|sr_register\ *|WAITING4LPA\|EXCLUDE as posible takeover node|SAPHanaSR|failed|${HOSTNAME}\ |PROMOTED|DEMOTED|UNDEFINED|master_walk|SWAIT|WaitforStopped|FAILED"' alias tmss='tail -1000f /var/log/messages | grep -v systemd\ | egrep -s "secondary with sync status|Settingmaster-rsc_SAPHana_${SID}_HDB${Instance}\ |sr_register|WAITING4LPA|EXCLUDE as posible takeover node|SAPHanaSR\ |failed|${HOSTNAME}|PROMOTED|DEMOTED|UNDEFINED|master_walk|SWAIT|WaitforStopped|FAILED"' alias tmm='tail -1000f /var/log/messages | egrep -s \ "Settingmaster-rsc_SAPHana_${SID}_HDB${Instance}|sr_register\ |WAITING4LPA|PROMOTED|DEMOTED|UNDEFINED|master_walk|SWAIT|W aitforStopped\ |FAILED|LPT|SOK|SFAIL|SAPHanaSR-mon"| grep -v systemd' alias tmsl='tail -1000f /var/log/messages | egrep -s\ "Settingmaster-rsc_SAPHana_${SID}_HDB${Instance}|sr_register|WAITING4LPA\ |PROMOTED|DEMOTED|UNDEFINED|ERROR|Warning|mast er_walk|SWAIT\ |WaitforStopped|FAILED|LPT|SOK|SFAIL|SAPHanaSR-mon"' alias vih='vim /usr/lib/ocf/resource.d/heartbeat/SAPHanaStart' alias switch1='pcs constraint location rsc_SAPHana_HDB_HDB00-clone \ rule role=master score=100 \#uname eq saphdb1' alias switch3='pcs constraint location rsc_SAPHana_HDB_HDB00-clone \ rule role=master score=100 \#uname eq saphdb3' alias switch0='pcs constraint remove location-rsc_SAPHana_HDB_HDB00-clone alias switchl='pcs constraint location | grep pcs resource | grep promotable\ | awk "{ print $4 }"` | grep Constraint| awk "{ print $NF }"' alias scl='pcs constraint location |grep " Constraint"'
5.2.2. Aliases for the SIDadm user
These aliases are added to ~/.customer.sh:
alias tm='tail -100f /var/log/messages |grep -v systemd' alias tms='tail -1000f /var/log/messages | egrep -s \ "Settingmaster-rsc_SAPHana_$SAPSYSTEMNAME_HDB${TINSTANCE}|sr_register\ |WAITING4LPA|EXCLUDE as posible takeover node|SAPHanaSR|failed\ |${HOSTNAME}|PROMOTED|DEMOTED|UNDEFINED|master_walk|SWAIT|WaitforStopped|FAILED"' alias tmsl='tail -1000f /var/log/messages | egrep -s \ "Settingmaster-rsc_SAPHana_$SAPSYSTEMNAME_HDB${TINSTANCE}|sr_register\ |WAITING4LPA|PROMOTED|DEMOTED|UNDEFINED|master_walk|SWAIT|WaitforStopped|FAILED|LPT"' alias sapstart='sapcontrol -nr ${TINSTANCE} -function StartSystem HDB;hdbi' alias sapstop='sapcontrol -nr ${TINSTANCE} -function StopSystem HDB;hdbi' alias sapmode='watch -n 5 "hdbnsutil -sr_state --sapcontrol=1 |grep site.\*Mode"' alias sapprim='hdbnsutil -sr_stateConfiguration| grep -i primary' alias sgsi='watch sapcontrol -nr ${TINSTANCE} -function GetSystemInstanceList' alias spl='watch sapcontrol -nr ${TINSTANCE} -function GetProcessList' alias splh='watch "sapcontrol -nr ${TINSTANCE} -function GetProcessList\ | grep hdbdaemon"' alias srs="watch -n 5 'python \ /usr/sap/$SAPSYSTEMNAME/HDB${TINSTANCE}/exe/python_support/systemReplicationStatus.py * *; echo Status \$?'" alias cdb="cd /usr/sap/${SAPSYSTEMNAME}/HDB${TINSTANCE}/backup" alias srstate='watch -n 10 hdbnsutil -sr_state' alias hdb='watch -n 5 "sapcontrol -nr ${TINSTANCE} -function GetProcessList\ | egrep -s hdbdaemon\|hdbnameserver\|hdbindexserver "' alias hdbi='watch -n 5 "sapcontrol -nr ${TINSTANCE} -function GetProcessList\ | egrep -s hdbdaemon\|hdbnameserver\|hdbindexserver\ ;sapcontrol -nr ${TINSTANCE} -function GetSystemInstanceList "' alias hgrep='history | grep $1' alias vglo="vim /usr/sap/$SAPSYSTEMNAME/SYS/global/hdb/custom/config/global.ini" alias vgloh="vim /hana/shared/${SAPSYSTEMNAME}/HDB${TINSTANCE}/${HOSTNAME}/global.ini" alias hri='hdbcons -e hdbindexserver "replication info"' alias hris='hdbcons -e hdbindexserver "replication info" \ | egrep -e "SiteID|ReplicationStatus_"' alias gtr='watch -n 10 /usr/sap/$SAPSYSTEMNAME/HDB${TINSTANCE}/exe/Python/bin/python \ /usr/sap/$SAPSYSTEMNAME/HDB${TINSTANCE}/exe/python_support/getTakeoverRecommendation.py \ --sapcontrol=1' alias lhc='/usr/sap/$SAPSYSTEMNAME/HDB${TINSTANCE}/exe/Python/bin/python \ /usr/sap/$SAPSYSTEMNAME/HDB${TINSTANCE}/exe/python_support/landscapeHostConfiguration.py\ ;echo $?' alias reg1='hdbnsutil -sr_register --remoteHost=hana07 -remoteInstance=${TINSTANCE} \ --replicationMode=syncmem --name=DC3 --remoteName=DC1 \ --operationMode=logreplay --online' alias reg2='hdbnsutil -sr_register --remoteHost=hana08 -remoteInstance=${TINSTANCE} \ --replicationMode=syncmem --name=DC3 --remoteName=DC2 \ --operationMode=logreplay --online' alias reg3='hdbnsutil -sr_register --remoteHost=hana09 -remoteInstance=${TINSTANCE} \ --replicationMode=syncmem --name=DC3 --remoteName=DC3 --operationMode=logreplay \ --online' PS1="\[\033[m\][\[\e[1;33m\]\u\[\e[1;33m\]\[\033[m\]@\[\e[1;36m\]\h\[\033[m\]: \[\e[0m\]\[\e[1;32m\]\W\[\e[0m\]]# "
5.3. Monitoring failover example
There are many ways to force a takeover. This example forces a takeover without shutting off a node. The SAP resource agents work with scores to decide which node will promote the SAPHana clone resource. The current status is seen using this command:
[root@saphdb2:~]# alias pss='pcs status --full | egrep -e "Node|master|clone_state|roles"' [root@saphdb2:~]# pss Node List: Node Attributes: * Node: saphdb1 (1): * hana_hdb_clone_state : PROMOTED * hana_hdb_roles : master1:master:worker:master * master-rsc_SAPHana_HDB_HDB00 : 150 * Node: saphdb2 (2): * hana_hdb_clone_state : DEMOTED * hana_hdb_roles : slave:slave:worker:slave * master-rsc_SAPHana_HDB_HDB00 : -10000 * Node: saphdb3 (3): * hana_hdb_clone_state : DEMOTED * hana_hdb_roles : master1:master:worker:master * master-rsc_SAPHana_HDB_HDB00 : 100 * Node: saphdb4 (4): * hana_hdb_clone_state : DEMOTED * hana_hdb_roles : slave:slave:worker:slave * master-rsc_SAPHana_HDB_HDB00 : -12200
In this example, the SAPHana clone resource is promoted on saphdb1. So the primary database runs on saphdb1. The score of this node is 150 and you can adjust the score of the secondary saphdb3 to force pacemaker to takeover the database to the secondary.
