Chapter 5. Configuring the Pacemaker cluster

Procedure

1. Create the directory /etc/systemd/system/pacemaker.service.d/ for the pacemaker service drop-in file:

   [root]# mkdir /etc/systemd/system/pacemaker.service.d/

2. Create the systemd drop-in file for the pacemaker service with the following content:

   [root]# cat << EOF > /etc/systemd/system/pacemaker.service.d/00-pacemaker.conf
   [Unit]
   Description=Pacemaker needs the SAP HANA instance service
   Wants=SAP<SID>_<instance>.service
   After=SAP<SID>_<instance>.service
   EOF

   • Replace <SID> with your HANA SID.
   • Replace <instance> with your HANA instance number.

3. Reload the systemctl daemon to activate the drop-in file:

   [root]# systemctl daemon-reload

4. Repeat steps 1-3 on the other cluster nodes.

Verification

1. Check the systemd service of your HANA instance and verify that it is loaded:

   [root]# systemctl status SAPRH1_02.service
   ● SAPRH1_02.service - SAP Instance SAPRH1_02
        Loaded: loaded (/etc/systemd/system/SAPRH1_02.service; disabled; preset: disabled)
        Active: active (running) since xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      Main PID: 5825 (sapstartsrv)
         Tasks: 841
        Memory: 88.6G
           CPU: 4h 50min 2.033s
        CGroup: /SAP.slice/SAPRH1_02.service
                ├─ 5825 /usr/sap/RH1/HDB02/exe/sapstartsrv pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_node1
                ├─ 5986 sapstart pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_node1
                ├─ 5993 /usr/sap/RH1/HDB02/node1/trace/hdb.sapRH1_HDB02 -d -nw -f /usr/sap/RH1/HDB02/node1/daemon.ini pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_node1
   ...

2. Verify that the SAP HANA instance service is known to the pacemaker service now:

   [root]# systemctl show pacemaker.service | grep SAP
   Wants=SAPRH1_02.service resource-agents-deps.target dbus-broker.service
   After=... SAPRH1_02.service rsyslog.service

   Make sure that the SAP<SID>_<instance>.service is listed in the After= and Wants= lists.

Procedure

1. Stop the HANA instances on all nodes as the <sid>adm user:

   rh1adm $ HDB stop

2. Verify as <sid>adm on all nodes that the HANA instances are stopped completely:

   rh1adm $ sapcontrol -nr ${TINSTANCE} -function GetProcessList | column -s ',' -t
   ...
   name                  description   dispstatus   textstatus   starttime   elapsedtime   pid
   hdbdaemon             HDB Daemon    GRAY         Stopped                                5381

3. Change to the HANA configuration directory, as the <sid>adm user, using the command alias cdcoc, which is built into the <sid>adm user shell. This automatically changes into the /hana/shared/<SID>/global/hdb/custom/config/ path:

   rh1adm $ cdcoc

4. Update the global.ini file of the SAP HANA instance to configure the HanaSR hook. Edit the configuration file on all HANA instance nodes and add the following configuration:

   [ha_dr_provider_hanasr]
   provider = HanaSR
   path = /usr/share/sap-hana-ha/
   execution_order = 1
   
   [trace]
   ha_dr_hanasr = info

   Set execution_order to 1 to ensure that the HanaSR hook is always executed with the highest priority.

5. Optional: When you also want to configure the optional ChkSrv hook for taking action on an hdbindexserver failure, you can add the changes to the global.ini at the same time, see step 1 in Configuring the ChkSrv HA/DR provider for the srServiceStateChanged() hook method:

   [ha_dr_provider_hanasr]
   provider = HanaSR
   path = /usr/share/sap-hana-ha/
   execution_order = 1
   
   [ha_dr_provider_chksrv]
   provider = ChkSrv
   path = /usr/share/sap-hana-ha/
   execution_order = 2
   action_on_lost = stop
   
   [trace]
   ha_dr_hanasr = info
   ha_dr_chksrv = info

6. Create the file /etc/sudoers.d/20-saphana, as the root user, on each cluster node with the following content. These command privileges allow the <sid>adm user to update certain cluster node attributes as part of the HanaSR hook execution:

   [root]# visudo -f /etc/sudoers.d/20-saphana
   <sid>adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_*
   Defaults:<sid>adm !requiretty

   • For further information on why the Defaults setting is needed, refer to The srHook attribute is set to SFAIL in a Pacemaker cluster managing SAP HANA system replication, even though replication is in a healthy state.

7. Start the HANA instances on both cluster nodes manually without starting the HA cluster. Run as <sid>adm:

   rh1adm $ HDB start

Verification

1. Change to the SAP HANA directory, as the <sid>adm user, where trace log files are stored. Use the command alias cdtrace, which is built into the <sid>adm user shell:

   rh1adm $ cdtrace

2. Check the HANA nameserver service logs for the HA/DR provider loading message:

   1. If only the HanaSR provider is configured:

      rh1adm $ grep -he "loading HA/DR Provider.*" nameserver_*
      loading HA/DR Provider 'HanaSR' from /usr/share/sap-hana-ha/

   2. If the optional ChkSrv provider is also implemented:

      rh1adm $ grep -he "loading HA/DR Provider.*" nameserver_*
      loading HA/DR Provider 'ChkSrv' from /usr/share/sap-hana-ha/
      loading HA/DR Provider 'HanaSR' from /usr/share/sap-hana-ha/

3. Verify as user root in the system secure log that the sudo command executed successfully. If the sudoers file is not correct, an error is logged when the sudo command is executed:

   [root]# grep -e 'sudo.*crm_attribute.*' /var/log/secure
    sudo[4267]:  rh1adm : PWD=/hana/shared/RH1/HDB02/node1 ; USER=root ; COMMAND=/usr/sbin/crm_attribute -n hana_rh1_site_srHook_DC2 -v SFAIL -t crm_config -s SAPHanaSR
    sudo[4319]:  rh1adm : PWD=/hana/shared/RH1/HDB02/node1 ; USER=root ; COMMAND=/usr/sbin/crm_attribute -n hana_rh1_site_srHook_DC2 -v SOK -t crm_config -s SAPHanaSR

   After the HANA instance starts on both nodes, you can usually see several srHook attribute updates. At first it is setting SFAIL, because immediately after the primary starts, it is not yet in sync with the secondary, which is still synchronizing at this time.

   The last update to SOK is triggered by the HANA event after the system replication status finally changes to fully in sync.

4. Repeat the verification steps 1-2 on the second instance, if not already done at the same time. The sudo log messages of step 3 are only visible on the primary instance node where the system replication events are sent.

5. Check the cluster attributes on any node and verify that the value for the hana_<sid>_site_srHook_<DC2> attribute is updated as expected:

   [root]# cibadmin --query | grep -e 'SAPHanaSR.*srHook'
           <nvpair id="SAPHanaSR-hana_rh1_site_srHook_DC2" name="hana_rh1_site_srHook_DC2" value="SOK"/>

   • SOK is set when the HANA system replication is in ACTIVE state, which means established and fully in sync.
   • SFAIL is set when the system replication is in any other state.

Procedure

1. Change to the HANA configuration directory as the <sid>adm user. Use the command alias cdcoc, which is built into the <sid>adm user shell. This automatically changes into the /hana/shared/<SID>/global/hdb/custom/config/ path:

   rh1adm $ cdcoc

2. Update the global.ini file of the HANA instance to configure the hook script. Edit the configuration file on all HANA instance systems and add the following content in addition to the HanaSR provider definition:

   [ha_dr_provider_chksrv]
   provider = ChkSrv
   path = /usr/share/sap-hana-ha/
   execution_order = 2
   action_on_lost = stop
   
   [trace]
   ha_dr_hanasr = info
   ha_dr_chksrv = info

3. Optional: If you want to use the fence option for action_on_lost, you must add the SAPHanaSR-hookHelper to the sudo configuration of the <sid>adm user:

   [root]# visudo -f /etc/sudoers.d/20-saphana
   …
   <sid>adm ALL=(ALL) NOPASSWD: /usr/bin/SAPHanaSR-hookHelper

4. Optional: Activate the ChkSrv provider while HANA is running by reloading the HA/DR providers. Skip this step when configuring the hook script while the instance is down, the HA/DR provider is loaded automatically at the next instance start.

   rh1adm $ hdbnsutil -reloadHADRProviders

Verification

1. Change to the SAP HANA directory, as the <sid>adm user, where trace log files are stored. Use the command alias cdtrace, which is built into the <sid>adm user shell:

   rh1adm $ cdtrace

2. Check that the changes are loaded:

   rh1adm $ grep -e "loading HA/DR Provider.*ChkSrv.*" nameserver_*
   loading HA/DR Provider 'ChkSrv' from /usr/share/sap-hana-ha/

3. Check that the dedicated trace log file is created and the provider loaded with the correct configuration parameters:

   rh1adm $ cat nameserver_chksrv.trc
   init called
   ChkSrv.init() version 1.001.1, parameter info: action_on_lost=stop stop_timeout=20 kill_signal=9

Procedure

1. Create the SAPHanaTopology resource as a clone resource, which means it runs on all cluster nodes at the same time:

   [root]# pcs resource create rsc_SAPHanaTop_<SID>_HDB<instance> \
   ocf:heartbeat:SAPHanaTopology \
   SID=<SID> \
   InstanceNumber=<instance> \
   op start timeout=600 \
   op stop timeout=300 \
   op monitor interval=30 timeout=300 \
   clone cln_SAPHanaTop_<SID>_HDB<instance> \
   meta clone-max=2 clone-node-max=1 interleave=true --future

   • Replace <SID> with your HANA SID.

   • Replace <instance> with your HANA instance number.

     Note

     Since RHEL 9.3, a deprecation warning is displayed when not using the meta keyword in the clone command and the attributes are automatically assigned to the base resource.

     In the future, the meta keyword for clone attributes will be required for assigning attributes to the clone resource. Until then, add the --future parameter to already apply this behavior now.

     See also New pcs parsing requires meta keyword when specifying clone meta attributes.

2. Create the SAPHanaController resource as a promotable clone resource. This means it runs on all cluster nodes at the same time, but on one node it functions as the active, or primary, instance:

   [root]# pcs resource create rsc_SAPHanaCon_<SID>_HDB<instance> \
   ocf:heartbeat:SAPHanaController \
   SID=<SID> \
   InstanceNumber=<instance> \
   PREFER_SITE_TAKEOVER=true \
   DUPLICATE_PRIMARY_TIMEOUT=7200 \
   AUTOMATED_REGISTER=false \
   op stop timeout=3600 \
   op monitor interval=59 role=Promoted timeout=700 \
   op monitor interval=61 role=Unpromoted timeout=700 \
   meta priority=100 \
   promotable cln_SAPHanaCon_<SID>_HDB<instance> \
   meta clone-max=2 clone-node-max=1 interleave=true --future

   We recommend creating the resource with AUTOMATED_REGISTER=false and then verify the correct behavior and data consistency through tests to complete the setup. For more information, see Testing the setup. You can enable this already at creation time by setting the parameter to true.

   See SAPHanaController resource parameters for more details.

3. Optional: If you want the cluster to fence the node when the SAPHanaController resource fails, then update the resource with the ON_FAIL_ACTION parameter and set it to fence:

   [root]# pcs resource update rsc_SAPHanaCon_<SID>_HDB<instance> ON_FAIL_ACTION=fence

4. You must start SAPHanaTopology before SAPHanaController, because it collects HANA landscape information, which the SAPHanaController resource requires to start correctly. Create the cluster constraint that enforces the correct start order of the two resources:

   [root]# pcs constraint order cln_SAPHanaTop_<SID>_HDB<instance> \
   then cln_SAPHanaCon_<SID>_HDB<instance> symmetrical=false

   Setting symmetrical=false indicates that the constraint only influences the start order of the resources, but it does not apply to the stop order.

5. Optional: Create the SAPHanaFilesystem resource as a clone resource:

   [root]# pcs resource create rsc_SAPHanaFil_<SID>_HDB<instance> \
   ocf:heartbeat:SAPHanaFilesystem \
   SID=<SID> \
   InstanceNumber=<instance> \
   ON_FAIL_ACTION="fence" \
   op start interval=0 timeout=10 \
   op stop interval=0 timeout=20 \
   op monitor interval=120 timeout=120 \
   clone cln_SAPHanaFil_<SID>_HDB<instance> \
   meta clone-node-max=1 interleave=true --future

   Instead of setting ON_FAIL_ACTION=fence you can set it to ignore. This can be useful for testing the functionality first. The resource writes information to the system logs, which can be examined for evaluating if the resource would take the wanted action when activated to use the fence action.

Verification

1. Review the SAPHanaTopology resource clone. Example resource configuration:

   [root]# pcs resource config cln_SAPHanaTop_RH1_HDB02
   Clone: cln_SAPHanaTop_RH1_HDB02
     Meta Attributes: cln_SAPHanaTop_RH1_HDB02-meta_attributes
       clone-max=2
       clone-node-max=1
       interleave=true
     Resource: rsc_SAPHanaTop_RH1_HDB02 (class=ocf provider=heartbeat type=SAPHanaTopology)
       Attributes: rsc_SAPHanaTop_RH1_HDB02-instance_attributes
         InstanceNumber=02
         SID=RH1
       Operations:
         methods: rsc_SAPHanaTop_RH1_HDB02-methods-interval-0s
           interval=0s timeout=5
         monitor: rsc_SAPHanaTop_RH1_HDB02-monitor-interval-30
           interval=30 timeout=300
         reload: rsc_SAPHanaTop_RH1_HDB02-reload-interval-0s
           interval=0s timeout=5
         start: rsc_SAPHanaTop_RH1_HDB02-start-interval-0s
           interval=0s timeout=600
         stop: rsc_SAPHanaTop_RH1_HDB02-stop-interval-0s
           interval=0s timeout=300

2. Review the SAPHanaController resource clone. Example resource configuration:

   [root]# pcs resource config cln_SAPHanaCon_RH1_HDB02
   Clone: cln_SAPHanaCon_RH1_HDB02
     Meta Attributes: cln_SAPHanaCon_RH1_HDB02-meta_attributes
       clone-max=2
       clone-node-max=1
       interleave=true
       promotable=true
     Resource: rsc_SAPHanaCon_RH1_HDB02 (class=ocf provider=heartbeat type=SAPHanaController)
       Attributes: rsc_SAPHanaCon_RH1_HDB02-instance_attributes
         AUTOMATED_REGISTER=false
         DUPLICATE_PRIMARY_TIMEOUT=7200
         InstanceNumber=02
         PREFER_SITE_TAKEOVER=true
         SID=RH1
       Meta Attributes: rsc_SAPHanaCon_RH1_HDB02-meta_attributes
         priority=100
       Operations:
         demote: rsc_SAPHanaCon_RH1_HDB02-demote-interval-0s
           interval=0s timeout=320
         methods: rsc_SAPHanaCon_RH1_HDB02-methods-interval-0s
           interval=0s timeout=5
         monitor: rsc_SAPHanaCon_RH1_HDB02-monitor-interval-59
           interval=59 timeout=700 role=Promoted
         monitor: rsc_SAPHanaCon_RH1_HDB02-monitor-interval-61
           interval=61 timeout=700 role=Unpromoted
         promote: rsc_SAPHanaCon_RH1_HDB02-promote-interval-0s
           interval=0s timeout=900
         reload: rsc_SAPHanaCon_RH1_HDB02-reload-interval-0s
           interval=0s timeout=5
         start: rsc_SAPHanaCon_RH1_HDB02-start-interval-0s
           interval=0s timeout=3600
         stop: rsc_SAPHanaCon_RH1_HDB02-stop-interval-0s
           interval=0s timeout=3600

3. Check that the start order constraint is in place:

   [root]# pcs constraint order
   Order Constraints:
     start resource 'cln_SAPHanaTop_RH1_HDB02' then start resource 'cln_SAPHanaCon_RH1_HDB02'
       symmetrical=0

4. Optional: Review the SAPHanaFilesystem resource clone if you created it. Example resource configuration:

   [root]# pcs resource config cln_SAPHanaFil_RH1_HDB02
   Clone: cln_SAPHanaFil_RH1_HDB02
     Meta Attributes: cln_SAPHanaFil_RH1_HDB02-meta_attributes
       clone-node-max=1
       interleave=true
     Resource: rsc_SAPHanaFil_RH1_HDB02 (class=ocf provider=heartbeat type=SAPHanaFilesystem)
       Attributes: rsc_SAPHanaFil_RH1_HDB02-instance_attributes
         InstanceNumber=02
         ON_FAIL_ACTION=fence
         SID=RH1
       Operations:
         methods: rsc_SAPHanaFil_RH1_HDB02-methods-interval-0s
           interval=0s timeout=5
         monitor: rsc_SAPHanaFil_RH1_HDB02-monitor-interval-120
           interval=120 timeout=120
         reload: rsc_SAPHanaFil_RH1_HDB02-reload-interval-0s
           interval=0s timeout=5
         start: rsc_SAPHanaFil_RH1_HDB02-start-interval-0
           interval=0 timeout=10
         stop: rsc_SAPHanaFil_RH1_HDB02-stop-interval-0
           interval=0 timeout=20

5. Check the cluster status. Use --full to include node attributes, which are updated by the HANA resources:

   [root]# pcs status --full
   ...
   
   Full List of Resources:
     * Clone Set: cln_SAPHanaTop_RH1_HDB02 [rsc_SAPHanaTop_RH1_HDB02]:
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started node1
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started node2
     * Clone Set: cln_SAPHanaCon_RH1_HDB02 [rsc_SAPHanaCon_RH1_HDB02] (promotable):
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Promoted node1
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted node2
     * Clone Set: cln_SAPHanaFil_RH1_HDB02 [rsc_SAPHanaFil_RH1_HDB02]:
       * rsc_SAPHanaFil_RH1_HDB02  (ocf:heartbeat:SAPHanaFilesystem):       Started node1
       * rsc_SAPHanaFil_RH1_HDB02  (ocf:heartbeat:SAPHanaFilesystem):       Started node2
   
   Node Attributes:
     * Node: node1 (1):
       * hana_rh1_clone_state              : PROMOTED
       * hana_rh1_roles                    : master1:master:worker:master
       * hana_rh1_site                     : DC1
       * hana_rh1_srah                     : -
       * hana_rh1_version                  : 2.00.078.00
       * hana_rh1_vhost                    : node1
       * master-rsc_SAPHanaCon_RH1_HDB02   : 150
     * Node: node2 (2):
       * hana_rh1_clone_state              : DEMOTED
       * hana_rh1_roles                    : master1:master:worker:master
       * hana_rh1_site                     : DC2
       * hana_rh1_srah                     : -
       * hana_rh1_version                  : 2.00.078.00
       * hana_rh1_vhost                    : node2
       * master-rsc_SAPHanaCon_RH1_HDB02   : 100
   
   ...

Procedure

1. Use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running. Adjust the parameters according to the resource agent you are using. Create the cluster resource for the primary virtual IP, for example using the IPaddr2 agent:

   [root]# pcs resource create rsc_vip_<SID>_HDB<instance>_primary \
   ocf:heartbeat:IPaddr2 ip=<address> cidr_netmask=<netmask> nic=<device>

   • Replace <SID> with your HANA SID.
   • Replace <instance> with your HANA instance number.
   • Replace <address>, <netmask> and <device> with the details of your primary virtual IP address.

2. Create a cluster constraint that places the VIP resource with the SAPHanaController resource on the HANA primary node:

   [root]# pcs constraint colocation add rsc_vip_<SID>_HDB<instance>_primary \
   with promoted cln_SAPHanaCon_<SID>_HDB<instance> 2000

   The constraint applies a score of 2000 instead of the default INFINITY. This softens the resource dependency and allows the virtual IP resource to stay active in the case when there is no promoted SAPHanaController resource. This way it is still possible to use tools like the SAP Management Console (MMC) or SAP Landscape Management (LaMa) that can reach this IP address to query the status information of the HANA instance.

Verification

1. Check the resource configuration of the virtual IP resource:

   [root]# pcs resource config rsc_vip_RH1_HDB02_primary
   Resource: rsc_vip_RH1_HDB02_primary (class=ocf provider=heartbeat type=IPaddr2)
     Attributes: rsc_vip_RH1_HDB02_primary-instance_attributes
       cidr_netmask=32
       ip=192.168.1.100
       nic=eth0
     Operations:
       monitor: rsc_vip_RH1_HDB02_primary-monitor-interval-10s
         interval=10s timeout=20s
       start: rsc_vip_RH1_HDB02_primary-start-interval-0s
         interval=0s timeout=20s
       stop: rsc_vip_RH1_HDB02_primary-stop-interval-0s
         interval=0s timeout=20s

2. Check that the constraint is defined correctly:

   [root]# pcs constraint colocation
   Colocation Constraints:
     Started resource 'rsc_vip_RH1_HDB02_primary' with Promoted resource 'cln_SAPHanaCon_RH1_HDB02'
       score=2000

Procedure

1. Use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running. Adjust the parameters according to the resource agent you are using. Create the cluster resource for the secondary virtual IP, for example using the IPaddr2 agent:

   [root]# pcs resource create rsc_vip_<SID>_HDB<instance>_readonly \
   ocf:heartbeat:IPaddr2 ip=<address> cidr_netmask=<netmask> nic=<device>

   • Replace <SID> with your HANA SID.
   • Replace <instance> with your HANA instance number.
   • Replace <address>, <netmask> and <device> with the details of your read-only secondary virtual IP address.

2. Create a location constraint rule to ensure that the secondary virtual IP is assigned to the secondary instance during normal operations:

   [root]# pcs constraint location rsc_vip_<SID>_HDB<instance>_readonly \
   rule score=INFINITY master-rsc_SAPHanaCon_<SID>_HDB<instance> eq 100 \
   and hana_<sid>_clone_state eq DEMOTED

   • Replace <SID> with your HANA SID.
   • Replace <sid> with the lower-case HANA SID.
   • Replace <instance> with your HANA instance number.

3. Create a location constraint rule to ensure that the secondary virtual IP runs on the primary instance as an alternative whenever necessary:

   [root]# pcs constraint location rsc_vip_<SID>_HDB<instance>_readonly \
   rule score=2000 master-rsc_SAPHanaCon_<SID>_HDB<instance> eq 150 \
   and hana_<sid>_clone_state eq PROMOTED