Chapter 8. Maintenance procedures

Procedure

1. Clean up resource failures:

   [root]# pcs resource cleanup

2. Clean up the STONITH failure history:

   [root]# pcs stonith history cleanup

Procedure

1. Set maintenance mode for the entire cluster:

   [root]# pcs property set maintenance-mode=true

   Setting maintenance for the whole cluster ensures that no activity during the maintenance phase can trigger cluster actions and impact the HANA update process.

2. Verify that the cluster resource management is fully disabled:

   [root]# pcs status
   ...
   
                 *** Resource management is DISABLED ***
     The cluster will not attempt to start, stop or recover services
   
   Node List:
     * Online: [ node1 node2 ]
   
   Full List of Resources:
     * Clone Set: cln_SAPHanaTop_RH1_HDB02 [rsc_SAPHanaTop_RH1_HDB02] (maintenance):
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started node2 (maintenance)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started node1 (maintenance)
     * Clone Set: cln_SAPHanaCon_RH1_HDB02 [rsc_SAPHanaCon_RH1_HDB02] (promotable, maintenance):
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted node2 (maintenance)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Promoted node1 (maintenance)
     * Clone Set: cln_SAPHanaFil_RH1_HDB02 [rsc_SAPHanaFil_RH1_HDB02] (maintenance):
       * rsc_SAPHanaFil_RH1_HDB02  (ocf:heartbeat:SAPHanaFilesystem):       Started node2 (maintenance)
       * rsc_SAPHanaFil_RH1_HDB02  (ocf:heartbeat:SAPHanaFilesystem):       Started node1 (maintenance)
     * rsc_vip_RH1_HDB02_primary   (ocf:heartbeat:IPaddr2):         Started node1 (maintenance)
     * rsc_vip_RH1_HDB02_readonly  (ocf:heartbeat:IPaddr2):         Started node2 (maintenance)
   
   ...

3. Update the HANA instances using the SAP procedure. If you have to perform a takeover during the HANA update, you can use the SAP HANA Takeover with Handshake option. For more information see also Is it possible to use SAP HANA "Takeover with Handshake" option with the HA Solutions for managing HANA System Replication?.

   If you stop the cluster in this step, ensure that you start it again before you proceed with the next steps. Keep the maintenance mode enabled.

4. After the HANA update, verify that the HANA system replication is working correctly. Use the systemReplicationStatus.py script to show the status of the HANA system replication on the primary instance. Below is the example of post manual takeover to node2, during the maintenance:

   [root]# su - <sid>adm
   
   rh1adm $ cdpy; python systemReplicationStatus.py --sapcontrol=1 | grep -i replication_status=
   service/node2/30201/REPLICATION_STATUS=ACTIVE
   service/node2/30207/REPLICATION_STATUS=ACTIVE
   service/node2/30203/REPLICATION_STATUS=ACTIVE
   site/1/REPLICATION_STATUS=ACTIVE
   overall_replication_status=ACTIVE

   Before you proceed, ensure that the system replication is healthy and reported as ACTIVE.

5. Refresh all cluster resources to execute one monitor operation and update their status:

   [root]# pcs resource refresh
   Waiting for 1 reply from the controller
   ... got reply (done)

   It is crucial that the HANA resources update cluster and node attributes, to reflect the new HANA system replication status. It ensures that the cluster has the correct information and does not trigger recovery actions due to incorrect status information, after the maintenance stops.

6. Check the cluster status and verify the resource status and main HANA resource score attribute. All resources must show as Started and the promotable resources must show as Unpromoted on all nodes:

   [root]# pcs status
   ...
   Full List of Resources:
     * Clone Set: cln_SAPHanaTop_RH1_HDB02 [rsc_SAPHanaTop_RH1_HDB02] (maintenance):
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started node2 (maintenance)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started node1 (maintenance)
     * Clone Set: cln_SAPHanaCon_RH1_HDB02 [rsc_SAPHanaCon_RH1_HDB02] (promotable, maintenance):
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted node2 (maintenance)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted node1 (maintenance)
     * Clone Set: cln_SAPHanaFil_RH1_HDB02 [rsc_SAPHanaFil_RH1_HDB02] (maintenance):
       * rsc_SAPHanaFil_RH1_HDB02  (ocf:heartbeat:SAPHanaFilesystem):       Started node2 (maintenance)
       * rsc_SAPHanaFil_RH1_HDB02  (ocf:heartbeat:SAPHanaFilesystem):       Started node1 (maintenance)
     * rsc_vip_RH1_HDB02_primary   (ocf:heartbeat:IPaddr2):         Started node1 (maintenance)
     * rsc_vip_RH1_HDB02_readonly  (ocf:heartbeat:IPaddr2):         Started node2 (maintenance)
   
   ...

7. Check the cluster attributes and verify that srHook, roles and score attributes are in the correct new state:

   [root]# SAPHanaSR-showAttr
   ...
   
   Site lpt        lss mns   opMode    srHook srMode srPoll srr
   -------------------------------------------------------------
   DC2  1746793335 4   node2 logreplay PRIM   sync   SOK    S
   DC1  30         4   node1 logreplay SOK    sync   PRIM   P
   
   Host  clone_state roles                        score site sra srah version     vhost
   -------------------------------------------------------------------------------------
   node1 DEMOTED     master1:master:worker:master 100   DC1  -   -    2.00.078.00 node1
   node2 DEMOTED     master1:master:worker:master 150   DC2  -   -    2.00.078.00 node2

   • The srHook is PRIM on the node that is running the primary instance and it shows SOK on the correct secondary.
   • The score is 150 for the node where the primary is running and is 100 on the other.

8. When the checks of steps 6 and 7 show the landscape in the expected healthy state, you can remove the maintenance mode of the cluster again:

   [root]# pcs property set maintenance-mode=

   When you lift the maintenance it triggers a monitor run of all resources again. The cluster updates the status of the promotable resources to Promoted and Unpromoted in correspondence to the location of the primary and secondary instances. The resources now also update the srPoll attribute again to match the srHook attribute value.

Procedure

1. Register the former primary as the new secondary. Run as user <sid>adm on the stopped former primary instance:

   rh1adm $ hdbnsutil -sr_register --remoteHost=<node> \
   --remoteInstance=${TINSTANCE} --replicationMode=sync \
   --operationMode=logreplay --name=<DC>

   • Replace <node> with the new primary instance host, for example node2 if there was a takeover from node1 to node2.
   • Replace <DC> with your new secondary HANA site name, for example DC1 if node1 is to be registered as a secondary.
   • Choose the values for replicationMode and operationMode according to your requirements for the system replication.
   • TINSTANCE is an environment variable that is set automatically for user <sid>adm by reading the HANA instance profile. The variable value is the HANA instance number.

2. Start the secondary HANA instance. Run as <sid>adm on the new secondary instance node:

   rh1adm $ HDB start