Chapter 3. Configuring the HA cluster to manage the SAP HANA Scale-Up System Replication setup

The resource agents and other SAP HANA specific components required for setting up an HA cluster for managing SAP HANA Scale-Up System Replication setup are provided via the resource-agents-sap-hana RPM package from the “RHEL for SAP Solutions” repo.

To install the package please use the following command:

[root]# dnf install resource-agents-sap-hana

As documented in SAP’s Implementing a HA/DR Provider, recent versions of SAP HANA provide so-called "hooks" that allow SAP HANA to send out notifications for certain events. The srConnectionChanged() hook can be used to improve the ability of the HA cluster to detect when a change in the status of the SAP HANA System Replication occurs that requires the HA cluster to take action and to avoid data loss/data corruption by preventing accidental takeovers from being triggered in situations where this should be avoided.

To avoid unnecessary failovers of the resources, the following default values for the resource-stickiness and migration-threshold parameters must be set (this only needs to be done on one node):

[root]# pcs resource defaults update resource-stickiness=1000
[root]# pcs resource defaults update migration-threshold=5000

[root]# pcs resource defaults update resource-stickiness=1000
[root]# pcs resource defaults update migration-threshold=5000

resource-stickiness=1000 will encourage the resource to stay running where it is, while migration-threshold=5000 will cause the resource to move to a new node only after 5000 failures. 5000 is generally sufficient to prevent the resource from prematurely failing over to another node. This also ensures that the resource failover time stays within a controllable limit.

The SAPHanaTopology resource agent gathers information about the status and configuration of SAP HANA System Replication on each node. In addition, it starts and monitors the local SAP HostAgent, which is required for starting, stopping, and monitoring the SAP HANA instances.

The SAPHanaTopology resource agent has the following attributes:

Below is an example command to create the SAPHanaTopology cloned resource.

[root]# pcs resource create SAPHanaTopology_RH1_02 SAPHanaTopology SID=RH1 InstanceNumber=02 \
op start timeout=600 \
op stop timeout=300 \
op monitor interval=10 timeout=600 \
clone clone-max=2 clone-node-max=1 interleave=true

The resulting resource should look like the following:

[root]# pcs resource config SAPHanaTopology_RH1_02-clone
Clone: SAPHanaTopology_RH1_02-clone
Meta Attrs: clone-max=2 clone-node-max=1 interleave=true Resource: SAPHanaTopology_RH1_02 (class=ocf provider=heartbeat type=SAPHanaTopology)
  Attributes: SID=RH1 InstanceNumber=02
  Operations: start interval=0s timeout=600 (SAPHanaTopology_RH1_02-start-interval-0s)
              stop interval=0s timeout=300 (SAPHanaTopology_RH1_02-stop-interval-0s)
              monitor interval=10 timeout=600 (SAPHanaTopology_RH1_02-monitor-interval-10s)

Once the resource is started, you will see the collected information stored in the form of node attributes that can be viewed with the command pcs status --full. Below is an example of what attributes can look like when only SAPHanaTopology is started.

[root]# pcs status --full
...
Node Attributes:
* Node node1:
    + hana_rh1_remoteHost               : node2
    + hana_rh1_roles                    : 1:P:master1::worker:
    + hana_rh1_site                     : DC1
    + hana_rh1_srmode                   : syncmem
    + hana_rh1_vhost                    : node1
* Node node2:
    + hana_rh1_remoteHost               : node1
    + hana_rh1_roles                    : 1:S:master1::worker:
    + hana_rh1_site                     : DC2
    + hana_rh1_srmode                   : syncmem
    + hana_rh1_vhost                    : node2
...

The SAPHana resource agent manages the SAP HANA instances that are part of the SAP HANA Scale-Up System Replication and also monitors the status of SAP HANA System Replication. In the event of a failure of the SAP HANA primary replication instance, the SAPHana resource agent can trigger a takeover of SAP HANA System Replication based on how the resource agent parameters have been set.

The SAPHana resource agent has the following attributes:

The PREFER_SITE_TAKEOVER, AUTOMATED_REGISTER and DUPLICATE_PIMARY_TIMEOUT parameters must be set according to the requirements for availability and data protection of the SAP HANA System Replication that is managed by the HA cluster.

In general, PREFER_SITE_TAKEOVER should be set to true, to allow the HA cluster to trigger a takeover in case a failure of the primary SAP HANA instance has been detected, since it usually takes less time for the new SAP HANA primary instance to become fully active than it would take for the original SAP HANA primary instance to restart and reload all data back from disk into memory.

To be able to verify that all data on the new primary SAP HANA instance is correct after a takeover triggered by the HA cluster has occurred, AUTOMATED_REGISTER should be set to false. This will give an operator the possibility to either switch back to the old primary SAP HANA instance in case a takeover happened by accident, or if the takeover was correct, the old primary SAP HANA instance can be registered as the new secondary SAP HANA instance to get SAP HANA System Replication working again.

If AUTOMATED_REGISTER is set to true, then an old primary SAP HANA instance will be automatically registered as the new secondary SAP HANA instance by the SAPHana resource agent after a takeover by the HA cluster has occurred. This will increase the availability of the SAP HANA System Replication setup and prevent so-called “dual-primary” situations in the SAP HANA System Replication environment. But it can potentially increase the risk of data-loss/data-corruption, because if a takeover was triggered by the HA cluster even though the data on the secondary SAP HANA instance wasn’t fully in sync, then the automatic registration of the old primary SAP HANA instance as the new secondary SAP HANA instance would result in all data on this instance being deleted and therefore any data that has not been synced before the takeover occurred won’t be available anymore.

The promotable SAPHana cluster resource for managing the SAP HANA instances and SAP HANA System Replication can be created as in the following example:

[root]# pcs resource create SAPHana_RH1_02 SAPHana SID=RH1 InstanceNumber=02 \ PREFER_SITE_TAKEOVER=true DUPLICATE_PRIMARY_TIMEOUT=7200 AUTOMATED_REGISTER=true \
op start timeout=3600 \
op stop timeout=3600 \
op monitor interval=61 role="Slave" timeout=700 \
op monitor interval=59 role="Master" timeout=700 \
op promote timeout=3600 \
op demote timeout=3600 \
promotable notify=true clone-max=2 clone-node-max=1 interleave=true

The resulting HA cluster resource should look like the following:

[root]# pcs resource config SAPHana_RH1_02-clone
Clone: SAPHana_RH1_02-clone
Meta Attrs: clone-max=2 clone-node-max=1 interleave=true notify=true promotable=true
Resource: SAPHana_RH1_02 (class=ocf provider=heartbeat type=SAPHana)
  Attributes: AUTOMATED_REGISTER=true DUPLICATE_PRIMARY_TIMEOUT=180 InstanceNumber=02 PREFER_SITE_TAKEOVER=true SID=RH1
  Operations: methods interval=0s timeout=5 (SAPHana_RH1_02-methods-interval-0s)
              monitor interval=61 role=Slave timeout=700 (SAPHana_RH1_02-monitor-interval-61)
              monitor interval=59 role=Master timeout=700 (SAPHana_RH1_02-monitor-interval-59)
              promote interval=0s timeout=3600 (SAPHana_RH1_02-promote-interval-0s)
              demote interval=0s timeout=3600 (SAPHana_RH1_02-demote-interval-0s)
              start interval=0s timeout=3600 (SAPHana_RH1_02-start-interval-0s)
              stop interval=0s timeout=3600 (SAPHana_RH1_02-stop-interval-0s)

Once the resource is started and the HA cluster has executed the first monitor operation, it will add additional node attributes describing the current state of SAP HANA databases on nodes, as seen below:

[root]# pcs status --full
...
Node Attributes:
* Node node1:
    + hana_rh1_clone_state              : PROMOTED
    + hana_rh1_op_mode                  : delta_datashipping
    + hana_rh1_remoteHost               : node2
    + hana_rh1_roles                    : 4:P:master1:master:worker:master
    + hana_rh1_site                     : DC1
    + hana_rh1_sync_state               : PRIM
    + hana_rh1_srmode                   : syncmem
    + hana_rh1_version                	: 2.00.064.00.1660047502
    + hana_rh1_vhost                    : node1
    + lpa_rh1_lpt                       : 1495204085
    + master-SAPHana_RH1_02             : 150
* Node node2:
    + hana_r12_clone_state              : DEMOTED
    + hana_rh1_op_mode                  : delta_datashipping
    + hana_rh1_remoteHost               : node1
    + hana_rh1_roles                    : 4:S:master1:master:worker:master
    + hana_rh1_site                     : DC2
    + hana_rh1_srmode                   : syncmem
    + hana_rh1_sync_state               : SOK
    + hana_rh1_version                	: 2.00.064.00.1660047502
    + hana_rh1_vhost                    : node2
    + lpa_rh1_lpt                       : 30
    + master-SAPHana_RH1_02             : -INFINITY
...

In order for clients to be able to access the primary SAP HANA instance independently from the HA cluster node it is currently running on, a virtual IP address is needed, which the HA cluster will enable on the node where the primary SAP HANA instance is running.

To allow the HA cluster to manage the VIP, create IPaddr2 resource with IP 192.168.0.15.

[root]# pcs resource create vip_RH1_02 IPaddr2 ip="192.168.0.15"

Please use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running.

The resulting HA cluster resource should look as follows:

[root]# pcs resource show vip_RH1_02
Resource: vip_RH1_02 (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.0.15
Operations: start interval=0s timeout=20s (vip_RH1_02-start-interval-0s)
            stop interval=0s timeout=20s (vip_RH1_02-stop-interval-0s)
            monitor interval=10s timeout=20s (vip_RH1_02-monitor-interval-10s)

For correct operation, we need to ensure that SAPHanaTopology resources are started before starting SAPHana resources and also that the virtual IP address is present on the node where the primary SAP HANA instance is running.

To achieve this, the following constraints are required.

[root]# pcs constraint order SAPHanaTopology_RH1_02-clone then SAPHana_RH1_02-clone symmetrical=false

[root]# pcs constraint
...
Ordering Constraints:
  start SAPHanaTopology_RH1_02-clone then start SAPHana_RH1_02-clone (kind:Mandatory) (non-symmetrical)
...

[root]# pcs constraint colocation add vip_RH1_02 with master SAPHana_RH1_02-clone 2000

[root]# pcs constraint
...
Colocation Constraints:
  vip_RH1_02 with SAPHana_RH1_02-clone (score:2000) (rsc-role:Started) (with-rsc-role:Master)
...

Starting with SAP HANA 2.0 SPS1, SAP HANA supports Active/Active (Read Enabled) setups for SAP HANA System Replication, where the secondary instance of a SAP HANA System Replication setup can be used for read-only access.

To be able to support such setups, a second virtual IP address is required, which enables clients to access the secondary SAP HANA instance. To ensure that the secondary replication site can still be accessed after a takeover has occurred, the HA cluster needs to move the virtual IP address around with the slave of the promotable SAPHana resource.

To enable the Active/Active (Read Enabled) mode in SAP HANA, the operationMode must be set to logreplay_readaccess when registering the secondary SAP HANA instance.

[root]# pcs resource create vip2_RH1_02 IPaddr2 ip="192.168.1.11"

[root]# pcs constraint location vip2_RH1_02 rule score=INFINITY hana_rh1_sync_state eq SOK and hana_rh1_roles eq 4:S:master1:master:worker:master
[root]# pcs constraint location vip2_RH1_02 rule score=2000 hana_rh1_sync_state eq PRIM and hana_rh1_roles eq 4:P:master1:master:worker:master

When HANA detects an issue with an indexserver process, it recovers it by stopping and restarting it automatically via the in-built functionality built into SAP HANA.

However, in some cases, the service can take a very long time for the "stopping" phase. During that time, the System Replication may get out of sync, while HANA still proceeds to work and accept new connections. Eventually, the service completes the stop-and-restart process and recovers.

Instead of waiting for this long-running restart, which poses a risk to data consistency, should anything else fail in the instance during that time, the ChkSrv.py hook script can react to the situation and stop the HANA instance for a faster recovery. In a setup with automated failover enabled, the instance stop leads to a takeover being initiated, if the secondary node is in a healthy state. Otherwise, recovery would continue locally, but the enforced instance restart would speed it up.

When configured in the global.ini config file, SAP HANA calls the ChkSrv.py hook script for any events in the instance. The script processes the events and executes actions based on the results of the filters it applies to event details. This way, it can distinguish a HANA indexserver process that is being stopped-and-restarted by HANA after a failure from the same process being stopped as part of an instance shutdown.

Below are the different possible actions that can be taken:

Please note that both the stop and kill actions lead to a stopped HANA instance, with the kill being a bit faster in the end.

At this point, the cluster notices the failure of the HANA resource and reacts to it in the way it has been configured; typically, it restarts the instance, and if enabled, it also takes care of a takeover.