Chapter 7. Testing the setup

Procedure

1. As user <sid>adm go to the HANA Python directory on the primary site and check the current system replication state. Verify that it is ACTIVE and fully synced:

   rh1adm $ cdpy; python systemReplicationStatus.py
   …
   status system replication site "2": ACTIVE
   overall system replication status: ACTIVE
   …

2. Verify that the srHook and srPoll cluster attributes are both SOK in the attributes summary of the secondary site. Run this command as the root user on any node in a separate terminal to keep track of the attribute changes:

   [root]# watch SAPHanaSR-showAttr
   ...
   Site lpt        lss mns      opMode    srHook srMode srPoll srr
   ----------------------------------------------------------------
   DC2  30         4   dc2hana1 logreplay SOK    sync   SOK    S
   DC1  1757076772 4   dc1hana1 logreplay PRIM   sync   PRIM   P
   ...

   You can use the watch command to run the command in a loop at a default interval of 2 seconds.

3. On an instance on the secondary site, for example, dc2hana2, get the process ID (PID) of the hdbindexserver process. For example, you can get it from the PID column of the HDB info output as user <sid>adm:

   rh1adm $ HDB info

4. On the same instance on the secondary site, use the PID to simulate a hanging hdbindexserver process by sending the STOP signal to the process. This freezes the process and blocks it from communicating and syncing the instance between the nodes:

   rh1adm $ kill -STOP <PID>

Verification

1. On the primary site, watch the system replication status for the change on any primary instance. In the following example the system’s cut utility helps you limit the output to certain fields for readability. Remove it to see all columns of the table formatted text output. In the example we froze the indexserver on the secondary node dc2hana2, which results in a replication error with that node’s counterpart on the primary site, dc1hana2:

   rh1adm $ cdpy; watch "python systemReplicationStatus.py | cut -d '|' -f 1-3,5,9,13-"
   ...
   |Database |Host     |Service Name |Secondary |Secondary     |Replication |Replication |Replication                   |Secondary    |
   |         |         |             |Host      |Active Status |Mode        |Status      |Status Details                |Fully Synced |
   |-------- |-------- |------------ |--------- |------------- |----------- |----------- |----------------------------- |------------ |
   |RH1      |dc1hana2 |indexserver  |dc2hana2  |YES           |SYNC        |ERROR       |Log shipping timeout occurred |       False |
   |SYSTEMDB |dc1hana1 |nameserver   |dc2hana1  |YES           |SYNC        |ACTIVE      |                              |        True |
   |RH1      |dc1hana1 |xsengine     |dc2hana1  |YES           |SYNC        |ACTIVE      |                              |        True |
   |RH1      |dc1hana1 |indexserver  |dc2hana1  |YES           |SYNC        |ACTIVE      |                              |        True |
   
   status system replication site "2": ERROR
   overall system replication status: ERROR
   ...

   The replication status changes to ERROR for the indexserver service after a bit. It can take a while to react on an idle instance, wait a minute or more.

2. On the primary site’s master name server node, check the HANA nameserver process log for the related messages as the <sid>adm user:

   rh1adm $ cdtrace; grep -he 'HanaSR.srConnectionChanged.*' nameserver_*
   ha_dr_HanaSR     HanaSR.py(00056) : HanaSR 1.001.1 HanaSR.srConnectionChanged method called with Dict={'hostname': 'dc1hana2', 'port': '30003', 'volume': 4, 'service_name': 'indexserver', 'database': 'RH1', 'status': 11, 'database_status': 11, 'system_status': 11, 'timestamp': '2025-09-12T11:15:08.003728+00:00', 'is_in_sync': False, 'system_is_in_sync': False, 'reason': '', 'siteName': 'DC2'}
   ha_dr_HanaSR     HanaSR.py(00065) : HanaSR HanaSR.srConnectionChanged system_status=11 SID=RH1 in_sync=False reason=
   ha_dr_HanaSR     HanaSR.py(00091) : HanaSR.srConnectionChanged() CALLING CRM: <sudo /usr/sbin/crm_attribute -n hana_rh1_site_srHook_DC2  -v SFAIL -t crm_config -s SAPHanaSR> ret_code=

   The nameserver process log contains the event that HANA triggers with details. It also includes the sudo command that the HanaSR hook script runs to update the srHook cluster attribute.

3. Verify that both cluster attributes for the system replication status, srHook and srPoll, show the SFAIL status of the secondary site. Run the following as the root user on any HANA node or use the open terminal from the previous steps to watch the changes:

   [root]# SAPHanaSR-showAttr
   ...
   Site lpt        lss mns      opMode    srHook srMode srPoll srr
   ----------------------------------------------------------------
   DC2  10         4   dc2hana1 logreplay SFAIL  sync   SFAIL  S
   DC1  1757079061 4   dc1hana1 logreplay PRIM   sync   PRIM   P
   ...

4. Unblock the previously frozen hdbindexserver PID to enable it again. Run this on the secondary instance on which you blocked the hdbindexserver process for the test:

   rh1adm $ kill -CONT <PID>

5. Repeat the previous checks to verify that the system replication recovers fully after a bit. The cluster does not trigger any actions during this test since the resources remain running. Ensure that the system replication status is healthy again, fully synced and the cluster attributes are set to SOK again for the secondary site.

Procedure

1. Use a separate terminal to monitor the HANA processes as user <sid>adm on the instance on which you run this test:

   rh1adm $ watch "sapcontrol -nr ${TINSTANCE} -function GetProcessList | column -s ',' -t"

2. In another terminal on the same HANA instance, kill the hdbindexserver process:

   rh1adm $ kill <PID>

Verification

1. Check the dedicated HANA nameserver trace log on the same instance and identify the event and related action, as user <sid>adm:

   rh1adm $ cdtrace; less nameserver_chksrv.trc
   ...
   ChkSrv version 1.001.1. Method srServiceStateChanged method called.
   ChkSrv srServiceStateChanged method called with Dict={'hostname': 'dc2hana2',
    'service_name': 'indexserver', 'service_port': '30203', 'service_status': 'stopping',
    'service_previous_status': 'yes', 'timestamp': '2025-09-15T15:07:09.353198+00:00',
    'daemon_status': 'yes', 'database_id': '3', 'database_name': 'RH1',
    'database_status': 'yes', 'details': ''}
   ChkSrv srServiceStateChanged method called with SAPSYSTEMNAME=RH1
   srv:indexserver-30203-stopping-yes db:RH1-3-yes daem:yes
   LOST: indexserver event looks like a lost indexserver (status=stopping)
   LOST: stop instance. action_on_lost=stop
   ...

2. Check the cluster status for resource failure information on any cluster node, as user root:

   [root]# pcs status --full
   ...
   
   Failed Resource Actions:
     * rsc_SAPHanaCon_RH1_HDB02_monitor_61000 on dc2hana1 'not running' (7): call=26, status='complete', ...
   
   ...

3. Check the system log for the related cluster actions on the on the test node, for example, dc2hana2, as user root:

   [root]# grep rsc_SAPHanaCon_RH1_HDB02 /var/log/messages
   ...
   Sep 15 15:08:17 dc2hana1 pacemaker-controld[17045]: notice: Result of monitor operation for rsc_SAPHanaCon_RH1_HDB02 on dc2hana1: not running
   Sep 15 15:08:17 dc2hana1 pacemaker-controld[17045]: notice: rsc_SAPHanaCon_RH1_HDB02_monitor_61000@dc2hana1 output [ 10 ]
   Sep 15 15:08:17 dc2hana1 pacemaker-controld[17045]: notice: Transition 32 action 29 (rsc_SAPHanaCon_RH1_HDB02_monitor_61000 on dc2hana1): expected 'ok' but got 'not running'
   Sep 15 15:08:17 dc2hana1 pacemaker-attrd[17043]: notice: Setting last-failure-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana1] in instance_attributes: (unset) -> 1757948897
   Sep 15 15:08:17 dc2hana1 pacemaker-attrd[17043]: notice: Setting fail-count-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana1] in instance_attributes: (unset) -> 1
   Sep 15 15:08:17 dc2hana1 pacemaker-schedulerd[17044]: warning: Unexpected result (not running) was recorded for monitor of rsc_SAPHanaCon_RH1_HDB02:2 on dc2hana1 at Sep 15 15:08:17 2025
   Sep 15 15:08:17 dc2hana1 pacemaker-schedulerd[17044]: notice: Actions: Recover    rsc_SAPHanaCon_RH1_HDB02:2     (             Unpromoted dc2hana1 )
   Sep 15 15:08:17 dc2hana1 pacemaker-schedulerd[17044]: warning: Unexpected result (not running) was recorded for monitor of rsc_SAPHanaCon_RH1_HDB02:2 on dc2hana1 at Sep 15 15:08:17 2025
   Sep 15 15:08:17 dc2hana1 pacemaker-schedulerd[17044]: notice: Actions: Recover    rsc_SAPHanaCon_RH1_HDB02:2     (             Unpromoted dc2hana1 )
   Sep 15 15:08:17 dc2hana1 pacemaker-controld[17045]: notice: Initiating stop operation rsc_SAPHanaCon_RH1_HDB02_stop_0 locally on dc2hana1
   Sep 15 15:08:17 dc2hana1 pacemaker-controld[17045]: notice: Requesting local execution of stop operation for rsc_SAPHanaCon_RH1_HDB02 on dc2hana1
   ...

   The next SAPHanaController resource monitor reports the unexpectedly stopped HANA instance as a failure and initiates the recovery steps according to the configuration. If PREFER_SITE_TAKEOVER is enabled and you executed the test on a primary instance, it triggers a HANA takeover to the secondary site.

Procedure

1. Create a temporary file on a local filesystem of the node you want to test:

   [root]# touch /tmp/test

2. Set the local file to be immutable, which prevents write access:

   [root]# chattr +i /tmp/test

3. Go to the hidden directory which the SAPHanaFilesystem resource uses to test read and write filesystem access and enter the node you want to test:

   [root]# cd /hana/shared/<SID>/.heartbeat_SAPHanaFilesystem/<node>

4. Change the test file, which the SAPHanaFilesystem resource creates, to become a symbolic link that points to the temporary local file. Since the temporary target test file cannot be modified, the resource fails after the next monitor cycle:

   [root]# ln -sf /tmp/test test

   NFS filesystems do not support extended attributes by default. The symbolic link bridges this gap for the test.

5. Verify the behavior during the simulated failure.

   1. You can check the /var/log/messages file for the related log message if the resource action is set to ignore:

      [root]# grep -e 'SAPHanaFil.*ON_FAIL_ACTION' /var/log/messages
      ... SAPHanaFilesystem(rsc_SAPHanaFil_RH1_HDB02)[715184]: INFO: -2- RA monitor() ON_FAIL_ACTION=ignore => ignore FS error, do not create poison pill file

   2. If the resource action is set to fence you can observe the fencing action:

      [root]# pcs status --full
      ...
      
      Failed Resource Actions:
        * rsc_SAPHanaFil_RH1_HDB02_stop_0 on dc1hana1 'error' (1): ...
      
      Pending Fencing Actions:
        * reboot of dc1hana1 pending: client=pacemaker-controld.1694, origin=dc1hana2

6. You must remove the blocker again after the test. If the node is fenced, delete the symbolic link after the node is running again. The resource creates the regular test file again during the next check:

   [root]# rm -f /hana/shared/<SID>/.heartbeat_SAPHanaFilesystem/<node>/test

7. Clean up the temporary local test file:

   [root]# chattr -i /tmp/test; rm -f /tmp/test