Automating SAP HANA Scale-Out System Replication using the RHEL HA Add-On

Procedure

1. Create the directories for the shared filesystems:

   [root]# mkdir -p /hana/{shared,data,log}

2. Add the shared NFS filesystems to /etc/fstab to mount them automatically on system start. Configure the mount options that apply to your environment. The following is a basic example:

   [root]# vi /etc/fstab

   …
   <nfs_server>:/<site_path>/data /hana/data nfs4 defaults 0 0
   <nfs_server>:/<site_path>/log /hana/log nfs4 defaults 0 0
   <nfs_server>:/<site_path>/shared /hana/shared nfs4 defaults 0 0

   • Replace <nfs_server> with the NFS server DNS name or the IP address of each share, for example, nfs01-datacenter1a.example.com.
   • Replace <site_path> with the site specific root path, for example, dc1 on the nodes of one HANA site.

3. Reload the systemctl daemon to make the new /etc/fstab entries known to systemd:

   [root]# systemctl daemon-reload

4. Mount any new filesystems that you configured in the /etc/fstab:

   [root]# mount -a

5. Repeat the configuration steps on each cluster node.

Verification

1. Check that the filesystems are mounted, for example, on HANA site 1:

   [root]# df -hP | grep hana

   nfs01-datacenter1a.example.com:/dc1/hana/data    8.0E   32G  8.0E   1% /hana/data
   nfs01-datacenter1a.example.com:/dc1/hana/log     8.0E   32G  8.0E   1% /hana/log
   nfs01-datacenter1a.example.com:/dc1/hana/shared  8.0E   32G  8.0E   1% /hana/shared

2. Check that the systemd mount targets exist for the filesystems configured in the /etc/fstab:

   [root]# systemctl list-units --all | grep -e 'hana*.*mount' | column -t

   hana-data.mount    loaded  active  mounted  /hana/data
   hana-log.mount     loaded  active  mounted  /hana/log
   hana-shared.mount  loaded  active  mounted  /hana/shared

3. Repeat the verification steps on each cluster node.

Procedure

1. Create the sapsys group. Use the prepared group ID, for example, ID 10001:

   [root]# groupadd -g 10001 sapsys

2. Create the sapadm user as a member of the sapsys group. The user does not need a login shell. Use the prepared user ID, for example ID, 10200:

   [root]# useradd -u 10200 -g sapsys sapadm \
   -c 'SAP Local Administrator' -s /sbin/nologin

3. Create the <sid>adm user as a member of the sapsys group. Use the prepared user ID, for example, ID 10210 for user rh1adm:

   [root]# useradd -u 10210 -g sapsys rh1adm \
   -c 'SAP HANA Administrator' -s /bin/sh

   As the user shell, we recommend that you either use /bin/sh or /bin/csh. SAP installations provide user profiles and useful shell aliases in these shells.

4. Repeat the steps on all nodes.

Verification

1. Check that the users sapadm and <sid>adm exist and have the correct groups and IDs configured, for example:

   [root]# id sapadm rh1adm

   uid=10200(sapadm) gid=10001(sapsys) groups=10001(sapsys)
   uid=10210(rh1adm) gid=10001(sapsys) groups=10001(sapsys)

2. Check that the users have the correct description, home directory and shell defined:

   [root]# grep -E 'sapadm|rh1adm' /etc/passwd

   sapadm:x:10200:10001:SAP Local Administrator:/home/sapadm:/sbin/nologin
   rh1adm:x:10210:10001:SAP HANA Administrator:/home/rh1adm:/bin/sh

3. Repeat the check on all nodes and verify that the names and IDs are identical.

Procedure

1. Generate an ssh key pair. When no key type is defined, it creates an Ed25519 key by default, like in the following example for the root user:

   [root]# ssh-keygen

2. Option 1, if you have ssh PasswordAuthentication enabled on the remote system: Use the ssh-copy-id tool to add the ssh public key to the authorized_keys on the remote system. This automatically creates the .ssh/ directory and authorized_keys file with correct permissions for the target user on the remote system. Run it on the host on which you created the ssh key in the previous step and enter the target user password when prompted:

   [root]# ssh-copy-id <remote_system>

   In the case of the root user, this only works if the ssh config allows PermitRootLogin and you can provide the root user password in the prompt. Check the ssh configuration setting on the remote system if you face access permission issues even after you have enabled PasswordAuthentication. Consult your security policies before you enable these parameters on your HANA systems.

3. Option 2, if password login to the target user on the remote host is prohibited or otherwise not possible: Configure the ssh key access on the remote system manually.

   1. Create the .ssh/ directory in the target user’s home path on the remote system, if it does not exist yet. Run this on the remote system, for example, for the root user:

      [root]# mkdir /root/.ssh

   2. Change the permissions of the new .ssh/ directory. For security reasons the ssh key access does not work when the permissions are not correct. Run this on the remote system:

      [root]# chmod 0700 /root/.ssh

   3. Copy the ssh public key from the .pub file that was created by the previous ssh-keygen, for example, id_ed25519.pub in the default setting:

      [root]# cat /root/.ssh/id_ed25519.pub

   4. Add the public key to the authorized_keys file. The command creates the file if it does not exist yet, otherwise it appends the key to the existing content. Run this on the remote system, for example, on dc1hana2:

      [root]# cat << EOF >> /root/.ssh/authorized_keys
      ssh-ed25519 … root@<node1>
      EOF

   5. Ensure that the authorized_keys file has the correct permissions, otherwise the ssh key access is blocked for security reasons:

      [root]# chmod 0600 /root/.ssh/authorized_keys

4. Access each system and log in from any source host to any remote host that you require for the setup. On first login you must accept each new connection once in an interactive prompt. This saves each host and key in the ssh known_hosts file by default.

   1. Option 1: Log in from each host to each other host and accept the key fingerprint once to save it to the local known_hosts file. Subsequent logins to the same host will not require further interaction, unless the key changes. This is a security measure to prevent unsolicited changes of the ssh keys. The following example confirms the authenticity of host dc1hana2:

      [root]# ssh dc1hana2

      The authenticity of host 'dc1hana2 (pass:[***])' can't be established.
      ED25519 key fingerprint is SHA256:pass:[*********************************].
      This key is not known by any other names
      Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
      …

   2. Option 2: If you configure ssh key access between multiple systems you can use ssh-keyscan to collect the public host key from multiple hosts and save it to the local known_hosts file in a single step per host. Run this on each system for which you distributed the public key and list all remote hosts that you potentially access from this node and user, for example, for the root user on host dc1hana1:

      [root]# ssh-keyscan -f - >> /root/.ssh/known_hosts
      dc1hana1
      dc1hana2
      dc1hana3
      dc1hana4
      dc2hana1
      dc2hana2
      dc2hana3
      dc2hana4
      <Ctrl-d>

      # dc1hana1:22 SSH-2.0-OpenSSH_8.7
      dc1hana1 ssh-ed25519 …
      # dc1hana2:22 SSH-2.0-OpenSSH_8.7
      dc1hana2 ssh-ed25519 …
      # dc1hana3:22 SSH-2.0-OpenSSH_8.7
      dc1hana3 ssh-ed25519 …
      # dc1hana4:22 SSH-2.0-OpenSSH_8.7
      dc1hana4 ssh-ed25519 …
      # dc2hana1:22 SSH-2.0-OpenSSH_8.7
      dc2hana1 ssh-ed25519 …
      # dc2hana2:22 SSH-2.0-OpenSSH_8.7
      dc2hana2 ssh-ed25519 …
      # dc2hana3:22 SSH-2.0-OpenSSH_8.7
      dc2hana3 ssh-ed25519 …
      # dc2hana4:22 SSH-2.0-OpenSSH_8.7
      dc2hana4 ssh-ed25519 …

      • -f - allows you to provide a list of hosts on the standard input. Instead of the - you can use a file, which you prepare upfront with the list of hosts. You can also enter a single hostname instead of the -f parameter to collect the key of one host at a time.
      • In the case of the standard input list you end the input with Ctrl and d.
      • The >> shell redirection after the scan command directly appends the collected keys to the known_hosts file. If the file does not exist yet it is created in the process.

Procedure

1. Go to the directory which contains the installation media, for example, /sapmedia/hana:

   [root]# cd /sapmedia/hana

2. Unpack the installation media:

   [root]# unzip <sap_hana_software>.ZIP

3. Go into the path of the unpacked installation media:

   [root]# cd /sapmedia/hana/DATA_UNITS/HDB_LCM_LINUX_<arch>

4. Run the SAP HANA Lifecycle Management tool (HDBLCM) for an interactive installation:

   [root]# ./hdblcm

   In the interactive mode the installer asks you for all the required information, including the System ID (SID), Installation Number (instance), the filesystem location of data and log volumes, and more.

   In a scale-out installation you run the installer on the main node of one HANA site and provide any additional nodes of the same site as an installation parameter. For example, you run the installer for site 1 on dc1hana1 and add node dc1hana2 as an additional host name to add when the prompt asks for it.

   Optionally you can use the batch mode of the command-line installation tool and provide your configuration parameters in one step. For more details see Use Batch Mode to Perform Platform LCM Tasks in the SAP HANA Server Installation and Update Guide.

5. Repeat all steps on the main node of the second site. For the HANA system replication to work you must ensure that each HANA site consists of the same amount of systems with an identical HANA configuration.

Verification

1. Switch to the <sid>adm user:

   [root]# su - rh1adm

2. Check the HANA instance runtime information as user <sid>adm:

   rh1adm$HDB info

   USER          PID     PPID  %CPU        VSZ        RSS COMMAND
   rh1adm      12525    12524   0.2       8836       5568 -sh
   rh1adm      12584    12525   0.0       7520       3968  \_ /bin/sh /usr/sap/RH1/HDB02/HDB info
   rh1adm      12613    12584   0.0      10104       3484      \_ ps fx -U rh1adm -o user:8,pid:8,ppid:8,pcpu:5,vsz:10,rss:10,args
   rh1adm       8813        1   0.0     566804      41000 hdbrsutil  --start --port 30203 --volume 3 …
   rh1adm       8124        1   0.0     566724      40972 hdbrsutil  --start --port 30201 --volume 1 …
   rh1adm       7947        1   0.0       9312       3352 sapstart pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_dc1hana1
   rh1adm       7955     7947   0.0     460036      89176  \_ /usr/sap/RH1/HDB02/dc1hana1/trace/hdb.sapRH1_HDB02 -d -nw -f /usr/sap/RH1/HDB02/dc1hana1/daemon.ini pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_dc1hana1
   rh1adm       7981     7955  26.1   18612328   14092076      \_ hdbnameserver
   rh1adm       8642     7955   0.5    1465380     212048      \_ hdbcompileserver
   rh1adm       8645     7955   294    6616736    6049012      \_ hdbpreprocessor
   rh1adm       8687     7955  33.9   18931580   14929092      \_ hdbindexserver -port 30203
   rh1adm       8690     7955   2.0    5073572    1390440      \_ hdbxsengine -port 30207
   rh1adm       9202     7955   0.8    2772836     482088      \_ hdbwebdispatcher
   rh1adm       7782        1   0.1     566772      58444 /usr/sap/RH1/HDB02/exe/sapstartsrv pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_dc1hana1
   root        11868     7782   0.1      10464       4644  \_ sapuxuserchk 0 128

3. Verify as <sid>adm on all sites that the HANA instances are running on all nodes in the site and their status is GREEN in the instance list, for example, on site 1:

   rh1adm$sapcontrol -nr ${TINSTANCE} -function GetSystemInstanceList

   hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
   dc1hana4, 2, 50213, 50214, 0.3, HDB|HDB_STANDBY, GREEN
   dc1hana1, 2, 50213, 50214, 0.3, HDB|HDB_WORKER, GREEN
   dc1hana3, 2, 50213, 50214, 0.3, HDB|HDB_WORKER, GREEN
   dc1hana2, 2, 50213, 50214, 0.3, HDB|HDB_WORKER, GREEN

4. Additionally, you can verify the landscapeHostConfiguration.py output for status ok:

   rh1adm$cdpy; python landscapeHostConfiguration.py

   | Host     | Host   | Host   | Failover | Remove | Storage   | Storage   | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host    | Host    | Worker  | Worker  |
   |          | Active | Status | Status   | Status | Config    | Actual    | Config   | Actual   | Config     | Actual     | Config      | Actual      | Config  | Actual  | Config  | Actual  |
   |          |        |        |          |        | Partition | Partition | Group    | Group    | Role       | Role       | Role        | Role        | Roles   | Roles   | Groups  | Groups  |
   | -------- | ------ | ------ | -------- | ------ | --------- | --------- | -------- | -------- | ---------- | ---------- | ----------- | ----------- | ------- | ------- | ------- | ------- |
   | dc1hana1 | yes    | ok     |          |        |         1 |         1 | default  | default  | master 1   | master     | worker      | master      | worker  | worker  | default | default |
   | dc1hana2 | yes    | ok     |          |        |         2 |         2 | default  | default  | master 2   | slave      | worker      | slave       | worker  | worker  | default | default |
   | dc1hana3 | yes    | ok     |          |        |         3 |         3 | default  | default  | slave      | slave      | worker      | slave       | worker  | worker  | default | default |
   | dc1hana4 | yes    | ignore |          |        |         0 |         0 | default  | default  | master 3   | slave      | standby     | standby     | standby | standby | default | -       |
   
   overall host status: ok

5. Check that the systemd units are installed for the HANA instance and the SAP Host Agent:

   [root]# systemctl list-unit-files --all sap* SAP*

   UNIT FILE            STATE     PRESET
   sapmedia.mount       generated -
   saphostagent.service enabled   disabled
   sapinit.service      generated -
   SAPRH1_02.service    enabled   disabled
   SAP.slice            static    -
   
   5 unit files listed.

6. Repeat the steps on all nodes. Note that the HANA profiles contain the individual node name in the format <SID>_HDB<instance>_<node>.

Procedure

1. Go to the HANA instance profile directory:

   [root]# cd /hana/shared/<SID>/profile

2. Edit the instance profile:

   [root]# vi <SID>_HDB<instance>_<hostname>

   Ensure that Autostart is set to 0.

3. Repeat the previous steps for each HANA instance that will be managed as part of the HA cluster.

Procedure

1. Optional: Create a dedicated directory for the backup in a suitable path, for example:

   [root]# mkdir <path>/<SID>-backup

   Replace <path> with a path on your system, which has enough free space for the initial backup files.

2. Change the owner of the backup path to user <sid>adm if the target directory is not already owned or writable by the HANA user, for example:

   [root]# chown <sid>adm:sapsys <path>/<SID>-backup

3. Change to the <sid>adm user for the remaining steps:

   [root]# su - <sid>adm

4. Create a backup of the SYSTEMDB as the <sid>adm user. Specify the path to the files the backups will be stored in. Ensure that the target filesystem has enough free space left, then create the backup:

   rh1adm$ hdbsql -i ${TINSTANCE} -u system -d SYSTEMDB \
   "BACKUP DATA USING FILE ('<path>/${SAPSYSTEMNAME}-backup/bkp-SYS')"
   Password: <HANA_SYSTEM_PASSWORD>

   • $TINSTANCE and $SAPSYSTEMNAME are environment variables that are part of the <sid>adm user shell environment. $TINSTANCE is the instance number and $SAPSYSTEMNAME is the SID. Both are automatically set to the instance values related to the <sid>adm user.
   • Replace <path> with the path where the <sid>adm user has write access and where there is enough free space left.

5. Create a backup of all tenant databases as the <sid>adm user. Specify the path to the files the backups will be stored in. Ensure that the target filesystem has enough free space left. Create the tenant DB backup:

   rh1adm$ hdbsql -i ${TINSTANCE} -u system -d SYSTEMDB \
   "BACKUP DATA FOR ${SAPSYSTEMNAME} USING FILE ('<path>/${SAPSYSTEMNAME}-backup/bkp-${SAPSYSTEMNAME}')"
   Password: <HANA_SYSTEM_PASSWORD>

   Replace <path> with the path where the <sid>adm user has write access and where there is enough free space left.

Verification

1. List the resulting backup files. Example when using /hana/log/RH1-backup as the directory to store the initial backup:

   rh1adm$ ls -lh /hana/log/RH1-backup/

   total 7.4G
   -rw-r-----. 1 rh1adm sapsys 156K Dec  9 16:13 bkp-RH1_databackup_0_1
   -rw-r-----. 1 rh1adm sapsys  81M Dec  9 16:13 bkp-RH1_databackup_2_1
   -rw-r-----. 1 rh1adm sapsys 3.6G Dec  9 16:13 bkp-RH1_databackup_3_1
   -rw-r-----. 1 rh1adm sapsys  81M Dec  9 16:13 bkp-RH1_databackup_4_1
   -rw-r-----. 1 rh1adm sapsys  81M Dec  9 16:13 bkp-RH1_databackup_5_1
   -rw-r-----. 1 rh1adm sapsys 172K Dec  9 16:11 bkp-SYS_databackup_0_1
   -rw-r-----. 1 rh1adm sapsys 3.6G Dec  9 16:12 bkp-SYS_databackup_1_1

2. Use the HANA command hdbbackupcheck to confirm the sanity of each backup file you created:

   rh1adm$ for i in $(ls /hana/log/RH1-backup/*); do hdbbackupcheck $i; done

   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Backup '/hana/log/RH1-backup/bkp-RH1_databackup_0_1' successfully checked.
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Backup '/hana/log/RH1-backup/bkp-RH1_databackup_2_1' successfully checked.
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Backup '/hana/log/RH1-backup/bkp-RH1_databackup_3_1' successfully checked.
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Backup '/hana/log/RH1-backup/bkp-RH1_databackup_4_1' successfully checked.
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Backup '/hana/log/RH1-backup/bkp-RH1_databackup_5_1' successfully checked.
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Backup '/hana/log/RH1-backup/bkp-SYS_databackup_0_1' successfully checked.
   Loaded library 'libhdbcsaccessor'
   Loaded library 'libhdblivecache'
   Backup '/hana/log/RH1-backup/bkp-SYS_databackup_1_1' successfully checked.

Procedure

1. Enable the system replication on the HANA site that becomes the initial primary. Run the command as <sid>adm on the first, or primary, node:

   rh1adm$hdbnsutil -sr_enable --name=<site1>

   nameserver is active, proceeding ...
   successfully enabled system as system replication source site
   done.

   • Replace <site1> with your primary HANA site name, for example, DC1.

Procedure

1. Stop the secondary HANA instances. Run as the <sid>adm user on one secondary instance, for example, on dc2hana1:

   rh1adm$ sapcontrol -nr ${TINSTANCE} -function StopSystem HDB

2. Run this and the following steps only on one node on the secondary site, for example, on dc2hana1. Change to the directory on the shared filesystem where HANA stores the keys of the system replication encryption on one node of the secondary site:

   [root]# cd /hana/shared/<SID>/global/security/rsecssfs

3. Copy the HANA system PKI file SSFS_<SID>.KEY from the primary HANA site to the secondary site on one secondary node:

   [root]# rsync -av <node1>:$PWD/key/SSFS_<SID>.KEY key/

   • Replace <node1> with a primary node, for example, dc1hana1.
   • Replace <SID> with your HANA SID, for example, RH1.

4. Copy the PKI file SSFS_<SID>.DAT from the primary HANA site to the secondary site on one secondary node in the same way as the previous step:

   [root]# rsync -av <node1>:$PWD/data/SSFS_<SID>.DAT data/

5. Register the secondary HANA site to the primary site. Run this in the <sid>adm user terminal on a secondary node:

   rh1adm$ hdbnsutil -sr_register --remoteHost=<node1> \
   --remoteInstance=${TINSTANCE} --replicationMode=sync \
   --operationMode=logreplay --name=<site2>

   adding site ...
   collecting information ...
   updating local ini files ...
   done.

   • Replace <node1> with a primary node, for example, dc1hana1.
   • Replace <site2> with your secondary HANA site name, for example, DC2.
   • Choose the values for replicationMode and operationMode according to your requirements for the system replication.
   • $TINSTANCE is an environment variable that is set automatically for user <sid>adm by reading the HANA instance profile. The variable value is the HANA instance number.

6. Start the secondary HANA instances. Run as <sid>adm on one HANA instance on the secondary site:

   rh1adm$ sapcontrol -nr ${TINSTANCE} -function StartSystem HDB

Verification

1. Check that the system replication is running on the secondary site and the mode matches the value you used for the replicationMode parameter in the hdbnsutil -sr_register command. Run as <sid>adm on one node on the secondary site, for example, dc2hana1:

   rh1adm$ hdbnsutil -sr_stateConfiguration

   System Replication State
   ~~~~~~~~~~~~~~~~~~~~~~~~
   
   mode: sync
   site id: 2
   site name: DC2
   active primary site: 1
   
   primary masters: dc1hana1 dc1hana2 dc1hana4

2. Change to the Python script directory of the HANA instance as user <sid>adm, on both sites. The easiest way to do this is to use cdpy, which is an alias built into the <sid>adm user shell that SAP HANA populates during the instance installation:

   rh1adm$ cdpy

   In our example this command changes the current directory to /usr/sap/RH1/HDB02/exe/python_support/.

3. Show the current status of the established HANA system replication, on both HANA sites.

   1. On the primary site the system replication status is always displayed with all details:

      rh1adm$ python systemReplicationStatus.py

      |Database |Host     |Port  |Service Name |Volume ID |Site ID |Site Name |Secondary |Secondary |Secondary |Secondary |Secondary     |Replication |Replication |Replication    |Secondary    |
      |         |         |      |             |          |        |          |Host      |Port      |Site ID   |Site Name |Active Status |Mode        |Status      |Status Details |Fully Synced |
      |-------- |-------- |----- |------------ |--------- |------- |--------- |--------- |--------- |--------- |--------- |------------- |----------- |----------- |-------------- |------------ |
      |RH1      |dc1hana3 |30203 |indexserver  |        5 |      1 |DC1       |dc2hana3  |    30203 |        2 |DC2       |YES           |SYNC        |ACTIVE      |               |        True |
      |RH1      |dc1hana2 |30203 |indexserver  |        4 |      1 |DC1       |dc2hana2  |    30203 |        2 |DC2       |YES           |SYNC        |ACTIVE      |               |        True |
      |SYSTEMDB |dc1hana1 |30201 |nameserver   |        1 |      1 |DC1       |dc2hana1  |    30201 |        2 |DC2       |YES           |SYNC        |ACTIVE      |               |        True |
      |RH1      |dc1hana1 |30207 |xsengine     |        2 |      1 |DC1       |dc2hana1  |    30207 |        2 |DC2       |YES           |SYNC        |ACTIVE      |               |        True |
      |RH1      |dc1hana1 |30203 |indexserver  |        3 |      1 |DC1       |dc2hana1  |    30203 |        2 |DC2       |YES           |SYNC        |ACTIVE      |               |        True |
      
      status system replication site "2": ACTIVE
      overall system replication status: ACTIVE
      
      Local System Replication State
      ~~~~~~~~~~
      
      mode: PRIMARY
      site id: 1
      site name: DC1

      • ACTIVE means that the HANA system replication is in a healthy state and fully synced.
      • SYNCING is displayed while the system replication is being updated on the secondary site, for example after a takeover of the secondary site to become the new primary.
      • INITIALIZING is shown after the system replication has first been enabled or a full sync has been triggered.

   2. On the secondary site, the output of systemReplicationStatus.py is less detailed. Check the status on one secondary node:

      rh1adm$ python systemReplicationStatus.py
      this system is either not running or not primary system replication site
      
      Local System Replication State
      ~~~~~~~~~~
      
      mode: SYNC
      site id: 2
      site name: DC2
      active primary site: 1
      primary masters: dc1hana1 dc1hana2 dc1hana4

Procedure

1. Create the directory /etc/systemd/system/pacemaker.service.d/ for the pacemaker service drop-in file:

   [root]# mkdir /etc/systemd/system/pacemaker.service.d/

2. Create the systemd drop-in file for the pacemaker service with the following content:

   [root]# cat << EOF > /etc/systemd/system/pacemaker.service.d/00-pacemaker.conf
   [Unit]
   Description=Pacemaker needs the SAP HANA instance service
   Wants=SAP<SID>_<instance>.service
   After=SAP<SID>_<instance>.service
   EOF

   • Replace <SID> with your HANA SID.
   • Replace <instance> with your HANA instance number.

3. Reload the systemctl daemon to activate the drop-in file:

   [root]# systemctl daemon-reload

4. Repeat steps 1-3 on the other HANA cluster nodes.

Verification

1. Check the systemd service of your HANA instance and verify that it is loaded:

   [root]# systemctl status SAPRH1_02.service

   ● SAPRH1_02.service - SAP Instance SAPRH1_02
        Loaded: loaded (/etc/systemd/system/SAPRH1_02.service; disabled; preset: disabled)
        Active: active (running) since xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      Main PID: 5825 (sapstartsrv)
         Tasks: 841
        Memory: 88.6G
           CPU: 4h 50min 2.033s
        CGroup: /SAP.slice/SAPRH1_02.service
                ├─ 5825 /usr/sap/RH1/HDB02/exe/sapstartsrv pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_dc1hana1
                ├─ 5986 sapstart pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_dc1hana1
                ├─ 5993 /usr/sap/RH1/HDB02/dc1hana1/trace/hdb.sapRH1_HDB02 -d -nw -f /usr/sap/RH1/HDB02/dc1hana1/daemon.ini pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_dc1hana1
   ...

2. Verify that the SAP HANA instance service is known to the pacemaker service now:

   [root]# systemctl show pacemaker.service | grep -E 'Wants=|After=|SAP.{6}.service'

   Wants=SAPRH1_02.service resource-agents-deps.target dbus-broker.service
   After=... SAPRH1_02.service …

   Make sure that the SAP<SID>_<instance>.service is listed in the After= and Wants= lists.

Procedure

1. Stop the HANA instances on all nodes. Run this as the <sid>adm user on one HANA instance per site:

   rh1adm$ sapcontrol -nr ${TINSTANCE} -function StopSystem HDB

2. Verify as <sid>adm on all sites that the HANA instances are stopped completely and their status is GRAY in the instance list. Run this on one host on each site:

   rh1adm$ sapcontrol -nr ${TINSTANCE} -function GetSystemInstanceList

   GetSystemInstanceList
   OK
   hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
   dc1hana2, 2, 50213, 50214, 0.3, HDB|HDB_WORKER, GRAY
   dc1hana3, 2, 50213, 50214, 0.3, HDB|HDB_WORKER, GRAY
   dc1hana1, 2, 50213, 50214, 0.3, HDB|HDB_WORKER, GRAY
   dc1hana4, 2, 50213, 50214, 0.3, HDB|HDB_STANDBY, GRAY

3. Change to the HANA configuration directory, as the <sid>adm user, using the command alias cdcoc, which is built into the <sid>adm user shell. This automatically changes into the /hana/shared/<SID>/global/hdb/custom/config/ path:

   rh1adm$ cdcoc

4. Update the global.ini file of the SAP HANA site to configure the SAPHanaSR hook. Edit the configuration file on one node of each HANA site and add the following configuration:

   [ha_dr_provider_SAPHanaSR]
   provider = SAPHanaSR
   path = /usr/share/SAPHanaSR-ScaleOut
   execution_order = 1
   
   [trace]
   ha_dr_saphanasr = info

   Set execution_order to 1 to ensure that the SAPHanaSR hook is always executed with the highest priority.

   Due to the shared /hana/shared filesystem between the nodes of each HANA site, you only adjust the configuration once per site. Do not try to edit the same file on the shared filesystem simultaneously on more than one node of the same site.

5. Optional: When you also want to configure the optional ChkSrv hook for taking action on an hdbindexserver failure, you can add the changes to the global.ini at the same time, see step 1 in Configuring the ChkSrv HA/DR provider for the srServiceStateChanged() hook method:

   [ha_dr_provider_SAPHanaSR]
   provider = SAPHanaSR
   path = /usr/share/SAPHanaSR-ScaleOut
   execution_order = 1
   
   [ha_dr_provider_chksrv]
   provider = ChkSrv
   path = /usr/share/SAPHanaSR-ScaleOut
   execution_order = 2
   action_on_lost = stop
   
   [trace]
   ha_dr_saphanasr = info
   ha_dr_chksrv = info

6. Create the file /etc/sudoers.d/20-saphana, as the root user, on each cluster node with the following content. These command privileges allow the <sid>adm user to update certain cluster node attributes as part of the SAPHanaSR hook execution:

   [root]# visudo -f /etc/sudoers.d/20-saphana

   Defaults:<sid>adm !requiretty
   <sid>adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_*

   • Replace <sid> with the lower-case HANA SID.
   • For further information on why the Defaults setting is needed, refer to The srHook attribute is set to SFAIL in a Pacemaker cluster managing SAP HANA system replication, even though replication is in a healthy state.

7. Start the HANA instances on all cluster nodes manually without starting the HA cluster. Run as <sid>adm on one HANA instance per site:

   rh1adm$ sapcontrol -nr ${TINSTANCE} -function StartSystem HDB

Verification

1. Change to the SAP HANA directory, as the <sid>adm user, where trace log files are stored. Use the command alias cdtrace, which is built into the <sid>adm user shell:

   rh1adm$ cdtrace

2. Check the HANA nameserver process logs for the HA/DR provider loading message:

   rh1adm$ grep -he "loading HA/DR Provider.*" nameserver_*

   1. If only the SAPHanaSR provider is configured:

      loading HA/DR Provider 'SAPHanaSR' from /usr/share/SAPHanaSR-ScaleOut

   2. If the optional ChkSrv provider is also implemented:

      loading HA/DR Provider 'ChkSrv' from /usr/share/SAPHanaSR-ScaleOut
      loading HA/DR Provider 'SAPHanaSR' from /usr/share/SAPHanaSR-ScaleOut

3. Verify as user root in the system secure log that the sudo command executed successfully. If the sudoers file is not correct, an error is logged when the sudo command is executed. Check this on the primary node on which the HANA master nameserver is running, for example, dc1hana1:

   [root]# grep -e 'sudo.*crm_attribute.*' /var/log/secure

   sudo[17141]:  rh1adm : PWD=/hana/shared/RH1/HDB02/dc1hana1 ; USER=root ; COMMAND=/usr/sbin/crm_attribute -n hana_rh1_gsh -v 1.0 -l reboot
   sudo[17160]:  rh1adm : PWD=/hana/shared/RH1/HDB02/dc1hana1 ; USER=root ; COMMAND=/usr/sbin/crm_attribute -n hana_rh1_glob_srHook -v SFAIL -t crm_config -s SAPHanaSR
   …
   sudo[17584]:  rh1adm : PWD=/hana/shared/RH1/HDB02/dc1hana1 ; USER=root ; COMMAND=/usr/sbin/crm_attribute -n hana_rh1_glob_srHook -v SOK -t crm_config -s SAPHanaSR

   After the HANA instance starts on both nodes, you can usually see several srHook attribute updates. At first it is setting SFAIL, because immediately after the primary starts, it is not yet in sync with the secondary, which is still synchronizing at this time.

   The last update to SOK is triggered by the HANA event after the system replication status finally changes to fully in sync.

4. Repeat the verification steps 1-2 on the second site, if not already done at the same time. The sudo log messages of step 3 are only visible on the primary instance coordinator nameserver node, on which the system replication events are logged.

5. Check the cluster attributes on any node and verify that the value for the hana_<sid>_glob_srHook attribute is updated as expected:

   [root]# cibadmin --query | grep -e 'SAPHanaSR.*srHook'

    <nvpair id="SAPHanaSR-hana_rh1_glob_srHook" name="hana_rh1_glob_srHook" value="SOK"/>

   • SOK is set when the HANA system replication is in ACTIVE state, which means established and fully in sync.
   • SFAIL is set when the system replication is in any other state.

Procedure

1. Change to the HANA configuration directory as the <sid>adm user. Use the command alias cdcoc, which is built into the <sid>adm user shell. This automatically changes into the /hana/shared/<SID>/global/hdb/custom/config/ path:

   rh1adm$ cdcoc

2. Update the global.ini file of the HANA site to configure the hook script. Edit the configuration file one node of each HANA site and add the following content in addition to the SAPHanaSR provider definition:

   [ha_dr_provider_chksrv]
   provider = ChkSrv
   path = /usr/share/SAPHanaSR-ScaleOut
   execution_order = 2
   action_on_lost = stop
   
   [trace]
   ha_dr_saphanasr = info
   ha_dr_chksrv = info

   Due to the shared /hana/shared filesystem between the nodes of each HANA site, you only adjust the configuration once per site. Do not try to edit the same file on the shared filesystem simultaneously on more than one node of the same site.

3. Optional: Activate the ChkSrv provider while HANA is running by reloading the HA/DR providers. Skip this step when configuring the hook script while the instance is down, the HA/DR provider is loaded automatically at the next instance start.

   rh1adm$ hdbnsutil -reloadHADRProviders

Verification

1. Change to the SAP HANA directory, as the <sid>adm user, where trace log files are stored. Use the command alias cdtrace, which is built into the <sid>adm user shell:

   rh1adm$ cdtrace

2. Check that the changes are loaded:

   rh1adm$ grep -e "loading HA/DR Provider.*ChkSrv.*" nameserver_*

   loading HA/DR Provider 'ChkSrv' from /usr/share/SAPHanaSR-ScaleOut

3. Check that the dedicated trace log file is created and the provider loaded with the correct configuration parameters:

   rh1adm$ cat nameserver_chksrv.trc

   init called
   ChkSrv.init() version 0.7.8, parameter info: action_on_lost=stop stop_timeout=20 kill_signal=9
   …

Procedure

1. Create the SAPHanaTopology resource as a clone resource, which means it runs on all cluster nodes at the same time:

   [root]# pcs resource create rsc_SAPHanaTop_<SID>_HDB<instance> \
   ocf:heartbeat:SAPHanaTopology \
   SID=<SID> \
   InstanceNumber=<instance> \
   op start timeout=600 \
   op stop timeout=300 \
   op monitor interval=30 timeout=300 \
   clone cln_SAPHanaTop_<SID>_HDB<instance>

   • Replace <SID> with your HANA SID.
   • Replace <instance> with your HANA instance number.

2. Update the meta attributes of the new SAPHanaTopology clone resource:

   [root]# pcs resource update cln_SAPHanaTop_<SID>_HDB<instance> \
   meta clone-node-max=1 interleave=true

3. Create the SAPHanaController resource as a promotable clone resource. This means it runs on all cluster nodes at the same time, but on one node it functions as the active, or primary, resource:

   [root]# pcs resource create rsc_SAPHanaCon_<SID>_HDB<instance> \
   ocf:heartbeat:SAPHanaController \
   SID=<SID> \
   InstanceNumber=<instance> \
   PREFER_SITE_TAKEOVER=true \
   DUPLICATE_PRIMARY_TIMEOUT=7200 \
   AUTOMATED_REGISTER=false \
   op stop timeout=3600 \
   op monitor interval=59 role=Promoted timeout=700 \
   op monitor interval=61 role=Unpromoted timeout=700 \
   meta priority=100 \
   promotable cln_SAPHanaCon_<SID>_HDB<instance>

   We recommend that you create the resource with AUTOMATED_REGISTER=false and then verify the correct behavior and data consistency through tests to complete the setup. For more information, see Testing the setup. You can enable this already at creation time by setting the parameter to true.

   See SAPHanaController resource parameters for more details.

4. Update the meta attributes of the new SAPHanaController clone resource:

   [root]# pcs resource update cln_SAPHanaCon_<SID>_HDB<instance> \
   meta clone-node-max=1 interleave=true

5. You must start the SAPHanaTopology resource before the SAPHanaController resource, because it collects HANA landscape information, which the SAPHanaController resource requires to start correctly. Create the cluster constraint that enforces the correct start order of the two resources:

   [root]# pcs constraint order cln_SAPHanaTop_<SID>_HDB<instance> \
   then cln_SAPHanaCon_<SID>_HDB<instance> symmetrical=false

   Setting symmetrical=false indicates that the constraint only influences the start order of the resources, but it does not apply to the stop order.

Verification

1. Review the SAPHanaTopology resource clone. For operations that you do not define at resource creation, pcs automatically applies default values. Example resource configuration:

   [root]# pcs resource config cln_SAPHanaTop_RH1_HDB02

   Clone: cln_SAPHanaTop_RH1_HDB02
     Meta Attributes: cln_SAPHanaTop_RH1_HDB02-meta_attributes
       clone-node-max=1
       interleave=true
     Resource: rsc_SAPHanaTop_RH1_HDB02 (class=ocf provider=heartbeat type=SAPHanaTopology)
       Attributes: rsc_SAPHanaTop_RH1_HDB02-instance_attributes
         InstanceNumber=02
         SID=RH1
       Operations:
         methods: rsc_SAPHanaTop_RH1_HDB02-methods-interval-0s
           interval=0s
           timeout=5
         monitor: rsc_SAPHanaTop_RH1_HDB02-monitor-interval-30
           interval=30
           timeout=300
         reload: rsc_SAPHanaTop_RH1_HDB02-reload-interval-0s
           interval=0s
           timeout=5
         start: rsc_SAPHanaTop_RH1_HDB02-start-interval-0s
           interval=0s
           timeout=600
         stop: rsc_SAPHanaTop_RH1_HDB02-stop-interval-0s
           interval=0s
           timeout=300

2. Review the SAPHanaController resource clone. Example resource configuration:

   [root]# pcs resource config cln_SAPHanaCon_RH1_HDB02

   Clone: cln_SAPHanaCon_RH1_HDB02
     Meta Attributes: cln_SAPHanaCon_RH1_HDB02-meta_attributes
       clone-node-max=1
       interleave=true
       promotable=true
     Resource: rsc_SAPHanaCon_RH1_HDB02 (class=ocf provider=heartbeat type=SAPHanaController)
       Attributes: rsc_SAPHanaCon_RH1_HDB02-instance_attributes
         AUTOMATED_REGISTER=false
         DUPLICATE_PRIMARY_TIMEOUT=7200
         InstanceNumber=02
         PREFER_SITE_TAKEOVER=true
         SID=RH1
       Meta Attributes: rsc_SAPHanaCon_RH1_HDB02-meta_attributes
         priority=100
       Operations:
         demote: rsc_SAPHanaCon_RH1_HDB02-demote-interval-0s
           interval=0s
           timeout=320
         methods: rsc_SAPHanaCon_RH1_HDB02-methods-interval-0s
           interval=0s
           timeout=5
         monitor: rsc_SAPHanaCon_RH1_HDB02-monitor-interval-59
           interval=59
           timeout=700
           role=Promoted
         monitor: rsc_SAPHanaCon_RH1_HDB02-monitor-interval-61
           interval=61
           timeout=700
           role=Unpromoted
         promote: rsc_SAPHanaCon_RH1_HDB02-promote-interval-0s
           interval=0s
           timeout=3600
         reload: rsc_SAPHanaCon_RH1_HDB02-reload-interval-0s
           interval=0s
           timeout=5
         start: rsc_SAPHanaCon_RH1_HDB02-start-interval-0s
           interval=0s
           timeout=3600
         stop: rsc_SAPHanaCon_RH1_HDB02-stop-interval-0s
           interval=0s
           timeout=3600

3. Check that the start order constraint is in place:

   [root]# pcs constraint order

   Order Constraints:
     start resource 'cln_SAPHanaTop_RH1_HDB02' then start resource 'cln_SAPHanaCon_RH1_HDB02'
       symmetrical=0

4. Check the cluster status. Use --full to include node attributes, which are updated by the HANA resources:

   [root]# pcs status --full

   ...
   Full List of Resources:
     * rsc_fence       (stonith:<fence agent>):     Started dc1hana1
     * Clone Set: cln_SAPHanaTop_RH1_HDB02 [rsc_SAPHanaTop_RH1_HDB02]:
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana4
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana1
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana2
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana3
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana4
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana1
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana2
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana3
     * Clone Set: cln_SAPHanaCon_RH1_HDB02 [rsc_SAPHanaCon_RH1_HDB02] (promotable):
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana4
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana1
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana2
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana3
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana4
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Promoted dc1hana1
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana2
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana3
   
   Node Attributes:
     * Node: dc1hana1 (1):
       * hana_rh1_clone_state              : PROMOTED
       * hana_rh1_gra                      : 2.0
       * hana_rh1_gsh                      : 1.0
       * hana_rh1_roles                    : master1:master:worker:master
       * hana_rh1_site                     : DC1
       * master-rsc_SAPHanaCon_RH1_HDB02   : 150
     * Node: dc1hana2 (2):
       * hana_rh1_clone_state              : DEMOTED
       * hana_rh1_gra                      : 2.0
       * hana_rh1_gsh                      : 1.0
       * hana_rh1_roles                    : master2:slave:worker:slave
       * hana_rh1_site                     : DC1
       * master-rsc_SAPHanaCon_RH1_HDB02   : 140
     * Node: dc1hana3 (3):
       * hana_rh1_clone_state              : DEMOTED
       * hana_rh1_gra                      : 2.0
       * hana_rh1_gsh                      : 1.0
       * hana_rh1_roles                    : slave:slave:worker:slave
       * hana_rh1_site                     : DC1
       * master-rsc_SAPHanaCon_RH1_HDB02   : -10000
     * Node: dc1hana4 (4):
       * hana_rh1_clone_state              : DEMOTED
       * hana_rh1_gra                      : 2.0
       * hana_rh1_gsh                      : 1.0
       * hana_rh1_roles                    : master3:slave:standby:standby
       * hana_rh1_site                     : DC1
       * master-rsc_SAPHanaCon_RH1_HDB02   : 140
     * Node: dc2hana1 (5):
       * hana_rh1_clone_state              : DEMOTED
       * hana_rh1_gra                      : 2.0
       * hana_rh1_gsh                      : 1.0
       * hana_rh1_roles                    : master1:master:worker:master
       * hana_rh1_site                     : DC2
       * master-rsc_SAPHanaCon_RH1_HDB02   : 100
     * Node: dc2hana2 (6):
       * hana_rh1_clone_state              : DEMOTED
       * hana_rh1_gra                      : 2.0
       * hana_rh1_gsh                      : 1.0
       * hana_rh1_roles                    : master2:slave:worker:slave
       * hana_rh1_site                     : DC2
       * master-rsc_SAPHanaCon_RH1_HDB02   : 80
     * Node: dc2hana3 (7):
       * hana_rh1_clone_state              : DEMOTED
       * hana_rh1_gra                      : 2.0
       * hana_rh1_gsh                      : 1.0
       * hana_rh1_roles                    : slave:slave:worker:slave
       * hana_rh1_site                     : DC2
       * master-rsc_SAPHanaCon_RH1_HDB02   : -12200
     * Node: dc2hana4 (8):
       * hana_rh1_clone_state              : DEMOTED
       * hana_rh1_gra                      : 2.0
       * hana_rh1_gsh                      : 1.0
       * hana_rh1_roles                    : master3:slave:standby:standby
       * hana_rh1_site                     : DC2
       * master-rsc_SAPHanaCon_RH1_HDB02   : 80
   
   ...

Procedure

1. Use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running. Adjust the parameters according to the resource agent you are using. Create the cluster resource for the primary virtual IP, for example, using the IPaddr2 agent:

   [root]# pcs resource create rsc_vip_<SID>_HDB<instance>_primary \
   ocf:heartbeat:IPaddr2 ip=<address> cidr_netmask=<netmask> nic=<device>

   • Replace <SID> with your HANA SID.
   • Replace <instance> with your HANA instance number.
   • Replace <address>, <netmask> and <device> with the details of your primary virtual IP address.

2. Create a cluster constraint that places the VIP resource with the SAPHanaController resource on the HANA primary node:

   [root]# pcs constraint colocation add rsc_vip_<SID>_HDB<instance>_primary \
   with promoted cln_SAPHanaCon_<SID>_HDB<instance> 2000

   The constraint applies a score of 2000 instead of the default INFINITY. This softens the resource dependency and allows the virtual IP resource to stay active in the case when there is no promoted SAPHanaController resource. This way it is still possible to use tools like the SAP Management Console (MMC) or SAP Landscape Management (LaMa) that can reach this IP address to query the status information of the HANA instance.

Verification

1. Check the resource configuration of the virtual IP resource, for example::

   [root]# pcs resource config rsc_vip_RH1_HDB02_primary

   Resource: rsc_vip_RH1_HDB02_primary (class=ocf provider=heartbeat type=IPaddr2)
     Attributes: rsc_vip_RH1_HDB02_primary-instance_attributes
       cidr_netmask=32
       ip=192.168.1.100
       nic=eth0
     Operations:
       monitor: rsc_vip_RH1_HDB02_primary-monitor-interval-10s
         interval=10s
         timeout=20s
       start: rsc_vip_RH1_HDB02_primary-start-interval-0s
         interval=0s
         timeout=20s
       stop: rsc_vip_RH1_HDB02_primary-stop-interval-0s
         interval=0s
         timeout=20s

2. Check that the constraint is defined correctly:

   [root]# pcs constraint colocation

   Colocation Constraints:
     rsc_vip_RH1_HDB02_primary with cln_SAPHanaCon_RH1_HDB02 (score:2000) (rsc-role:Started) (with-rsc-role:Promoted)

3. Check that the resource is running on the promoted primary node, for example, dc1hana1:

   [root]# pcs status resources rsc_vip_RH1_HDB02_primary

     * rsc_vip_RH1_HDB02_primary   (ocf:heartbeat:IPaddr2):         Started dc1hana1

Procedure

1. Use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running. Adjust the parameters according to the resource agent you are using. Create the cluster resource for the secondary virtual IP, for example, using the IPaddr2 agent:

   [root]# pcs resource create rsc_vip_<SID>_HDB<site>_readonly \
   ocf:heartbeat:IPaddr2 ip=<address> cidr_netmask=<netmask> nic=<device>

   • Replace <SID> with your HANA SID.
   • Replace <site> with your HANA site number.
   • Replace <address>, <netmask> and <device> with the details of your read-only secondary virtual IP address.

2. Create a location constraint rule to ensure that the secondary virtual IP is assigned to the secondary site during normal operations:

   [root]# pcs constraint location rsc_vip_<SID>_HDB<site>_readonly \
   rule score=INFINITY master-rsc_SAPHanaCon_<SID>_HDB<site> eq 100 \
   and hana_<sid>_clone_state eq DEMOTED

   • Replace <SID> with your HANA SID.
   • Replace <sid> with the lower-case HANA SID.
   • Replace <site> with your HANA site number.

3. Create a location constraint rule to ensure that the secondary virtual IP runs on the primary site as an alternative whenever necessary:

   [root]# pcs constraint location rsc_vip_<SID>_HDB<site>_readonly \
   rule score=2000 master-rsc_SAPHanaCon_<SID>_HDB<site> eq 150 \
   and hana_<sid>_clone_state eq PROMOTED

Verification

1. Check the resource configuration of the secondary virtual IP resource, for example:

   [root]# pcs resource config rsc_vip_RH1_HDB02_readonly

   Resource: rsc_vip_RH1_HDB02_readonly (class=ocf provider=heartbeat type=IPaddr2)
     Attributes: rsc_vip_RH1_HDB02_readonly-instance_attributes
       cidr_netmask=32
       ip=192.168.1.200
       nic=eth0
     Operations:
       monitor: rsc_vip_RH1_HDB02_readonly-monitor-interval-10s
         interval=10s
         timeout=20s
       start: rsc_vip_RH1_HDB02_readonly-start-interval-0s
         interval=0s
         timeout=20s
       stop: rsc_vip_RH1_HDB02_readonly-stop-interval-0s
         interval=0s
         timeout=20s

2. Check that the constraints are part of the cluster configuration:

   [root]#  pcs constraint location

   Location Constraints:
     Resource: rsc_vip_RH1_HDB02_readonly
       Constraint: location-rsc_vip_RH1_HDB02_readonly
         Rule: boolean-op=and score=INFINITY
           Expression: master-rsc_SAPHanaCon_RH1_HDB02 eq 100
           Expression: hana_rh1_clone_state eq DEMOTED
       Constraint: location-rsc_vip_RH1_HDB02_readonly-1
         Rule: boolean-op=and score=2000
           Expression: master-rsc_SAPHanaCon_RH1_HDB02 eq 150
           Expression: hana_rh1_clone_state eq PROMOTED

3. Check that the resource is running on the main secondary node, for example, dc2hana1:

   [root]# pcs status resources rsc_vip_RH1_HDB02_readonly

     * rsc_vip_RH1_HDB02_readonly  (ocf:heartbeat:IPaddr2):         Started dc2hana1

Procedure

1. As user <sid>adm go to the HANA Python directory on the primary site and check the current system replication state. Verify that it is ACTIVE and fully synced:

   rh1adm$ cdpy; python systemReplicationStatus.py

   …
   status system replication site "2": ACTIVE
   overall system replication status: ACTIVE
   …

2. Verify that the srHook and sync_state cluster attributes are both SOK in the attributes summary of the secondary site. Run this command as the root user on any node in a separate terminal to keep track of the attribute changes:

   [root]# watch SAPHanaSR-showAttr

   Global cib-time                 prim sec srHook sync_state upd
   ---------------------------------------------------------------
   RH1    Thu Dec 11 10:59:25 2025 DC1  DC2 SOK    SOK        ok
   
   
   ...

   You can use the watch command to run the command in a loop at a default interval of 2 seconds.

3. On an instance on the secondary site, for example, dc2hana2, get the process ID (PID) of the hdbindexserver process. For example, you can get it from the PID column of the HDB info output as user <sid>adm:

   rh1adm$ HDB info

4. On the same instance on the secondary site, use the PID to simulate a hanging hdbindexserver process by sending the STOP signal to the process. This freezes the process and blocks it from communicating and syncing the instance between the nodes:

   rh1adm$ kill -STOP <PID>

Verification

1. On the primary site, watch the system replication status for the change on any primary instance. In the following example the system’s cut utility helps you limit the output to certain fields for readability. Remove it to see all columns of the table formatted text output. In the example we froze the indexserver on the secondary node dc2hana2, which results in a replication error with that node’s counterpart on the primary site, dc1hana2:

   rh1adm$ cdpy; watch "python systemReplicationStatus.py | cut -d '|' -f 1-3,5,9,13-"

   |Database |Host     |Service Name |Secondary |Secondary     |Replication |Replication |Replication                   |Secondary    |
   |         |         |             |Host      |Active Status |Mode        |Status      |Status Details                |Fully Synced |
   |-------- |-------- |------------ |--------- |------------- |----------- |----------- |----------------------------- |------------ |
   |RH1      |dc1hana3 |indexserver  |dc2hana3  |YES           |SYNC        |ACTIVE      |                              |        True |
   |RH1      |dc1hana2 |indexserver  |dc2hana2  |YES           |SYNC        |ERROR       |Log shipping timeout occurred |       False |
   |SYSTEMDB |dc1hana1 |nameserver   |dc2hana1  |YES           |SYNC        |ACTIVE      |                              |        True |
   |RH1      |dc1hana1 |xsengine     |dc2hana1  |YES           |SYNC        |ACTIVE      |                              |        True |
   |RH1      |dc1hana1 |indexserver  |dc2hana1  |YES           |SYNC        |ACTIVE      |                              |        True |
   
   status system replication site "2": ERROR
   overall system replication status: ERROR
   
   ...

   The replication status changes to ERROR for the indexserver service after a bit. It can take a while to react on an idle instance, wait a minute or more.

2. On the primary site’s master name server node, check the HANA nameserver process log for the related messages as the <sid>adm user:

   rh1adm$ cdtrace; grep -he 'HanaSR.srConnectionChanged.*' nameserver_*

   ha_dr_SAPHanaSR  SAPHanaSR.py(00057) : SAPHanaSR (0.184.1) SAPHanaSR.srConnectionChanged method called with
    Dict={'hostname': 'dc1hana2', 'port': '30203', 'volume': 4, 'service_name': 'indexserver', 'database': 'RH1',
    'status': 11, 'database_status': 11, 'system_status': 11, 'timestamp': '2025-12-11T11:52:01.069691+00:00',
    'is_in_sync': False, 'system_is_in_sync': False, 'reason': '', 'siteName': 'DC2'}
   ha_dr_SAPHanaSR  SAPHanaSR.py(00068) : SAPHanaSR.srConnectionChanged() CALLING CRM: <sudo /usr/sbin/crm_attribute -n hana_rh1_gsh -v 1.0 -l reboot> rc=0
   ha_dr_SAPHanaSR  SAPHanaSR.py(00069) : SAPHanaSR.srConnectionChanged() Running old srHookGeneration 1.0, see attribute hana_rh1_gsh too
   ha_dr_SAPHanaSR  SAPHanaSR.py(00087) : SAPHanaSR SAPHanaSR.srConnectionChanged method called with
    Dict={'hostname': 'dc1hana2', 'port': '30203', 'volume': 4, 'service_name': 'indexserver', 'database': 'RH1',
    'status': 11, 'database_status': 11, 'system_status': 11, 'timestamp': '2025-12-11T11:52:01.069691+00:00',
    'is_in_sync': False, 'system_is_in_sync': False, 'reason': '', 'siteName': 'DC2'} ###

   The nameserver process log contains the SAPHanaSR hook logs.

3. Verify that both cluster attributes for the system replication status, srHook and sync_state, show the SFAIL status of the secondary site. Run the following as the root user on any HANA node or use the open terminal from the previous steps to watch the changes:

   [root]# SAPHanaSR-showAttr

   Global cib-time                 prim sec srHook sync_state upd
   ---------------------------------------------------------------
   RH1    Thu Dec 11 12:24:40 2025 DC1  DC2 SFAIL  SFAIL      ok
   
   ...

4. Unblock the previously frozen hdbindexserver PID to enable it again. Run this on the secondary instance on which you blocked the hdbindexserver process for the test:

   rh1adm$ kill -CONT <PID>

5. Repeat the previous checks to verify that the system replication recovers fully after a bit. The cluster does not trigger any actions during this test since the resources remain running. Ensure that the system replication status is healthy again, fully synced and the cluster attributes are set to SOK again for the secondary site.

Procedure

1. Use a separate terminal to monitor the HANA processes as user <sid>adm on the instance on which you run this test:

   rh1adm$ watch "sapcontrol -nr ${TINSTANCE} -function GetProcessList | column -s ',' -t"

2. In another terminal on the same HANA instance, kill the hdbindexserver process:

   rh1adm$ kill <PID>

Verification

1. Check the dedicated HANA nameserver trace log on the same instance and identify the event and related action, as user <sid>adm:

   rh1adm$ cdtrace; less nameserver_chksrv.trc

   ...
   ChkSrv version 0.7.8. Method srServiceStateChanged method called.
   ChkSrv srServiceStateChanged method called with Dict={'hostname': 'dc2hana2',
    'service_name': 'indexserver', 'service_port': '30203', 'service_status': 'stopping',
    'service_previous_status': 'yes', 'timestamp': '2025-12-11T13:01:10.872386+00:00',
    'daemon_status': 'yes', 'database_id': '3', 'database_name': 'RH1', 'database_status': 'yes',
    'details': ''}
   ChkSrv srServiceStateChanged method called with SAPSYSTEMNAME=RH1
   srv:indexserver-30203-stopping-yes db:RH1-3-yes daem:yes
   LOST: indexserver event looks like a lost indexserver (status=stopping)
   LOST: stop instance. action_on_lost=stop
   
   ...

2. Check the cluster status for resource failure information on any cluster node, as user root:

   [root]# pcs status --full

   ...
   Failed Resource Actions:
     * rsc_SAPHanaCon_RH1_HDB02_monitor_61000 on dc2hana3 'not running' (7): call=183, status='complete',
   last-rc-change='Thu Dec 11 13:02:31 2025', queued=0ms, exec=0ms
     * rsc_SAPHanaCon_RH1_HDB02_monitor_61000 on dc2hana4 'not running' (7): call=179, status='complete',
   last-rc-change='Thu Dec 11 13:02:35 2025', queued=0ms, exec=0ms
     * rsc_SAPHanaCon_RH1_HDB02_monitor_61000 on dc2hana1 'not running' (7): call=108, status='complete',
   last-rc-change='Thu Dec 11 13:01:49 2025', queued=0ms, exec=0ms
     * rsc_SAPHanaCon_RH1_HDB02_monitor_61000 on dc2hana2 'error' (1): call=179, status='complete',
   last-rc-change='Thu Dec 11 13:02:39 2025', queued=0ms, exec=0ms
   
   ...

3. Check the system log for the related cluster actions on the on the test node, for example, dc2hana2, as user root:

   [root]# grep rsc_SAPHanaCon_RH1_HDB02 /var/log/messages

   ...
   Dec 11 13:01:33 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[311166]: INFO: RA ==== begin action monitor_clone (0.185.3) ====
   Dec 11 13:01:33 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[311166]: INFO: ACT: systemd service SAPRH1_02.service is active
   Dec 11 13:01:33 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[311166]: ERROR: SAP instance service hdbdaemon status color is GRAY !
   Dec 11 13:01:34 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[311166]: WARNING: RA: HANA_CALL stderr from command 'su - rh1adm -c' is '', stderr from command 'python landscapeHostConfiguration.py --sapcontrol=1' is ''
   Dec 11 13:01:34 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[311166]: INFO: DEC: local instance AND landscape are down (lss=1)
   Dec 11 13:01:34 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[311166]: INFO: DEC: DEMOTED => OCF_SUCCESS
   Dec 11 13:01:34 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[311166]: INFO: RA ==== end action monitor_clone with rc=0 (0.185.3) (3s)====
   Dec 11 13:01:49 dc2hana2 pacemaker-attrd[1977]: notice: Setting last-failure-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana1]: (unset) -> 1765458109
   Dec 11 13:01:49 dc2hana2 pacemaker-attrd[1977]: notice: Setting fail-count-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana1]: (unset) -> 1
   Dec 11 13:01:58 dc2hana2 pacemaker-attrd[1977]: notice: Setting master-rsc_SAPHanaCon_RH1_HDB02[dc2hana1]: 100 -> 0
   Dec 11 13:02:31 dc2hana2 pacemaker-attrd[1977]: notice: Setting last-failure-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana3]: (unset) -> 1765458151
   Dec 11 13:02:31 dc2hana2 pacemaker-attrd[1977]: notice: Setting fail-count-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana3]: (unset) -> 1
   Dec 11 13:02:35 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[312314]: INFO: DEC: is_master_nameserver  rc=0
   Dec 11 13:02:35 dc2hana2 pacemaker-attrd[1977]: notice: Setting last-failure-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana4]: (unset) -> 1765458155
   Dec 11 13:02:35 dc2hana2 pacemaker-attrd[1977]: notice: Setting fail-count-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana4]: (unset) -> 1
   Dec 11 13:02:36 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[312314]: INFO: RA: multiTargetSupport attribute not set. May be no Hook is configured or the old-style Hook is used.
   Dec 11 13:02:36 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[312314]: INFO: DEC: is_master_nameserver  rc=0
   Dec 11 13:02:36 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[312314]: INFO: DEC: ===> master_walk: priorities for site DC2 master1= master2=dc2hana2 master3= ==> active_master= best_cold_master=dc2hana2
   Dec 11 13:02:36 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[312314]: INFO: DEC: ===> master_walk: the_master=dc2hana2;
   Dec 11 13:02:36 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[312314]: INFO: RA ==== begin action monitor_clone (0.185.3) ====
   Dec 11 13:02:38 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[312314]: INFO: ACT: systemd service SAPRH1_02.service is active
   Dec 11 13:02:39 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[312314]: INFO: RA ==== end action monitor_clone with rc=1 (0.185.3) (4s)====
   Dec 11 13:02:39 dc2hana2 pacemaker-controld[1979]: notice: Result of monitor operation for rsc_SAPHanaCon_RH1_HDB02 on dc2hana2: error
   Dec 11 13:02:39 dc2hana2 pacemaker-controld[1979]: notice: rsc_SAPHanaCon_RH1_HDB02_monitor_61000@dc2hana2 output [ 10\n ]
   Dec 11 13:02:39 dc2hana2 pacemaker-attrd[1977]: notice: Setting last-failure-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana2]: (unset) -> 1765458159
   Dec 11 13:02:39 dc2hana2 pacemaker-attrd[1977]: notice: Setting fail-count-rsc_SAPHanaCon_RH1_HDB02#monitor_61000[dc2hana2]: (unset) -> 1
   Dec 11 13:03:34 dc2hana2 pacemaker-attrd[1977]: notice: Setting master-rsc_SAPHanaCon_RH1_HDB02[dc2hana3]: -12200 -> -22200
   Dec 11 13:03:38 dc2hana2 pacemaker-attrd[1977]: notice: Setting master-rsc_SAPHanaCon_RH1_HDB02[dc2hana4]: 80 -> -32300
   Dec 11 13:03:41 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: DEC: is_master_nameserver  rc=0
   Dec 11 13:03:41 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: RA: multiTargetSupport attribute not set. May be no Hook is configured or the old-style Hook is used.
   Dec 11 13:03:41 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: DEC: is_master_nameserver  rc=0
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: DEC: ===> master_walk: priorities for site DC2 master1=dc2hana1 master2=dc2hana2 master3=dc2hana4 ==> active_master=dc2hana1 best_cold_master=
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: DEC: ===> master_walk: the_master=dc2hana1;
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: RA ==== begin action monitor_clone (0.185.3) ====
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: ACT: systemd service SAPRH1_02.service is active
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: DEC: the_master=<<dc2hana1>>
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: RA: SRHOOK0=
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: RA: SRHOOK1=SFAIL
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: RA: SRHOOK3=SFAIL
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: DEC: scoring 2:S:master2:slave:worker:slave : SFAIL
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: DEC: scoring_crm_master: roles(2:S:master2:slave:worker:slave) are matching pattern ([0-9]:S:)
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: DEC: scoring_crm_master: sync(SFAIL) is matching syncPattern (.*)
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: DEC: scoring_crm_master: set score -32300
   Dec 11 13:03:42 dc2hana2 pacemaker-attrd[1977]: notice: Setting master-rsc_SAPHanaCon_RH1_HDB02[dc2hana2]: 80 -> -32300
   Dec 11 13:03:42 dc2hana2 SAPHanaController(rsc_SAPHanaCon_RH1_HDB02)[313705]: INFO: RA ==== end action monitor_clone with rc=0 (0.185.3) (2s)====
   
   ...

   The next SAPHanaController resource monitor reports the unexpectedly stopped HANA instances as a failure and initiates the recovery steps according to the configuration. If PREFER_SITE_TAKEOVER is enabled and you executed the test on a primary instance, it triggers a HANA takeover to the secondary site.

Procedure

1. Clean up resource failures:

   [root]# pcs resource cleanup

2. Clean up the STONITH failure history:

   [root]# pcs stonith history cleanup

Procedure

1. Set maintenance mode for the entire cluster:

   [root]# pcs property set maintenance-mode=true

   Setting maintenance for the whole cluster ensures that no activity during the maintenance phase can trigger cluster actions and impact the HANA update process.

2. Verify that the cluster resource management is fully disabled:

   [root]# pcs status

   ...
                 *** Resource management is DISABLED ***
     The cluster will not attempt to start, stop or recover services
   
   Node List:
     * Online: [ dc1hana1 dc1hana2 dc1hana3 dc1hana4 dc2hana1 dc2hana2 dc2hana3 dc2hana4 dc3mm ]
   
   Full List of Resources:
     * rsc_fence       (stonith:<fence agent>):     Started dc1hana2 (unmanaged)
     * Clone Set: cln_SAPHanaTop_RH1_HDB02 [rsc_SAPHanaTop_RH1_HDB02] (unmanaged):
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana3 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana1 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana2 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana3 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana1 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana2 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana4 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana4 (unmanaged)
       * Stopped: [ dc3mm ]
     * Clone Set: cln_SAPHanaCon_RH1_HDB02 [rsc_SAPHanaCon_RH1_HDB02] (promotable, unmanaged):
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana3 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana1 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana2 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana3 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana1 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana2 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana4 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana4 (unmanaged)
       * Stopped: [ dc3mm ]
     * rsc_vip_RH1_HDB02_primary   (ocf:heartbeat:IPaddr2):         Started dc1hana1 (unmanaged)
     * rsc_vip_RH1_HDB02_readonly  (ocf:heartbeat:IPaddr2):         Started dc2hana1 (unmanaged)
   
   ...

3. Update the HANA instances using the SAP procedure. If you have to perform a takeover during the HANA update, you can use the SAP HANA Takeover with Handshake option. For more information see also Is it possible to use SAP HANA "Takeover with Handshake" option with the HA Solutions for managing HANA System Replication?.

   If you stop the cluster in this step, ensure that you start it again before you proceed with the next steps. Keep the maintenance mode enabled.

4. After the HANA update, verify that the HANA system replication is working correctly. Use the systemReplicationStatus.py script to show the status of the HANA system replication on the primary site. Below is the example after a manual takeover to the secondary site during the maintenance:

   [root]# su - <sid>adm -c "HDBSettings.sh systemReplicationStatus.py \
   --sapcontrol=1 | grep -i replication_status="

   service/dc2hana2/30203/REPLICATION_STATUS=ACTIVE
   service/dc2hana3/30203/REPLICATION_STATUS=ACTIVE
   service/dc2hana1/30201/REPLICATION_STATUS=ACTIVE
   service/dc2hana1/30207/REPLICATION_STATUS=ACTIVE
   service/dc2hana1/30203/REPLICATION_STATUS=ACTIVE
   site/1/REPLICATION_STATUS=ACTIVE
   overall_replication_status=ACTIVE

   Before you proceed, ensure that the system replication is healthy and reported as ACTIVE.

5. Refresh all cluster resources to execute one monitor operation and update their status:

   [root]# pcs resource refresh

   Waiting for 1 reply from the controller
   ... got reply (done)

   It is crucial that the HANA resources update cluster and node attributes to reflect the new HANA system replication status. It ensures that the cluster has the correct information and does not trigger recovery actions due to incorrect status information, after the maintenance stops.

6. Check the cluster status and verify the resource status and main HANA resource score attribute. All resources must show as Started and the promotable resources must show as Unpromoted on all nodes:

   [root]# pcs status resources

    * Clone Set: cln_SAPHanaTop_RH1_HDB02 [rsc_SAPHanaTop_RH1_HDB02] (unmanaged):
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana2 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana3 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana4 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana2 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana3 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc1hana1 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana4 (unmanaged)
       * rsc_SAPHanaTop_RH1_HDB02  (ocf:heartbeat:SAPHanaTopology):         Started dc2hana1 (unmanaged)
       * Stopped: [ dc3mm ]
     * Clone Set: cln_SAPHanaCon_RH1_HDB02 [rsc_SAPHanaCon_RH1_HDB02] (promotable, unmanaged):
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana2 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana3 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc1hana4 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana2 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana3 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Promoted dc1hana1 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana4 (unmanaged)
       * rsc_SAPHanaCon_RH1_HDB02  (ocf:heartbeat:SAPHanaController):       Unpromoted dc2hana1 (unmanaged)
       * Stopped: [ dc3mm ]
     * rsc_vip_RH1_HDB02_primary   (ocf:heartbeat:IPaddr2):         Started dc1hana1 (unmanaged)
     * rsc_vip_RH1_HDB02_readonly  (ocf:heartbeat:IPaddr2):         Started dc2hana1 (unmanaged)

7. Check the cluster attributes and verify that at least the sync_state attribute is SOK:

   [root]# SAPHanaSR-showAttr

   Global cib-time                 maintenance prim sec srHook sync_state upd
   ---------------------------------------------------------------------------
   RH1    Fri Dec 19 09:41:39 2025 true        DC2  DC1 SOK    SOK        ok
   
   …

   Depending on the maintenance activity, the rest of the attribute information can be different or empty, for example, when you stopped and restarted the cluster on all nodes.

8. When the checks of the previous steps show the landscape in the expected healthy state, you can remove the maintenance mode of the cluster again:

   [root]# pcs property set maintenance-mode=

   When you lift the maintenance it triggers a monitor run of all resources again. The cluster updates the status of the promotable resources to Promoted and Unpromoted in correspondence to the location of the primary and secondary instances. The resources now also update the srPoll attribute again to match the srHook attribute value.

Procedure

1. Register the former primary site as the new secondary site. Run as user <sid>adm on one stopped former primary instance:

   rh1adm$ hdbnsutil -sr_register --remoteHost=<node> \
   --remoteInstance=${TINSTANCE} --replicationMode=sync \
   --operationMode=logreplay --name=<site>

   • Replace <node> with the new primary instance host, for example, dc2hana1 if there was a takeover from dc1hana1 to dc2hana1.
   • Replace <site> with your new secondary HANA site name, for example, DC1 if dc1hana1 is to be registered as a secondary.
   • Choose the values for replicationMode and operationMode according to your requirements for the system replication.
   • $TINSTANCE is an environment variable that is set automatically for user <sid>adm by reading the HANA instance profile. The variable value is the HANA instance number.

2. Start the secondary HANA site. Run as <sid>adm on one new secondary instance node:

   rh1adm$ sapcontrol -nr ${TINSTANCE} -function StartSystem HDB

Procedure

1. Go to the HANA trace logs directory, as the <sid>adm user:

   rh1adm$ cdtrace

2. Check for errors related to the HA/DR providers in the HANA nameserver process alert log:

   rh1adm$ grep ha_dr_provider nameserver_alert_*.trc

   ... ha_dr_provider   PythonProxyImpl.cpp(00145) : import of saphanasr failed: No module named 'saphanasr'
   ... ha_dr_provider   HADRProviderManager.cpp(00100) : could not load HA/DR Provider 'saphanasr' from /usr/share/SAPHanaSR-ScaleOut

3. Identify the root cause, for example a misspelled HA/DR provider name or a wrong path. Check the path and the hook script name. In this example the HA/DR provider name saphanasr is not matching the hook script name SAPHanaSR:

   rh1adm$ ls /usr/share/SAPHanaSR-ScaleOut/

   ChkSrv.py  SAPHanaSR.py  SAPHanaSrMultiTarget.py  samples

4. Correct the SAPHanaSR HA/DR provider configuration:

   [ha_dr_provider_SAPHanaSR]
   provider = SAPHanaSR
   path = /usr/share/SAPHanaSR-ScaleOut
   execution_order = 1

   • provider must match the name of the Python hook script. It is case-sensitive without the .py file suffix.
   • path must be the path in which the hook script is stored.

Procedure

1. Use the systemReplicationStatus.py script to check the status of the HANA system replication on the primary site:

   [root]# su - <sid>adm -c "HDBSettings.sh systemReplicationStatus.py \
   --sapcontrol=1 | grep -i replication_status="

   service/dc1hana3/30203/REPLICATION_STATUS=ACTIVE
   service/dc1hana2/30203/REPLICATION_STATUS=ACTIVE
   service/dc1hana1/30201/REPLICATION_STATUS=ACTIVE
   service/dc1hana1/30207/REPLICATION_STATUS=ACTIVE
   service/dc1hana1/30203/REPLICATION_STATUS=ACTIVE
   site/2/REPLICATION_STATUS=ACTIVE
   overall_replication_status=ACTIVE

   Before you proceed, ensure that the system replication is healthy and reported as ACTIVE.

2. Review the sync_state and srHook attributes and the node score values during the conflict:

   [root]# SAPHanaSR-showAttr

   Global cib-time                 prim sec srHook sync_state upd
   ---------------------------------------------------------------
   RH1    Fri Dec 19 11:12:42 2025 DC1  DC1 SFAIL  SOK        ok
   
   Sites lpt        lss mns      srr
   ----------------------------------
   DC1   1766142750 4   dc1hana1 P
   DC2   10         4   dc2hana1 S
   
   Hosts    clone_state gra node_state roles                         score     site
   ---------------------------------------------------------------------------------
   dc1hana1 PROMOTED    2.0 online     master1:master:worker:master  150       DC1
   dc1hana2 DEMOTED     2.0 online     master2:slave:worker:slave    140       DC1
   dc1hana3 DEMOTED     2.0 online     slave:slave:worker:slave      -10000    DC1
   dc1hana4 DEMOTED     2.0 online     master3:slave:standby:standby 140       DC1
   dc2hana1 DEMOTED     2.0 online     master1:master:worker:master  -INFINITY DC2
   dc2hana2 DEMOTED     2.0 online     master2:slave:worker:slave    -32300    DC2
   dc2hana3 DEMOTED     2.0 online     slave:slave:worker:slave      -22200    DC2
   dc2hana4 DEMOTED     2.0 online     master3:slave:standby:standby -32300    DC2
   dc3mm                    online

   In this state, the sync_state attribute is correct, but the srHook attribute takes precedence. Therefore, the secondary site is excluded from taking over if the primary site fails.

3. Delete the srHook attribute to solve the conflict:

   [root]# crm_attribute --type crm_config -n hana_<sid>_glob_srHook --delete

   Deleted crm_config option: id=SAPHanaSR-hana_rh1_glob_srHook name=hana_rh1_glob_srHook