Deploying SAP NetWeaver or S/4HANA Application Server High Availability with simple mount

Procedure

1. Create the directories for the shared filesystems:

   [root]# mkdir -p /sapmnt/ /usr/sap/trans/ /usr/sap/<SID>

   • Replace <SID> with the SID of your planned instances.

2. Add the shared NFS filesystems to /etc/fstab to mount them automatically on system start:

   [root]# vi /etc/fstab
   …
   <nfs_server>:/usr/sap/<SID> /usr/sap/<SID> nfs4 defaults 0 0
   <nfs_server>:/usr/sap/trans /usr/sap/trans nfs4 defaults 0 0
   <nfs_server>:/sapmnt /sapmnt nfs4 defaults 0 0

   • Replace <SID> with the SID of your planned instances.
   • Replace <nfs_server> with the NFS server DNS name or the IP address of each share, for example, nfs01-datacenter1a.example.com.

3. Reload the systemctl daemon to make the new /etc/fstab entries known to systemd:

   [root]# systemctl daemon-reload

4. Mount any new filesystems that you configured in the /etc/fstab:

   [root]# mount -a

5. Repeat the configuration steps on each cluster node.

Verification

1. Check that the filesystems are mounted:

   [root]# df -hP | grep sap
   nfs01-datacenter1a.example.com:/sapmnt         8.0E  1.3G  8.0E   1% /sapmnt
   nfs01-datacenter1a.example.com:/usr/sap/trans  8.0E  1.3G  8.0E   1% /usr/sap/trans
   nfs01-datacenter1a.example.com:/usr/sap/S4H    8.0E  1.3G  8.0E   1% /usr/sap/S4H

2. Check that the systemd mount targets exist for the filesystems configured in the /etc/fstab:

   [root]# systemctl list-units --all | grep -e '.*sap.*mount' | column -t
   sapmnt.mount         loaded  active  mounted  /sapmnt
   usr-sap-S4H.mount    loaded  active  mounted  /usr/sap/S4H
   usr-sap-trans.mount  loaded  active  mounted  /usr/sap/trans

3. Repeat the verification steps on each cluster node. The results must be identical.

Procedure

1. Create the sapsys group. Use the prepared group ID, for example, ID 10001:

   [root]# groupadd -g 10001 sapsys

2. Create the sapadm user as a member of the sapsys group. The user does not need a login shell. Use the prepared user ID, for example, ID 10200:

   [root]# useradd -u 10200 -g sapsys sapadm \
   -c 'SAP Local Administrator' -s /sbin/nologin

3. Create the <sid>adm user as a member of the sapsys group. Use the prepared user ID, for example, ID 10201 for user s4hadm:

   [root]# useradd -u 10201 -g sapsys s4hadm \
   -c 'SAP System Administrator' -s /bin/sh

   As the user shell, we recommend that you either use /bin/sh or /bin/csh. SAP installations provide user profiles and useful shell aliases in these shells.

4. Repeat the steps on all nodes.

Verification

1. Check that the users sapadm and <sid>adm exist and have the correct groups and IDs configured, for example:

   [root]# id sapadm s4hadm
   uid=10200(sapadm) gid=10001(sapsys) groups=10001(sapsys)
   uid=10201(s4hadm) gid=10001(sapsys) groups=10001(sapsys)

2. Check that the users have the correct description, home directory and shell defined:

   [root]# grep -E 'sapadm|s4hadm' /etc/passwd
   sapadm:x:10200:10001:SAP Local Administrator:/home/sapadm:/sbin/nologin
   s4hadm:x:10201:10001:SAP System Administrator:/home/s4hadm:/bin/sh

3. Repeat the check on all nodes and verify that the names and IDs are identical.

Procedure

1. On the node where you install the ASCS instance, go to the directory where you have extracted the installation media:

   [root]# cd <software_path>

   • Replace <software_path> with the path to your unpacked media, for example, /sapmedia/SWPM20_SP19/.

2. Run the installer command and specify the virtual hostname of your ASCS instance, for example, s4hascs:

   [root]# ./sapinst SAPINST_USE_HOSTNAME=s4hascs

3. Open the web installer UI using the link provided in the terminal.
4. Open the SAP product you want to install and enter the installation option. Expand the High-Availability System option and select ASCS Instance for the installation of the ASCS instance. Click Next.

5. Provide the requested installation information on each page and click Next to move forward.

   Some steps, like extracting SAP packages, can take a while. Keep an eye on the terminal in which you started the installer for details of the ongoing process that are not displayed in the web UI.

Verification

1. Switch to the <sid>adm user:

   [root]# su - s4hadm

2. Check the instance status. Ensure that the service status is GREEN for all service components:

   s4hadm $ sapcontrol -nr 20 -function GetProcessList

   1. The following example shows a minimal setup with ENSA1:

      name, description, dispstatus, textstatus, starttime, elapsedtime, pid
      msg_server, MessageServer, GREEN, Running, YYYY MM DD 15:34:05, 0:05:38, 45041
      enserver, EnqueueServer, GREEN, Running, YYYY MM DD 15:34:05, 0:05:38, 45042

   2. The following example shows a minimal setup with ENSA2:

      name, description, dispstatus, textstatus, starttime, elapsedtime, pid
      msg_server, MessageServer, GREEN, Running, YYYY MM DD 15:43:49, 0:01:51, 5460
      enq_server, Enqueue Server 2, GREEN, Running, YYYY MM DD 15:43:49, 0:01:51, 5461

Procedure

1. On the node where you install the ERS instance, go to the directory where you have extracted the installation media:

   [root]# cd <software_path>

   • Replace <software_path> with the path to your unpacked media, for example, /sapmedia/SWPM20_SP19/.

2. Run the installer command and specify the virtual hostname of your ERS instance, for example, s4hers:

   [root]# ./sapinst SAPINST_USE_HOSTNAME=s4hers

3. Open the web installer UI using the link provided in the terminal.
4. Open the SAP product you want to install and enter the installation option. Expand the High-Availability System option and select ERS Instance for the installation of the ERS instance. Click Next.

5. Provide the requested installation information on each page and click Next to move forward.

   Some steps, like extracting SAP packages, can take a while. Keep an eye on the terminal in which you started the installer for details of the ongoing process that are not displayed in the web UI.

Verification

1. Switch to the <sid>adm user:

   [root]# su - s4hadm

2. Check the instance status. Ensure that the service status is GREEN for all service components:

   s4hadm $ sapcontrol -nr 29 -function GetProcessList

   1. The following example, shows a minimal setup with ENSA1:

      name, description, dispstatus, textstatus, starttime, elapsedtime, pid
      enrepserver, EnqueueReplicator, GREEN, Running, YYYY MM DD 16:34:05, 0:00:26, 11484

   2. The following example, shows a minimal setup with ENSA2:

      name, description, dispstatus, textstatus, starttime, elapsedtime, pid
      enq_replicator, Enqueue Replicator 2, GREEN, Running, YYYY MM DD 11:47:34, 0:33:14, 15623

Procedure

1. On the node where you install the application server instance, go to the directory where you have extracted the installation media:

   [root]# cd <software_path>

   • Replace <software_path> with the path to your unpacked media, for example, /sapmedia/SWPM20_SP19/.

2. Run the installer command and specify the virtual hostname of your application server instance, for example, s4hpas:

   [root]# ./sapinst SAPINST_USE_HOSTNAME=s4hpas

3. Open the web installer UI using the link provided in the terminal.
4. Open the SAP product you want to install and enter the installation option. Expand the High-Availability System option and select Primary Application Server Instance for the installation of the PAS instance, or select Additional Application Server Instance for the installation of an AAS instance. Click Next.

5. Provide the requested installation information on each page and click Next to move forward.

   Some steps, like extracting SAP packages, can take a while. Keep an eye on the terminal in which you started the installer for details of the ongoing process that are not displayed in the web UI.

Verification

1. Switch to the <sid>adm user:

   [root]# su - s4hadm

2. Check the instance status. Ensure that the service status is GREEN for all service components:

   s4hadm $ sapcontrol -nr 21 -function GetProcessList
   name, description, dispstatus, textstatus, starttime, elapsedtime, pid
   disp+work, Dispatcher, GREEN, Running, YYYY MM DD 16:40:47, 68:02:08, 17973
   igswd_mt, IGS Watchdog, GREEN, Running, YYYY MM DD 16:40:47, 68:02:08, 17974
   gwrd, Gateway, GREEN, Running, YYYY MM DD 16:40:50, 68:02:05, 18326
   icman, ICM, GREEN, Running, YYYY MM DD 16:40:50, 68:02:05, 18327

Procedure

1. Check that all nodes have the SAP ports in the services file. For example, count the entries that contain SAP System in their port description and compare the result on all nodes:

   [root]# grep -i "SAP System" /etc/services | wc -l
   401

2. Update the /etc/services file on any node that is missing entries.

Procedure

1. Register the ASCS instance. Run the following SAP command as the root user to create the systemd integration:

   [root]# export LD_LIBRARY_PATH=/usr/sap/<SID>/ASCS<instance>/exe && \
   /usr/sap/<SID>/ASCS<instance>/exe/sapstartsrv \
   pf=/usr/sap/<SID>/SYS/profile/<SID>_ASCS<instance>_<ascs_virtual_hostname> \
   -reg

   The command executes the sapstartsrv service for the selected instance profile and registers the instance service on the current system. It creates the systemd unit for the instance service, if it does not exist, and updates the local /usr/sap/sapservices file.

   • Replace <SID> with your ASCS instance SID, for example, S4H.
   • Replace <instance> with your ASCS instance number, for example, 20.
   • Replace <ascs_virtual_hostname> with the virtual hostname for your ASCS instance, for example, s4hascs.

2. Register the ERS instance by repeating step 1 for the ERS profile:

   [root]# export LD_LIBRARY_PATH=/usr/sap/<SID>/ERS<instance>/exe && \
   /usr/sap/<SID>/ERS<instance>/exe/sapstartsrv \
   pf=/usr/sap/<SID>/SYS/profile/<SID>_ERS<instance>_<ers_virtual_hostname> \
   -reg

   • Replace <SID> with your ERS instance SID, for example, S4H.
   • Replace <instance> with your ERS instance number, for example, 29.
   • Replace <ers_virtual_hostname> with the virtual hostname for your ERS instance, for example, s4hers.

3. Optional: Register any PAS or AAS instance by repeating step 1 for the respective application server profile. Skip this step if you do not configure PAS or AAS instances in this cluster:

   [root]# export LD_LIBRARY_PATH=/usr/sap/<SID>/D<instance>/exe && \
   /usr/sap/<SID>/D<instance>/exe/sapstartsrv \
   pf=/usr/sap/<SID>/SYS/profile/<SID>_D<instance>_<as_virtual_hostname> \
   -reg

   • Replace <SID> with your PAS or AAS instance SID, for example, S4H.
   • Replace <instance> with your PAS or AAS instance number, for example, 21.
   • Replace <as_virtual_hostname> with the virtual hostname for your PAS or AAS instance, for example, s4hpas.

4. Disable the ASCS, ERS and any other application instance service that the cluster manages after the setup:

   [root]# systemctl disable SAP<SID>_<instance>.service
   Removed "/etc/systemd/system/multi-user.target.wants/SAP<SID>_<instance>.service".

   Run this using the ASCS instance number and repeat the command using the ERS instance number.

   Optional: Repeat the same for the PAS or AAS instance services.

5. Create the systemd drop-in directory for the ASCS, ERS and any other application instance service that the cluster manages after the setup:

   [root]# mkdir /etc/systemd/system/SAP<SID>_<instance>.service.d

   Run this using the ASCS instance number and repeat the command using the ERS instance number.

   Optional: Repeat using the PAS or AAS instance number.

6. Create the drop-in files for the instances in the new directory:

   [root]# cat << EOF > /etc/systemd/system/SAP<SID>_<instance>.service.d/HA.conf
   [Service]
   Restart=no
   EOF

   Run this using the ASCS instance number and repeat the command using the ERS instance number.

   Optional: Repeat using the PAS or AAS instance number.

7. Reload the systemd units to activate the drop-in configuration:

   [root]# systemctl daemon-reload

8. Repeat all steps on each cluster node for every instance.

Verification

1. Check that all instances have instance systemd units and that they are disabled:

   [root]# systemctl list-unit-files SAPS4H*
   UNIT FILE         STATE    PRESET
   SAPS4H_20.service disabled disabled
   SAPS4H_29.service disabled disabled

   Optional: PAS or AAS instance service files are listed as well in all of the verification steps when you have configured the application server instances.

2. Check that the sapservices file contains entries for every instance:

   [root]# cat /usr/sap/sapservices
   systemctl --no-ask-password start SAPS4H_20 # sapstartsrv pf=/sapmnt/S4H/profile/S4H_ASCS20_s4hascs
   systemctl --no-ask-password start SAPS4H_29 # sapstartsrv pf=/sapmnt/S4H/profile/S4H_ERS29_s4hers

3. Check that all systemd configuration overrides are present:

   [root]# systemd-delta | grep SAP
   ...
   [EXTENDED]   /etc/systemd/system/SAPS4H_20.service → /etc/systemd/system/SAPS4H_20.service.d/HA.conf
   [EXTENDED]   /etc/systemd/system/SAPS4H_29.service → /etc/systemd/system/SAPS4H_29.service.d/HA.conf

4. Repeat the steps on every cluster node. The results must be the same on all nodes.

Procedure

1. Stop the instance on the node where it currently runs, for example, stop ASCS on node1:

   [root]# systemctl stop SAP<SID>_<instance>.service

   Stopping the SAP instance systemd service stops both the instance service and the instance itself.

2. Start the instance service sapstartsrv on the node where it was not running before, for example, start the ASCS service on node2:

   [root]# systemctl start SAP<SID>_<instance>.service

3. Start the instance using sapcontrol as the <sid>adm user. Replace <instance> with the instance number, for example, 20 for the ASCS instance:

   <sid>adm $ sapcontrol -nr <instance> -function Start

4. Verify the instance’s health after it starts on the other node. Run this check as the <sid>adm user. Replace <instance> with the instance number, for example, 20 for the ASCS instance. The status of all services must be GREEN:

   <sid>adm $ sapcontrol -nr <instance> -function GetProcessList

5. Optional: Repeat the previous steps to move the instance back to its original node, for example, move ASCS back to node1.
6. Repeat all steps for the ERS instance.
7. Optional: If you configure PAS or AAS instances, then repeat the steps for these application server instances as well.

Procedure

1. Install the resource-agents-sap package on all cluster nodes:

   [root]# dnf install resource-agents-sap

2. Enable the sapping and sappong services.

   [root]# systemctl enable sapping.service sappong.service
   Created symlink /etc/systemd/system/multi-user.target.wants/sapping.service → /usr/lib/systemd/system/sapping.service.
   Created symlink /etc/systemd/system/multi-user.target.wants/sappong.service → /usr/lib/systemd/system/sappong.service.

Verification

1. Check on all nodes that the package is installed:

   [root]# rpm -q resource-agents-sap
   resource-agents-sap-<version>.<release>.noarch

2. Verify that the sapping and sappong services are enabled:

   [root]# systemctl list-unit-files sapping.service sappong.service
   UNIT FILE       STATE   PRESET
   sapping.service enabled disabled
   sappong.service enabled disabled

Procedure

1. Use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running. Adjust the parameters according to the resource agent you are using. Create the cluster resource for the ASCS virtual IP, for example, using the IPaddr2 agent:

   [root]# pcs resource create rsc_vip_<SID>_ASCS<instance> \
   ocf:heartbeat:IPaddr2 \
   ip=<address> cidr_netmask=<netmask> nic=<device> \
   --group grp_<SID>_ASCS<instance>

   • Replace <SID> with your ASCS SID.
   • Replace <instance> with your ASCS instance number.
   • Replace <address>, <netmask> and <device> with the details of your virtual IP address.

2. Create the SAPStartSrv resource for the ASCS instance:

   [root]# pcs resource create rsc_SAPStartSrv_<SID>_ASCS<instance> \
   ocf:heartbeat:SAPStartSrv \
   InstanceName="<sap_instance_name>" \
   --group grp_<SID>_ASCS<instance> \
   op monitor interval=0 timeout=20 enabled=0

   • Replace <SID> with your ASCS SID, for example, S4H.
   • Replace <instance> with your ASCS instance number, for example, 20.
   • Replace <sap_instance_name> with the SAP start profile name of your ASCS instance, for example, S4H_ASCS_s4hascs.
   • Ensure that the recurring monitor operation is disabled using enabled=0. The single resource probe still runs at resource start, but recurring monitors must not run afterwards.

3. Create the SAPInstance resource for the ASCS instance:

   [root]# pcs resource create rsc_SAPInstance_<SID>_ASCS<instance> \
   ocf:heartbeat:SAPInstance \
   InstanceName="<sap_instance_name>" \
   MINIMAL_PROBE=true \
   meta resource-stickiness=5000 \
   --group grp_<SID>_ASCS<instance> \
   op monitor interval=20 on-fail=restart timeout=60

   • Replace <sap_instance_name> with the SAP start profile name of your ASCS instance, for example, S4H_ASCS_s4hascs.

     resource-stickiness=5000 balances out the failover constraint with the ERS resource so that the resource stays on the node where it started. When you use the SAPStartSrv resource for the simplified filesystem setup, then you must set MINIMAL_PROBE=true.

4. Optional: For an ENSA1 setup you must add migration-threshold=1 to the SAPInstance resource of the ASCS instance:

   [root]# pcs resource update rsc_SAPInstance_<SID>_ASCS<instance> \
   meta migration-threshold=1

   migration-threshold=1 ensures that the ASCS instance can not restart on the same node. In ENSA2 setups restarting the ASCS instance on the same node is allowed due to an improved lock table recovery mechanism through the network.

5. Add a resource-stickiness to the ASCS resource group to ensure that the ASCS instance stays on a HA cluster node if possible:

   [root]# pcs resource meta grp_<SID>_ASCS<instance> resource-stickiness=3000

Verification

1. Check the cluster status of the ASCS resources:

   [root]# pcs status --full | grep ASCS
     * Resource Group: grp_S4H_ASCS20:
       * rsc_vip_S4H_ASCS20        (ocf:heartbeat:IPaddr2):         Started node1
       * rsc_SAPStartSrv_S4H_ASCS20        (ocf:heartbeat:SAPStartSrv):     Started node1
       * rsc_SAPInstance_S4H_ASCS20        (ocf:heartbeat:SAPInstance):     Started node1

2. Verify the resource configuration details of all resources in the ASCS group:

   [root]# pcs resource config grp_S4H_ASCS20
   Group: grp_S4H_ASCS20
     Meta Attributes: grp_S4H_ASCS20-meta_attributes
       resource-stickiness=3000
     Resource: rsc_vip_S4H_ASCS20 (class=ocf provider=heartbeat type=IPaddr2)
       Attributes: rsc_vip_S4H_ASCS20-instance_attributes
         cidr_netmask=32
         ip=192.168.200.101
         nic=eth0
       Operations:
   ...
     Resource: rsc_SAPStartSrv_S4H_ASCS20 (class=ocf provider=heartbeat type=SAPStartSrv)
       Attributes: rsc_SAPStartSrv_S4H_ASCS20-instance_attributes
         InstanceName=S4H_ASCS20_s4hascs
       Operations:
         monitor: rsc_SAPStartSrv_S4H_ASCS20-monitor-interval-0s
           interval=0s timeout=20s enabled=0
   ...
     Resource: rsc_SAPInstance_S4H_ASCS20 (class=ocf provider=heartbeat type=SAPInstance)
       Attributes: rsc_SAPInstance_S4H_ASCS20-instance_attributes
         InstanceName=S4H_ASCS20_s4hascs
         MINIMAL_PROBE=true
       Meta Attributes: rsc_SAPInstance_S4H_ASCS20-meta_attributes
         resource-stickiness=5000
       Operations:
   ...
         monitor: rsc_SAPInstance_S4H_ASCS20-monitor-interval-20
           interval=20 timeout=60 on-fail=restart
   ...

Procedure

1. Use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running. Adjust the parameters according to the resource agent you are using. Create the cluster resource for the ERS virtual IP, for example, using the IPaddr2 agent:

   [root]# pcs resource create rsc_vip_<SID>_ERS<instance> \
   ocf:heartbeat:IPaddr2 \
   ip=<address> cidr_netmask=<netmask> nic=<device> \
   --group grp_<SID>_ERS<instance>

   • Replace <SID> with your ERS SID.
   • Replace <instance> with your ERS instance number.
   • Replace <address>, <netmask> and <device> with the details of your virtual IP address.

2. Create the SAPStartSrv resource for the ERS instance:

   [root]# pcs resource create rsc_SAPStartSrv_<SID>_ERS<instance> \
   ocf:heartbeat:SAPStartSrv \
   InstanceName="<sap_instance_name>" \
   --group grp_<SID>_ERS<instance> \
   op monitor interval=0 timeout=20 enabled=0

   • Replace <SID> with your ERS SID, for example, S4H.
   • Replace <instance> with your ERS instance number, for example, 29.
   • Replace <sap_instance_name> with the SAP start profile name of your ERS instance, for example, S4H_ERS_s4hers.
   • Ensure that the recurring monitor operation is disabled using enabled=0. The single resource probe is still run at resource start, but recurring monitors must not run afterwards.

3. Create the SAPInstance resource for the ERS instance:

   [root]# pcs resource create rsc_SAPInstance_<SID>_ERS<instance> \
   ocf:heartbeat:SAPInstance \
   InstanceName="<sap_instance_name>" \
   IS_ERS=true \
   --group grp_<SID>_ERS<instance> \
   op monitor interval=20 on-fail=restart timeout=60 \
   op start interval=0 timeout=600 \
   op stop interval=0 timeout=600

Verification

1. Check the cluster status of the ERS resources:

   [root]# pcs status --full | grep ERS
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29        (ocf:heartbeat:IPaddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ERS29        (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ERS29        (ocf:heartbeat:SAPInstance):     Started node2

2. Verify the resource configuration details of all resources in the ERS group:

   [root]# pcs resource config grp_S4H_ERS29
   Group: grp_S4H_ERS29
     Resource: rsc_vip_S4H_ERS29 (class=ocf provider=heartbeat type=IPaddr2)
       Attributes: rsc_vip_S4H_ERS29-instance_attributes
         cidr_netmask=32
         ip=192.168.200.102
         nic=eth0
   …
     Resource: rsc_SAPStartSrv_S4H_ERS29 (class=ocf provider=heartbeat type=SAPStartSrv)
       Attributes: rsc_SAPStartSrv_S4H_ERS29-instance_attributes
         InstanceName=S4H_ERS29_s4hers
       Operations:
         monitor: rsc_SAPStartSrv_S4H_ERS29-monitor-interval-0
           interval=0 timeout=20 enabled=0
   …
     Resource: rsc_SAPInstance_S4H_ERS29 (class=ocf provider=heartbeat type=SAPInstance)
       Attributes: rsc_SAPInstance_S4H_ERS29-instance_attributes
         IS_ERS=true
         InstanceName=S4H_ERS29_s4hers
       Operations:
   …
         monitor: rsc_SAPInstance_S4H_ERS29-monitor-interval-20
           interval=20 timeout=60 on-fail=restart
   …
         start: rsc_SAPInstance_S4H_ERS29-start-interval-0
           interval=0 timeout=600
         stop: rsc_SAPInstance_S4H_ERS29-stop-interval-0
           interval=0 timeout=600
   …

Procedure

1. Create the colocation constraint for the ASCS and ERS resource groups. The order of the groups in the command matters:

   [root]# pcs constraint colocation add grp_<SID>_ERS<ers_instance> \
   with grp_<SID>_ASCS<ascs_instance> -5000

   • Replace <SID> with your ASCS and ERS SID, for example, S4H.
   • Replace <ers_instance> with your ERS instance number, for example, 29.
   • Replace <ascs_instance> with your ASCS instance number, for example, 20.

2. Optional: When using ENSA1, you must ensure that the ASCS instance starts on the node where the ERS instance is running. This is required to recover the lock table from ERS into ASCS. Create a location constraint rule for the ASCS resource:

   [root]# pcs constraint location grp_<SID>_ASCS<instance> rule score=2000 runs_ers_<SID> eq 1

   • Replace <SID> with your ASCS and ERS SID, for example, S4H.

   • Replace <instance> with your ASCS instance number, for example, 20.

     runs_ers_<SID> is a cluster node attribute which the SAPInstance resource agent creates. It can be used to locate where the ERS instance resource is running in the cluster.

3. Create the order constraint for the ASCS and ERS resource groups. This ensures that the ERS instance is not moved before ASCS has fully started and imported the enqueue data from the ERS instance:

   [root]# pcs constraint order start grp_<SID>_ASCS<ascs_instance> \
   then stop grp_<SID>_ERS<ers_instance> symmetrical=false kind=Optional

   • Replace <SID> with your ASCS and ERS SID, for example, S4H.
   • Replace <ers_instance> with your ERS instance number, for example, 29.
   • Replace <ascs_instance> with your ASCS instance number, for example, 20.

Procedure

1. Option 1: Create the filesystem resource for the DB instance when the filesystem is an NFS share:

   [root]# pcs resource create rsc_fs_<SID>_db \
   ocf:heartbeat:Filesystem \
   device='<nfs_server>:<db_nfs_share>' \
   directory=<db_mountpoint> \
   fstype=nfs \
   force_unmount=safe \
   --group grp_<SID>_db

   • Replace <SID> with your database SID, for example, RH1.
   • Replace <nfs_server> with the NFS server name or IP of the DB instance share, for example, nfs01-datacenter1a.example.com.
   • Replace <db_nfs_share> with the NFS volume name for your DB instance.
   • Add the option force_unmount=safe when the NFS share is a directory on a shared tree, like it is common on Azure NetApp Files (ANF) or Amazon EFS.

2. Option 2: Create the filesystem resource for the DB instance when the filesystem is on SAN storage. Use HA-LVM to manage the filesystem you set up on SAN. Apply the configuration according to What is a Highly Available LVM (HA-LVM) configuration and how do I implement it?.

   • Create the HA-LVM cluster resource:

     [root]# pcs resource create rsc_lvm_<SID>_db \
     ocf:heartbeat:LVM-activate \
     volgrpname='<db_volume_group>' \
     vg_access_mode=system_id \
     --group grp_<SID>_db

   • Replace <db_volume_group> with the LVM group name of your DB volume, for example, vg_db.

   • Create the filesystem resource:

     [root]# pcs resource create rsc_fs_<SID>_db \
     ocf:heartbeat:Filesystem \
     device='<lvm_volume>' \
     directory=<db_mountpoint> \
     fstype=<fs_type> \
     force_unmount=safe \
     --group grp_<SID>_db

   • Replace <SID> with your database SID, for example, RH1.
   • Replace <lvm_volume> with the path to the LVM share of the DB instance, for example /dev/vg_db/lv_db.
   • Replace <db_mountpoint> with the path to the mountpoint of the DB instance filesystem, for example, /sybase.
   • Replace <fs_type> with the type of filesystem you configured for the DB instance filesystem, for example, xfs.

3. Create the virtual IP resource for the database. Use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running. Adjust the parameters according to the resource agent you are using. For example, use the IPaddr2 agent:

   [root]# pcs resource create rsc_vip_<SID>_db \
   ocf:heartbeat:IPaddr2 \
   ip=<address> cidr_netmask=<netmask> nic=<device> \
   --group grp_<SID>_db

   • Replace <SID> with your database SID, for example, RH1.
   • Replace <address>, <netmask> and <device> with the details of your virtual IP address.

4. Create the SAPDatabase resource for the DB instance:

   [root]# pcs resource create rsc_SAPDatabase_<SID>_db \
   ocf:heartbeat:SAPDatabase \
   DBTYPE="<db_type>" \
   SID="<SID>" \
   STRICT_MONITORING="TRUE" \
   AUTOMATIC_RECOVER="TRUE" \
   --group grp_<SID>_db

   • Replace <SID> with your database SID, for example, RH1.
   • Replace <db_type> with the type of your database. It must be one of ADA, DB6, ORA or SYB.

Verification

1. Check the cluster status of the database resources:

   [root]# pcs status --full | grep RH1
    * Resource Group: grp_RH1_db:
      * rsc_lvm_RH1_db	(ocf:heartbeat:LVM-activate):	 Started node1
      * rsc_fs_RH1_db	(ocf:heartbeat:Filesystem):	 Started node1
      * rsc_vip_RH1_db	(ocf:heartbeat:IPaddr2):	 Started node1
      * rsc_SAPDatabase_RH1_db	(ocf:heartbeat:SAPDatabase):	 Started node1

2. Verify the resource configuration details of all resources in the database resource group:

   [root]# pcs resource config grp_RH1_db
   Group: grp_RH1_db
    Resource: rsc_lvm_RH1_db (class=ocf provider=heartbeat type=LVM-activate)
      Attributes: vg_access_mode=system_id vgname=vg_db
      Operations: monitor interval=30s timeout=90s (rsc_lvm_RH1_db-monitor-interval-30s)
                  start interval=0s timeout=90s (rsc_lvm_RH1_db-start-interval-0s)
                  stop interval=0s timeout=90s (rsc_lvm_RH1_db-stop-interval-0s)
     Resource: rsc_fs_RH1_db (class=ocf provider=heartbeat type=Filesystem)
      Attributes: device=/dev/vg_db/lv_db directory=/sybase fstype=xfs
      Operations: monitor interval=20s timeout=40s (rsc_fs_RH1_db-monitor-interval-20s)
                  start interval=0s timeout=60s (rsc_fs_RH1_db-start-interval-0s)
                  stop interval=0s timeout=60s (rsc_fs_RH1_db-stop-interval-0s)
     Resource: rsc_vip_RH1_db (class=ocf provider=heartbeat type=IPaddr2)
      Attributes: ip=192.168.200.115
      Operations: monitor interval=10s timeout=20s (rsc_vip_RH1_db-monitor-interval-10s)
                  start interval=0s timeout=20s (rsc_vip_RH1_db-start-interval-0s)
                  stop interval=0s timeout=20s (rsc_vip_RH1_db-stop-interval-0s)
     Resource: rsc_SAPDatabase_RH1_db (class=ocf provider=heartbeat type=SAPDatabase)
      Attributes: AUTOMATIC_RECOVER=TRUE DBTYPE=SYB SID=RH1 STRICT_MONITORING=TRUE
      Operations: methods interval=0s timeout=5s (rsc_SAPDatabase_RH1_db-methods-interval-0s)
                  monitor interval=120s timeout=60s (rsc_SAPDatabase_RH1_db-monitor-interval-120s)
                  start interval=0s timeout=1800s (rsc_SAPDatabase_RH1_db-start-interval-0s)
                  stop interval=0s timeout=1800s (rsc_SAPDatabase_RH1_db-stop-interval-0s)

Procedure

1. Use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running. Adjust the parameters according to the resource agent you are using. Create the cluster resource for the PAS or AAS virtual IP, for example, using the IPaddr2 agent:

   [root]# pcs resource create rsc_vip_<SID>_D<instance> \
   ocf:heartbeat:IPaddr2 \
   ip=<address> cidr_netmask=<netmask> nic=<device> \
   --group grp_<SID>_D<instance>

   • Replace <SID> with your PAS or AAS SID, for example, S4H.
   • Replace <instance> with your PAS or AAS instance number, for example, 21 for PAS.
   • Replace <address>, <netmask> and <device> with the details of your virtual IP address.

2. Create the SAPStartSrv resource for the PAS or AAS instance for the simplified filesystem management:

   [root]# pcs resource create rsc_SAPStartSrv_<SID>_D<instance> \
   ocf:heartbeat:SAPStartSrv \
   InstanceName="<sap_instance_name>" \
   --group grp_<SID>_D<instance> \
   op monitor interval=0 timeout=20 enabled=0

   • Replace <SID> with your PAS or AAS SID, for example, S4H.
   • Replace <instance> with your PAS or AAS instance number, for example, 21 for PAS.
   • Replace <sap_instance_name> with the SAP start profile name of your PAS or AAS instance, for example, S4H_PAS_s4hpas for the PAS instance.
   • Ensure that the recurring monitor operation is disabled using enabled=0. The single resource probe is still run at resource start.

3. Create the SAPInstance resource for the PAS or AAS instance:

   [root]# pcs resource create rsc_SAPInstance_<SID>_D<instance> \
   ocf:heartbeat:SAPInstance \
   InstanceName="<sap_instance_name>" \
   MINIMAL_PROBE=true \
   --group grp_<SID>_D<instance> \
   op monitor interval=20 on-fail=restart timeout=60

   • Replace <sap_instance_name> with the SAP start profile name of your PAS or AAS instance, for example, S4H_PAS_s4hpas for the PAS instance.

Verification

1. Check the cluster status of the application resources:

   [root]# pcs status --full | grep S4H_D
    * Resource Group: grp_S4H_D21:
       * rsc_vip_S4H_D21	(ocf:heartbeat:IPaddr2):	 Started node1
       * rsc_SAPInstance_S4H_D21	(ocf:heartbeat:SAPInstance):	 Started node1

2. Verify the resource configuration details of all resources in the PAS group:

   [root]# pcs resource config grp_S4H_D21
   Group: grp_S4H_D21
     Meta Attributes: grp_S4H_D21-meta_attributes
     Resource: rsc_vip_S4H_D21 (class=ocf provider=heartbeat type=IPaddr2)
       Attributes: rsc_vip_S4H_D21-instance_attributes
         cidr_netmask=32
         ip=192.168.200.103
         nic=eth0
       Operations:
   ...
     Resource: rsc_SAPStartSrv_S4H_D21 (class=ocf provider=heartbeat type=SAPStartSrv)
       Attributes: rsc_SAPStartSrv_S4H_D21-instance_attributes
         InstanceName=S4H_PAS_s4hpas
       Operations:
         monitor: rsc_SAPStartSrv_S4H_D21-monitor-interval-0s
           interval=0s timeout=20s enabled=0
   ...
     Resource: rsc_SAPInstance_S4H_D21 (class=ocf provider=heartbeat type=SAPInstance)
       Attributes: rsc_SAPInstance_S4H_D21-instance_attributes
         InstanceName=S4H_PAS_s4hpas
         MINIMAL_PROBE=true
       Meta Attributes: rsc_SAPInstance_S4H_D21-meta_attributes
       Operations:
   ...
         monitor: rsc_SAPInstance_S4H_D21-monitor-interval-20
           interval=20 timeout=60 on-fail=restart
   ...

Procedure

1. Create the order constraint to start the PAS or AAS resource group only after the ASCS group is running:

   [root]# pcs constraint order start grp_<SID>_ASCS<ascs_instance> \
   then grp_<SID>_D<app_instance> kind=Optional symmetrical=false

   • Replace <SID> with your PAS or AAS SID, for example, S4H.
   • Replace <SID> with your S/4HANA environment SID, for example, S4H.
   • Replace <ascs_instance> with your ASCS instance number, for example, 20.
   • Replace <app_instance> with your PAS or AAS instance number, for example, 21 for PAS.

2. Optional: When you have configured the PAS and the AAS instances in the same cluster, create the colocation constraint to ensure that the PAS and the AAS instances avoid running on the same cluster node, when possible:

   [root]# pcs constraint colocation add grp_<SID>_D<pas_instance> \
   with grp_<SID>_D<aas_instance> score=-1000

   • Replace <SID> with your PAS or AAS SID, for example, S4H.
   • Replace <SID> with your PAS/AAS SID, for example, S4H.
   • Replace <pas_instance> with your PAS instance number, for example, 21.
   • Replace <aas_instance> with your AAS instance number, for example, 22.

3. Optional: When you have configured the HANA database in the same cluster, create a constraint to ensure that the HANA instance resource is promoted first:

   [root]# pcs constraint order promote cln_SAPHanaCon_<db_SID>_HDB<db_instance> \
   then grp_<app_SID>_D<app_instance> kind=Optional symmetrical=false

   • Replace <db_SID> with your database SID, for example, RH1.
   • Replace <db_instance> with your HANA instance number, for example, 02.
   • Replace <app_SID> with your NetWeaver PAS or AAS SID.
   • Replace <app_instance> with your PAS or AAS instance number.

4. Optional: When you have configured a resource group for managing a legacy database, create resource constraints to ensure that the database resources are started before the PAS instance:

   [root]# pcs constraint order grp_<db_SID>_db \
   then grp_<app_SID>_D<instance> kind=Optional symmetrical=false

   • Replace <db_SID> with your database SID, for example, RH1.
   • Replace <app_SID> with your NetWeaver PAS or AAS SID, for example, S4H.
   • Replace <instance> with your PAS or AAS instance number, for example, 21.

5. Optional: If you have configured PAS and AAS instances in the same cluster, ensure that these application server instances run on different cluster nodes for resiliency. Create a colocation constraint with a negative score so that the instances avoid running on the same node when possible:

   [root]# pcs constraint colocation add grp_S4H_D22 \
   with grp_S4H_D21 score=-1000

   Use a score of -1000 to allow the instances to run together if there is only one cluster node available. If you prefer to keep the AAS instance down during this situation, you can use score=-INFINITY, which prevents that AAS can run on the same node even when only one node is available.

Procedure

1. Install the sap-cluster-connector package on all cluster nodes:

   [root]# dnf install sap-cluster-connector

2. Add the SAP administrative user <sid>adm to the haclient group to allow the SAP user to run cluster commands:

   [root]# usermod -a -G haclient <sid>adm

3. Add the service/halib configuration to the profiles of all application instances that are managed by the cluster:

   [root]# vi /sapmnt/<SID>/profile/<profile_name>
   …
   service/halib = $(DIR_EXECUTABLE)/saphascriptco.so
   service/halib_cluster_connector = /usr/bin/sap_cluster_connector

   • Replace <SID> with your instance SID, for example, S4H.
   • Replace <profile_name> with each instance profile, for example, S4H_ASCS_s4hascs for the ASCS instance.

4. Restart the sapstartsrv process of all SAP instances of which you updated the instance profile in the previous step:

   [root]# su - <sid>adm -c "sapcontrol -nr <instance> -function RestartService"

   • Replace <sid> with your lower-case instance SID, for example, s4h.
   • Replace <instance> with the instance number, for example, 20 for the ASCS instance.
5. Repeat steps 3 and 4 for every ASCS, ERS and application server instance that the cluster manages.

Verification

1. Verify that sap_cluster_connector is loaded. You find the information in the sapstartsrv.log file on the node where the instance is running:

   [root]# grep -E "cluster_connector|HA_GetVersion" /usr/sap/S4H/ASCS20/work/sapstartsrv.log
   SAP HA Trace: profile_params: setting cluster_connector = "/usr/bin/sap_cluster_connector"
   SAP HA Trace: Fire system command /usr/bin/sap_cluster_connector init ...
   SAP HA Trace: === SAP_HA_GetVersionInfo ===
   SAP HA Trace: Fire system command /usr/bin/sap_cluster_connector gvi ...
   SAP HA Trace: SAP_HA_GetVersionInfo HA interface version: 3
   SAP HA Trace: SAP_HA_GetVersionInfo HAproduct: Pacemaker
   SAP HA Trace: SAP_HA_GetVersionInfo SAPinterface: sap_cluster_connector
   SAP HA Trace: SAP_HA_GetVersionInfo documentation: https://github.com/ClusterLabs/sap_cluster_connector
   SAP HA Trace: --- SAP_HA_GetVersionInfo Exit-Code: SAP_HA_OK ---

2. Run the SAP HA interface check on the node where the instance is running. Verify that each line returns a SUCCESS state, for example:

   [root]# su - s4hadm -c "sapcontrol -nr 20 -function HACheckConfig"
   state, category, description, comment
   SUCCESS, SAP CONFIGURATION, Redundant ABAP instance configuration, 0 ABAP instances detected
   SUCCESS, SAP CONFIGURATION, Enqueue separation, All Enqueue server separated from application server
   SUCCESS, SAP CONFIGURATION, MessageServer separation, All MessageServer separated from application server
   SUCCESS, SAP STATE, SCS instance running, SCS instance status ok
   SUCCESS, SAP CONFIGURATION, SAPInstance RA sufficient version (s4hascs_S4H_20), SAPInstance includes is-ers patch
   SUCCESS, SAP CONFIGURATION, Enqueue replication (s4hascs_S4H_20), Enqueue replication enabled
   SUCCESS, SAP STATE, Enqueue replication state (s4hascs_S4H_20), Enqueue replication active
   SUCCESS, SAP CONFIGURATION, SAPInstance RA sufficient version (s4hers_S4H_29), SAPInstance includes is-ers patch

3. Repeat steps 1-2 for all instances for which you configured the HA interface.

Verification

1. Check that the resource group starts fully on the other node, for example, after moving the ASCS group from node1 to node2:

   [root]# pcs resource
     * Resource Group: grp_S4H_ASCS20:
       * rsc_vip_S4H_ASCS20        (ocf:heartbeat:IPAddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ASCS20        (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ASCS20        (ocf:heartbeat:SAPInstance):     Started node2
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29 (ocf:heartbeat:IPAddr2):         Started node1
       * rsc_SAPStartSrv_S4H_ERS29 (ocf:heartbeat:SAPStartSrv):     Started node1
       * rsc_SAPInstance_S4H_ERS29 (ocf:heartbeat:SAPInstance):     Started node1

2. Optionally, when the cluster consists of 2 nodes: Verify that the ASCS resource group fully starts, then the ERS resource group automatically stops on this node and then moves to the node where the ASCS resource group was running before. This is triggered by the colocation constraint. Check for the related chain of actions by the pacemaker-controld in the system logfile:

   [root]# less /var/log/messages
   …
   … notice: Result of start operation for rsc_SAPInstance_S4H_ASCS20 on node2: ok
   …
   … notice: Requesting local execution of stop operation for rsc_SAPInstance_S4H_ERS29 on node2

Procedure

1. Run the HAFailoverToNode function of sapcontrol to move the ASCS instance to the other node. Execute as user <sid>adm:

   <sid>adm $ sapcontrol -nr <instance> -function HAFailoverToNode ""

2. Check that the instance stops on the current node and starts on the other node. The ERS instance automatically stops and starts as well after ASCS is fully up. For example, cluster resource status after you have moved the ASCS instance from node1 to node2:

   [root]# pcs resource
     * Resource Group: grp_S4H_ASCS20:
       * rsc_vip_S4H_ASCS20        (ocf:heartbeat:IPAddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ASCS20        (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ASCS20        (ocf:heartbeat:SAPInstance):     Started node2
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29 (ocf:heartbeat:IPAddr2):         Started node1
       * rsc_SAPStartSrv_S4H_ERS29 (ocf:heartbeat:SAPStartSrv):     Started node1
       * rsc_SAPInstance_S4H_ERS29 (ocf:heartbeat:SAPInstance):     Started node1

3. Check the new location constraint that has been created by the manual move due to the HA integration:

   [root]# pcs constraint
   Location Constraints:
     Started resource 'grp_S4H_ASCS20'
       Rules:
         Rule: boolean-op=and score=-INFINITY
           Expression: #uname eq string node1
           Expression: date lt YYYY-MM-DDT13:40:45Z

   The constraint defines that the ASCS resource group is banned for 5 minutes from the original node, which enforces the move to the other node. The date string in the rule defines the time at which the cluster deletes the constraint automatically.

4. Optional: Remove the temporary constraint to enable the ASCS resource group on the previous node immediately and end this test:

   [root]# pcs resource clear grp_S4H_ASCS20

Procedure

1. Identify the process ID (PID) of the enqueue server on the node where ASCS is running. Use sapcontrol… GetProcessList as user <sid>adm to find the PID as the last entry:

   <sid>adm $ sapcontrol -nr <instance> -function GetProcessList
   name, description, dispstatus, textstatus, starttime, elapsedtime, pid
   msg_server, MessageServer, GREEN, Running, YYYY MM DD 14:10:29, 0:01:00, 142607
   enq_server, Enqueue Server 2, GREEN, Running, YYYY MM DD 14:10:29, 0:01:00, 142608

   In the example, the enqueue server PID is 142608.

2. Send a SIGKILL signal to the identified process to kill it instantly:

   <sid>adm $ kill -9 <pid>

   • Replace <pid> with the PID of the enqueue server, for example, 142608.
3. Check that the ASCS instance recovers. With the default migration-threshold of all resources set to 3, the cluster restarts the resource on the same node twice, before it recovers on a different node.
4. Repeat steps 1 and 2 until you have killed the process 3 times. The cluster recovers the resource on another node after 3 consecutive failures of the same resource.

Verification

1. Check that the ASCS instance is running on the other node:

   [root]# pcs resource
     * Resource Group: grp_S4H_ASCS20:
       * rsc_vip_S4H_ASCS20        (ocf:heartbeat:IPAddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ASCS20        (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ASCS20        (ocf:heartbeat:SAPInstance):     Started node2
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29 (ocf:heartbeat:IPAddr2):         Started node1
       * rsc_SAPStartSrv_S4H_ERS29 (ocf:heartbeat:SAPStartSrv):     Started node1
       * rsc_SAPInstance_S4H_ERS29 (ocf:heartbeat:SAPInstance):     Started node1

   In our example, the ASCS instance failed on node1 and has been moved to node2. As configured, the ERS instance moved to node1 after ASCS was up on node2.

2. Check the fail count and notice that it has failed 3 times to trigger the recovery on the other node:

   [root]# pcs resource failcount
   Failcounts for resource 'rsc_SAPInstance_S4H_ASCS20'
     node1: 3

Procedure

1. Identify the process ID (PID) of the enqueue server on the node where ERS is running. Use sapcontrol… GetProcessList as user <sid>adm to find the PID as the last entry:

   <sid>adm $ sapcontrol -nr <instance> -function GetProcessList
   name, description, dispstatus, textstatus, starttime, elapsedtime, pid
   enq_replicator, Enqueue Replicator 2, GREEN, Running, YYYY MM DD 15:42:03, 0:00:04, 19124

   In the example, the enqueue server PID is 191240.

2. Send a SIGKILL signal to the identified process to kill it instantly:

   <sid>adm $ kill -9 <pid>

   • Replace <pid> with the PID of the enqueue server, for example, 191240.
3. Check that the ERS instance recovers. With the default migration-threshold of all resources set to 3, the cluster restarts the resource on the same node twice, before it recovers on a different node.
4. Repeat steps 1 and 2 until you have killed the process 3 times. The cluster recovers the resource on another node after 3 consecutive failures of the same resource.

Verification

1. Check that the ERS instance is running on the other node:

   [root]# pcs resource
     * Resource Group: grp_S4H_ERS20:
       * rsc_vip_S4H_ERS20        (ocf:heartbeat:IPAddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ERS20        (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ERS20        (ocf:heartbeat:SAPInstance):     Started node2
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29 (ocf:heartbeat:IPAddr2):         Started node1
       * rsc_SAPStartSrv_S4H_ERS29 (ocf:heartbeat:SAPStartSrv):     Started node1
       * rsc_SAPInstance_S4H_ERS29 (ocf:heartbeat:SAPInstance):     Started node1

   In our example, the ERS instance failed on node2 and has been moved to node1. When the ERS instance cannot run on a different node than ASCS anymore due to the failures, it is restarted on the same node as the ASCS instance.

   When you clear the failure, the cluster automatically moves the ERS instance back to the other node.

2. Check the fail count and notice that it has failed 3 times to trigger the recovery on the other node:

   [root]# pcs resource failcount
   Failcounts for resource 'rsc_SAPInstance_S4H_ERS20'
     node1: 3

Verification

1. Check that the cluster on the other node fences the crashed node:

   [root]# pcs stonith history
   reboot of node1 successful: delegate=node2, client=pacemaker-controld.1468, origin=node2, completed=...
   1 event found

2. Check that the cluster starts the ASCS resources on the remaining node. In a 2-node cluster this leads to ASCS and ERS running on the same node:

   [root]# pcs resource
     * Resource Group: grp_S4H_ASCS20:
       * rsc_vip_S4H_ASCS20        (ocf:heartbeat:IPAddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ASCS20        (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ASCS20        (ocf:heartbeat:SAPInstance):     Started node2
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29 (ocf:heartbeat:IPAddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ERS29 (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ERS29 (ocf:heartbeat:SAPInstance):     Started node2

3. Check that the cluster automatically moves the ERS instance to the previously failed node after the fenced node is running again:

   [root]# pcs resource
     * Resource Group: grp_S4H_ASCS20:
       * rsc_vip_S4H_ASCS20        (ocf:heartbeat:IPAddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ASCS20        (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ASCS20        (ocf:heartbeat:SAPInstance):     Started node2
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29 (ocf:heartbeat:IPAddr2):         Started node1
       * rsc_SAPStartSrv_S4H_ERS29 (ocf:heartbeat:SAPStartSrv):     Started node1
       * rsc_SAPInstance_S4H_ERS29 (ocf:heartbeat:SAPInstance):     Started node1

   The higher ASCS resource stickiness and the constraints you have configured earlier ensure that the ASCS instance stays in place to avoid the unnecessary disruption of the service again.

Verification

1. Check that the cluster on the other node fences the crashed node:

   [root]# pcs stonith history
   reboot of node2 successful: delegate=node1, client=pacemaker-controld.1426, origin=node1, completed=...
   1 event found

2. Check that the cluster starts the ERS resources on the remaining node. In a 2-node cluster this leads to ASCS and ERS running on the same node:

   [root]# pcs resource
     * Resource Group: grp_S4H_ASCS20:
       * rsc_vip_S4H_ASCS20        (ocf:heartbeat:IPAddr2):         Started node1
       * rsc_SAPStartSrv_S4H_ASCS20        (ocf:heartbeat:SAPStartSrv):     Started node1
       * rsc_SAPInstance_S4H_ASCS20        (ocf:heartbeat:SAPInstance):     Started node1
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29 (ocf:heartbeat:IPAddr2):         Started node1
       * rsc_SAPStartSrv_S4H_ERS29 (ocf:heartbeat:SAPStartSrv):     Started node1
       * rsc_SAPInstance_S4H_ERS29 (ocf:heartbeat:SAPInstance):     Started node1

3. Check that the cluster automatically moves the ERS instance to the previously failed node after the fenced node is running again:

   [root]# pcs resource
   * Resource Group: grp_S4H_ASCS20:
       * rsc_vip_S4H_ASCS20        (ocf:heartbeat:IPAddr2):         Started node1
       * rsc_SAPStartSrv_S4H_ASCS20        (ocf:heartbeat:SAPStartSrv):     Started node1
       * rsc_SAPInstance_S4H_ASCS20        (ocf:heartbeat:SAPInstance):     Started node1
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29 (ocf:heartbeat:IPAddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ERS29 (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ERS29 (ocf:heartbeat:SAPInstance):     Started node2

Procedure

1. Identify the process ID (PID) of the enqueue server on the node where ASCS is running. Use sapcontrol… GetProcessList as user <sid>adm to find the PID as the last entry:

   <sid>adm $ sapcontrol -nr <instance> -function GetProcessList
   name, description, dispstatus, textstatus, starttime, elapsedtime, pid
   enq_replicator, Enqueue Replicator 2, GREEN, Running, YYYY MM DD 13:20:07, 0:00:08, 161323

   In the example, the enqueue server PID is 161323.

2. Send a SIGKILL signal to the identified process to kill it instantly, for example, ASCS on node1:

   <sid>adm $ kill -9 <pid>

   • Replace <pid> with the PID of the enqueue server, for example, 161323.
3. Check that the ASCS instance recovers on the same node. With the default migration-threshold of all resources set to 3, the cluster restarts the resource on the same node twice, before it recovers on a different node.
4. Repeat steps 1 and 2 until you have killed the process 3 times. The cluster recovers the resource on another node after 3 consecutive failures of the same resource.

Verification

1. Check that the ASCS instance is now running on the additional node:

   [root]# pcs resource
     * Resource Group: grp_S4H_ASCS20:
       * rsc_vip_S4H_ASCS20        (ocf:heartbeat:IPAddr2):         Started node3
       * rsc_SAPStartSrv_S4H_ASCS20        (ocf:heartbeat:SAPStartSrv):     Started node3
       * rsc_SAPInstance_S4H_ASCS20        (ocf:heartbeat:SAPInstance):     Started node3
     * Resource Group: grp_S4H_ERS29:
       * rsc_vip_S4H_ERS29 (ocf:heartbeat:IPAddr2):         Started node2
       * rsc_SAPStartSrv_S4H_ERS29 (ocf:heartbeat:SAPStartSrv):     Started node2
       * rsc_SAPInstance_S4H_ERS29 (ocf:heartbeat:SAPInstance):     Started node2

   In our example, the ASCS instance failed on node1 and has been moved to node3. The ERS instance stays in place on node2.

2. Check the fail count and notice that it has failed 3 times to trigger the recovery on the other node:

   [root]# pcs resource failcount
   Failcounts for resource 'rsc_SAPInstance_S4H_ASCS20'
     node1: 3

Procedure

1. On the new node, go to the directory where you have extracted the installation media:

   [root]# cd <software_path>

   • Replace <software_path> with the path to your unpacked media, for example, /sapmedia/SWPM20_SP19/.

2. Run the installer command on the new node:

   [root]# ./sapinst

3. Open the web installer UI using the link provided in the terminal.
4. Open the SAP product you want to install and enter the installation option. Expand the High-Availability System option and select Prepare Additional Cluster Node. Click Next.

5. Provide the requested installation information on each page and click Next to move forward.

   Some steps, like extracting SAP packages, can take a while. Keep an eye on the terminal in which you started the installer for details of the ongoing process that are not displayed in the web UI.

Verification

1. Check that the new node has the SAP ports in the services file. For example, count the entries that contain SAP System in their port description and compare the result on all existing nodes:

   [root]# grep -i "SAP System" /etc/services | wc -l
   401

   Update the /etc/services file on the node if it is missing entries.

2. Check that the /usr/sap/hostctrl/ path is present and that the version is the same as on the existing cluster nodes:

   [root]# /usr/sap/hostctrl/exe/saphostexec -version

Verification

1. Check that the file exists and has the same owner and permissions as on the source node:

   [root]# ls -lh /usr/sap/sapservices
   -rwxr-xr-x. 1 root sapinst 208 Jun 16 13:59 /usr/sap/sapservices

2. Check that the file contains the configured instances, for example, ASCS and ERS:

   [root]# cat /usr/sap/sapservices
   systemctl --no-ask-password start SAPS4H_20 # sapstartsrv pf=/usr/sap/S4H/SYS/profile/S4H_ASCS20_s4hascs
   systemctl --no-ask-password start SAPS4H_29 # sapstartsrv pf=/usr/sap/S4H/SYS/profile/S4H_ERS29_s4hers

Procedure

1. Register the ASCS instance. Run the following SAP command as the root user on the new node to create the systemd integration:

   [root]# export LD_LIBRARY_PATH=/usr/sap/<SID>/ASCS<instance>/exe && \
   /usr/sap/<SID>/ASCS<instance>/exe/sapstartsrv \
   pf=/usr/sap/<SID>/SYS/profile/<SID>_ASCS<instance>_<ascs_virtual_hostname> \
   -reg

   The command executes the sapstartsrv service for the selected instance profile and registers the instance service on the current system. It creates the systemd unit for the instance service, if it does not exist, and updates the local /usr/sap/sapservices file.

   • Replace <SID> with your ASCS instance SID, for example, S4H.
   • Replace <instance> with your ASCS instance number, for example, 20.
   • Replace <ascs_virtual_hostname> with the virtual hostname for your ASCS instance, for example, s4hascs.

2. Register the ERS instance by repeating step 1 for the ERS profile:

   [root]# export LD_LIBRARY_PATH=/usr/sap/<SID>/ERS<instance>/exe && \
   /usr/sap/<SID>/ERS<instance>/exe/sapstartsrv \
   pf=/usr/sap/<SID>/SYS/profile/<SID>_ERS<instance>_<ers_virtual_hostname> \
   -reg

3. Optional: Register any PAS or AAS instance by repeating step 1 for the respective application server profile. Skip this step if you have not configured PAS or AAS instances in this cluster:

   [root]# export LD_LIBRARY_PATH=/usr/sap/<SID>/D<instance>/exe && \
   /usr/sap/<SID>/D<instance>/exe/sapstartsrv \
   pf=/usr/sap/<SID>/SYS/profile/<SID>_D<instance>_<as_virtual_hostname> \
   -reg

4. Disable the ASCS, ERS and any other application instance service that the cluster manages:

   [root]# systemctl disable SAP<SID>_<instance>.service
   Removed "/etc/systemd/system/multi-user.target.wants/SAP<SID>_<instance>.service".

   Run this using the ASCS instance number and repeat the command using the ERS instance number.

   Optional: Repeat the same for the PAS or AAS instance services.

5. Create the systemd drop-in directory for the ASCS, ERS and any other application instance service that the cluster manages:

   [root]# mkdir /etc/systemd/system/SAP<SID>_<instance>.service.d

   Run this using the ASCS instance number and repeat the command using the ERS instance number.

   Optional: Repeat using the PAS or AAS instance number.

6. Create the drop-in files for the instances in the new directory:

   [root]# cat << EOF > /etc/systemd/system/SAP<SID>_<instance>.service.d/HA.conf
   [Service]
   Restart=no
   EOF

   Run this using the ASCS instance number and repeat the command using the ERS instance number.

   Optional: Repeat using the PAS or AAS instance number.

7. Reload the systemd units to activate the drop-in configuration:

   [root]# systemctl daemon-reload

Verification

1. Check that all instances have instance systemd units and that they are disabled on the new node:

   [root]# systemctl list-unit-files SAPS4H*
   UNIT FILE         STATE    PRESET
   SAPS4H_20.service disabled disabled
   SAPS4H_29.service disabled disabled

   Optional: PAS or AAS instance service files are listed as well in all of the verification steps when you have configured the application server instances.

2. Check that the sapservices file contains entries for every instance on every cluster node:

   [root]# cat /usr/sap/sapservices
   systemctl --no-ask-password start SAPS4H_20 # sapstartsrv pf=/sapmnt/S4H/profile/S4H_ASCS20_s4hascs
   systemctl --no-ask-password start SAPS4H_29 # sapstartsrv pf=/sapmnt/S4H/profile/S4H_ERS29_s4hers

3. Check that all systemd configuration overrides are present:

   [root]# systemd-delta | grep SAP
   ...
   [EXTENDED]   /etc/systemd/system/SAPS4H_20.service → /etc/systemd/system/SAPS4H_20.service.d/HA.conf
   [EXTENDED]   /etc/systemd/system/SAPS4H_29.service → /etc/systemd/system/SAPS4H_29.service.d/HA.conf

Procedure

1. Install the Red Hat High Availability Add-On software packages from the High Availability repository. Choose the same fence agents as you have configured on the existing nodes and execute the installation on the new node:

   [root]# dnf install pcs pacemaker fence-agents-<model>

2. Start and enable the pcsd service:

   [root]# systemctl enable --now pcsd.service

3. Optional: If you are running the firewalld service, enable the ports that are required by the Red Hat High Availability Add-On:

   [root]# firewall-cmd --add-service=high-availability
   [root]# firewall-cmd --runtime-to-permanent

4. Set a password for the user hacluster:

   [root]# passwd hacluster

5. Authenticate the user hacluster for the new node in the existing cluster. Run this on an existing node, for example, node1:

   [root]# pcs host auth <node3>
   Username: hacluster
   Password:
   <node3>: Authorized

   • Enter the node names with or without FQDN, as defined in the /etc/hosts file.
   • Enter the hacluster user password in the prompt.

6. Add the new node to the existing cluster. This syncs cluster files between the nodes. Run this on an existing node, for example, node1:

   [root]# pcs cluster node add <node3>
   No addresses specified for host 'node3', using 'node3'
   Disabling sbd...
   node3: sbd disabled
   Sending 'corosync authkey', 'pacemaker authkey' to 'node3'
   node3: successful distribution of the file 'corosync authkey'
   node3: successful distribution of the file 'pacemaker authkey'
   Sending updated corosync.conf to nodes...
   node1: Succeeded
   node3: Succeeded
   node2: Succeeded
   node1: Corosync configuration reloaded

7. Start the cluster on the new node. Run this on the new node, for example, node3:

   [root]# pcs cluster start
   Starting Cluster...

8. Enable the cluster to be started automatically on system start, which enables the corosync and pacemaker services. Skip this step if you prefer to manually control the start of the cluster after a node restarts. Run on the new node:

   [root]# pcs cluster enable

Procedure

1. Clean up resource failures:

   [root]# pcs resource cleanup

2. Clean up the STONITH failure history:

   [root]# pcs stonith history cleanup

Verification

1. Check the overall cluster status and confirm that no failures are displayed anymore:

   [root]# pcs status --full

2. Check that the stonith history for fencing actions has 0 events:

Procedure

1. Set maintenance mode for the entire cluster:

   [root]# pcs property set maintenance-mode=true

   Setting maintenance for the whole cluster ensures that no activity during the maintenance phase can trigger cluster actions and impact the sap instance update process.

2. Verify that the cluster resource management is fully disabled:

   [root]# pcs status
   ...
   
                 *** Resource management is DISABLED ***
     The cluster will not attempt to start, stop or recover services
   ...

3. Perform the required maintenance on the application server instances using the SAP procedure.

4. Remove the maintenance mode of the cluster again:

   [root]# pcs property set maintenance-mode=

Procedure

1. Check the current values of the following TCP keepalive operating system settings on your SAP application servers, for example:

   [root]# sysctl -a --pattern net.ipv4.tcp_keepalive
   net.ipv4.tcp_keepalive_intvl = 75
   net.ipv4.tcp_keepalive_probes = 9
   net.ipv4.tcp_keepalive_time = 7200

2. Temporarily update the TCP keepalive settings in a way that helps in your particular situation. The following are example values that SAP recommends in SAP Note 1410736 - TCP/IP: setting keepalive interval. Apply changes only as required for your environment and on all cluster nodes:

   [root]# sysctl -w \
   net.ipv4.tcp_keepalive_time=300 \
   net.ipv4.tcp_keepalive_intvl=75 \
   net.ipv4.tcp_keepalive_probes=9

3. Test if the settings improve the situation.

4. Make the changes permanent. Add the TCP keepalive settings to a sysctl configuration file for a persistent setup. Do this on every cluster node:

   [root]# cat << EOF >> /etc/sysctl.d/sap.conf
   net.ipv4.tcp_keepalive_time=300
   net.ipv4.tcp_keepalive_intvl=75
   net.ipv4.tcp_keepalive_probes=9
   EOF