Automating SAP HANA Scale-Out System Replication using the RHEL HA Add-On

Procedure

1. If a staging configuration is not present, you can assign the registration of the SAP HANA test deployment directly to Red Hat Subscription Management (RHSM) with the following command:

   [root:~]# subscription-manager register

2. Enter the username and password.

3. List all pools available with the rhel-8-for-x86_64-sap-solutions-rpms repositories:

   [root:~]# subscription-manager list --available --matches="rhel-8-for-x86_64-sap-solutions-rpms"

   For more information, refer to Configuring basic system settings.

   Note

   The company pool ID is required. If the list is empty contact Red Hat for a list of the company’s subscriptions.

4. Attach the pool ID to your server instances:

   [root:~]# subscription-manager attach --pool=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

5. Check if the repo for sap-solutions is enabled:

   [root:~]# yum repolist | grep sap-solution
   rhel-8-for-x86_64-sap-solutions-rpms RHEL for x86_64 - SAP Solutions (RPMs)

   You can enable the RHEL 8 required repos:

   [root:~]# subscription-manager repos --enable=rhel-8-for-x86_64-sap-solutions-rpms --enable=rhel-8-for-x86_64-highavailability-rpms

   For more information, see RHEL for SAP Subscriptions and Repositories.

6. Update the packages on all systems to verify that the correct RPM packages and versions are installed:

   [root:~]# yum update -y

Procedure

1. Login as user root on every host and configure the /etc/hosts file.
2. Create a host entry for every SAP HANA host in the scale-out environment.

3. Copy the hosts file to every node. It is important to set the hostname in the order shown in the following output example. If not, the SAP HANA environment fails during the deployment or operating process.

   Note

   This configuration is based on the parameters listed in the Preparing the SAP HANA Scale-Out environment section of this document.

   [root:~]# cat << EOF >> /etc/hosts
   10.0.1.21 dc1hana01.example.com dc1hana01
   10.0.1.22 dc1hana02.example.com dc1hana02
   10.0.1.23 dc1hana03.example.com dc1hana03
   10.0.1.24 dc1hana04.example.com dc1hana04
   10.0.1.31 dc2hana01.example.com dc2hana01
   10.0.1.32 dc2hana02.example.com dc2hana02
   10.0.1.33 dc2hana03.example.com dc2hana03
   10.0.1.34 dc2hana04.example.com dc2hana04
   10.0.1.41 majoritymaker.example.com majoritymaker
   EOF

Procedure

1. Login as user root on every SAP HANA host for the additional /usr/sap partition.

   Note

   In general, the default XFS format and mount options are optimal for most workloads. Red Hat recommends that the default values are used unless specific configuration changes are expected to benefit the workload of the file system. All supported file systems can be used. For more information, refer to SAP Note 2972496 - SAP HANA Filesystem Types. If software RAID is used, the mks.xfs command automatically configures itself with the correct stripe unit and width to align with the hardware.

2. Create the required mount points:

   [root:~]# mkdir -p /usr/sap

3. On the logical volume, create file systems based on XFS:

   [root:~]# mkfs -t xfs -b size=4096 /dev/sdb

   For more information about the creation of an XFS filesystem and the tuning possibilities, run the man mkfs.xfs command. For optimal performance of the XFS file system, refer to the article What are some of best practices for tuning XFS filesystems.

4. Write the mount directives to /etc/fstab:

   [root:~]# echo "/dev/sdb /usr/sap xfs defaults 1 6" >> /etc/fstab

   Note

   If mount points are managed by filesystem resources, these file systems must then be later commented out again in the /etc/fstab file. 

5. Check if XFS filesystems from /etc/fstab can be mounted:

   [root:~]# mount /usr/sap

Procedure

1. Login as user root on every SAP HANA host for the shared storage configuration.

   Note

   The nfs-utils package is required. Every datacenter requires its own storage configuration. For this example, the storage configuration is built as a shared storage environment. Both scale-out environments are using its own NFS share. This configuration is based on the information in the Preparing the SAP HANA Scale-Out environment section of this document. In a production environment, this procedure should be configured as supported by your preferred hardware vendor.

2. Install the nfs-utils package:

   [root:~]# yum install -y nfs-utils

3. Configure the nodes in Datacenter 1:

   [root:~]# mkdir -p /hana/{shared,data,log}
   cat <<EOF >> /etc/fstab
   10.0.1.61:/data/dc1/shared /hana/shared nfs4 defaults 0 0
   10.0.1.61:/data/dc1/data /hana/data nfs4 defaults 0 0
   10.0.1.61:/data/dc1/log /hana/log nfs4 defaults 0 0
   EOF

   To mount the volumes run the following command:

   [root:~]# mount -a

4. Configure the nodes in Datacenter 2:

   [root:~]# mkdir -p /hana/{shared,data,log}
   cat <<EOF >> /etc/fstab
   10.0.1.62:/data/dc2/shared /hana/shared nfs4 defaults 0 0
   10.0.1.62:/data/dc2/data /hana/data nfs4 defaults 0 0
   10.0.1.62:/data/dc2/log /hana/log nfs4 defaults 0 0
   EOF

   To mount the volumes run the following command:

   [root:~]# mount -a

Prerequisites

1. You are logged in as user root on every host of the shared storage configuration.
2. You have prepared the installation source of SAP HANA.

3. You have set the hostname compatible with SAP HANA:

   [root:~]# hostnamectl set-hostname dc1hana01

Procedure: Verifying /etc/hosts

1. Verify that /etc/hosts contains an entry matching the hostname and IP address of the system: .example.com:

   [root:~]# hostname
   <hostname>
   [root:~]# hostname -s
   <hostname>
   [root:~]# hostname -f
   <hostname>.example.com
   [root:~]# hostname -d
   example.com

2. Set the system language to English:

   [root:~]# localectl set-locale LANG=en_US.UTF-8

Procedure: Configuring NTP

1. Edit /etc/chrony.conf and verify that the server lines reflect your ntp servers:

   [root:~]# yum -y install chrony
   [root:~]# systemctl stop chronyd.service

2. Check time server entries:

   [root:~]# grep ^server /etc/chrony.conf
   server 0.de.pool.ntp.org
   server 1.de.pool.ntp.org

3. Enable and start the chrony service:

   [root:~]# systemctl enable chronyd.service
   [root:~]# systemctl start chronyd.service
   [root:~]# systemctl restart systemd-timedated.service

4. Verify that the chrony service is enabled:

   [root:~]# systemctl status chronyd.service
   chronyd.service enabled
   [root:~]# chronyc sources
   210 Number of sources = 3
   MS Name/IP address Stratum Poll Reach LastRx Last sample
   =====================================================================
   ^* 0.de.pool.ntp.org 2 8 377 200 -2659ns[-3000ns] +/- 28ms
   ^-de.pool.ntp.org 2 8 377 135 -533us[ -533us] +/- 116ms
   ^-ntp2.example.com 2 9 377 445 +14ms[ +14ms] +/- 217ms

Procedure (optional)

1. Create the sapadm and sapsys user for the SAP Host Agent and set the password for the sapadm user. The UID 996 of the user sapadm and the GID 79 of the group sapsys are based on the parameters in the Preparing the SAP HANA Scale-Out environment section of this document.

   [root:~]# adduser sapadm --uid 996
   [root:~]# groupadd sapsys --gid 79
   [root:~]# passwd sapadm

2. Create a temp directory, unpack the installation source, and install the SAP Host Agent from the temp directory. The variable INSTALLDIRHOSTAGENT is an example:

   [root:~]# export TEMPDIR=$(mktemp -d)
   [root:~]# export
   INSTALLDIRHOSTAGENT=/install/HANA/DATA_UNITS/HDB_SERVER_LINUX_X86_64/
   [root:~]# systemctl disable abrtd
   [root:~]# systemctl disable abrt-ccpp
   [root:~]# cp -rp ${INSTALLDIRHOSTAGENT}/server/HOSTAGENT.TGZ $TEMPDIR/ cd $TEMPDIR
   [root:~]# tar -xzvf HOSTAGENT.TGZ
   [root:~]# cd global/hdb/saphostagent_setup/
   [root:~]# ./saphostexec -install

   Secure operation only works with an encrypted connection. You can configure a working SSL connection to achieve this. An SSL password is required. The following example is based on the parameters in the Preparing the SAP HANA Scale-Out environment section of this document.

   [root:~]# export MYHOSTNAME=$(hostname)
   [root:~]# export SSLPASSWORD=Us3Your0wnS3cur3Password
   [root:~]# export LD_LIBRARY_PATH=/usr/sap/hostctrl/exe/
   [root:~]# export SECUDIR=/usr/sap/hostctrl/exe/sec
   [root:~]# cd /usr/sap/hostctrl/exe
   [root:~]# mkdir /usr/sap/hostctrl/exe/sec
   [root:~]# /usr/sap/hostctrl/exe/sapgenpse gen_pse -p SAPSSLS.pse -x $SSLPASSWORD -r /tmp/${MYHOSTNAME}-csr.p10 "CN=$MYHOSTNAME"
   [root:~]# /usr/sap/hostctrl/exe/sapgenpse seclogin -p SAPSSLS.pse -x $SSLPASSWORD -O sapadm chown sapadm /usr/sap/hostctrl/exe/sec/SAPSSLS.pse
   [root:~]# /usr/sap/hostctrl/exe/saphostexec -restart*

3. Verify the SAP Host Agent is available for all SAP HANA nodes:

   [root:~]# netstat -tulpen | grep sapstartsrv
   tcp      0     0 0.0.0.0:50014   0.0.0.0:*  LISTEN   1002  84028   4319/sapstartsrv
   tcp      0     0 0.0.0.0:50013   0.0.0.0:*  LISTEN   1002  47542   4319/sapstartsrv

   Note

   Not all processes are identified. Non-owned process information are not be shown. You have to be root to see all processes.

   [root:~]# netstat -tulpen | grep 1129
   tcp 0 	0     0.0.0.0:1129 	0.0.0.0:* 	LISTEN 	996	 25632	 1345/sapstartsrv

   For more information about how to install SAP Host Agent, see SAP Host Agent Installation.

Procedure

1. Login as user root on one SAP HANA node in each datacenter to start the SAP HANA Scale-Out installation.

2. In this solution, the following command is executed on one node in each datacenter:

   [root:~]# INSTALLDIR=/install/51053381/DATA_UNITS HDB_SERVER_LINUX_X86_64/
   [root:~]# cd $INSTALLDIR
   [root:~]# ./hdblcm --dump_configfile_template=/tmp/templateFile

   Important

   The correct addhosts parameter must be used. This must not include the installation node.

3. Change the passwords in /tmp/templateFile.xml:

   Note

   The internal_network parameter is for the internal scale-out communication network. This prefills the SAP HANA configuration file global.ini with the correct configuration during the installation process.

   Datacenter 1 example:

   [root:~]# cat /tmp/templateFile.xml | ./hdblcm \ --batch \ --sid=RH1 \ --number=10 \ --action=install \ --hostname=dc1hana01 \ --addhosts=dc1hana02:role=worker,dc1hana03:role=worker,dc1hana04:role =standby \ --install_hostagent \ --system_usage=test \ --sapmnt=/hana/shared \ --datapath=/hana/data \ --logpath=/hana/log \ --root_user=root \ --workergroup=default \ --home=/usr/sap/RH1/home \ --userid=79 \ --shell=/bin/bash \ --groupid=79 \ --read_password_from_stdin=xml \ --internal_network=192.168.101.0/24 \ --remote_execution=saphostagent

   Datacenter 2 example:

   [root:~]# cat /tmp/templateFile.xml | ./hdblcm \ --batch \ --sid=RH1 \ --number=10 \ --action=install \ --hostname=dc2hana01 \ --addhosts=dc2hana02:role=worker,dc2hana03:role=worker,dc2hana04:role =standby \ --install_hostagent \ --system_usage=test \ --sapmnt=/hana/shared \ --datapath=/hana/data \ --logpath=/hana/log \ --root_user=root \ --workergroup=default \ --home=/usr/sap/RH1/home \ --userid=79 \ --shell=/bin/bash \ --groupid=79 \ --read_password_from_stdin=xml \ --internal_network=192.168.101.0/24 \ --remote_execution=saphostagent

4. Verify that everything is working on one host per datacenter after the installation process is complete:

   [root:~]# su - rh1adm /usr/sap/hostctrl/exe/sapcontrol -nr 10 -function GetSystemInstanceList
   10.04.2019 08:38:21
   GetSystemInstanceList OK
   hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
   dc1hana01,10,51013,51014,0.3,HDB|HDB_WORKER, GREEN
   dc1hana03,10,51013,51014,0.3,HDB|HDB_STANDBY, GREEN
   dc1hana02,10,51013,51014,0.3,HDB|HDB_WORKER, GREEN
   dc1hana04,10,51013,51014,0.3,HDB|HDB_WORKER, GREEN
   
   rh1adm@dc1hana01:/usr/sap/RH1/HDB10> HDBSettings.sh landscapeHostConfiguration.py
   | Host | Host | Host | Failover | Remove | Storage |
   Storage | Failover | Failover | NameServer | NameServer |
   IndexServer | IndexServer | Host | Host | Worker | Worker |
   | | Active | Status | Status | Status | Config |
   Actual | Config | Actual | Config | Actual | Config
   | Actual | Config | Actual | Config | Actual |
   | | | | | | Partition |
   Partition | Group | Group | Role | Role | Role
   | Role | Roles | Roles | Groups | Groups |
   | --------- | ------ | ------ | -------- | ------ | --------- |
   --------- | -------- | -------- | ---------- | ---------- |
   ----------- | ----------- | ------- | ------- | ------- | ------- |
   | dc1hana01 | yes | ok | | | 1 |
   1 | default | default | master 1 | master | worker |
   master | worker | worker | default | default |
   | dc1hana02 | yes | ok | | | 2 |
   2 | default | default | master 3 | slave | worker |
   slave | worker | worker | default | default |
   | dc1hana03 | yes | ok | | | 2 |
   2 | default | default | master 3 | slave | worker |
   slave | worker | worker | default | default |
   | dc1hana04 | yes | ignore | | | 0 |
   0 | default | default | master 2 | slave | standby |
   standby | standby | standby | default | - |
   
   rh1adm@dc1hana01: HDB info
   USER PID PPID %CPU VSZ RSS COMMAND
   rh1adm 31321 31320 0.0 116200 2824 -bash
   rh1adm 32254 31321 0.0 113304 1680 \_ /bin/sh
   /usr/sap/RH1/HDB10/HDB info
   rh1adm 32286 32254 0.0 155356 1868 \_ ps fx
   -U rh1adm -o user:8,pid:8,ppid:8,pcpu:5,vsz:10,rss:10,args
   rh1adm 27853 1 0.0 23916 1780 sapstart
   pf=/hana/shared/RH1/profile/RH1_HDB10_dc1hana01
   rh1adm 27863 27853 0.0 262272 32368 \_
   /usr/sap/RH1/HDB10/dc1hana01/trace/hdb.sapRH1_HDB10 -d -nw -f
   /usr/sap/RH1/HDB10/dc1hana01/daemon.ini
   pf=/usr/sap/RH1/SYS/profile/RH1_HDB10_dc1hana01
   rh1adm 27879 27863 53.0 9919108 6193868 \_
   hdbnameserver
   rh1adm 28186 27863 0.7 1860416 268304 \_
   hdbcompileserver
   rh1adm 28188 27863 65.8 3481068 1834440 \_
   hdbpreprocessor
   rh1adm 28228 27863 48.2 9431440 6481212 \_
   hdbindexserver -port 31003
   rh1adm 28231 27863 2.1 3064008 930796 \_
   hdbxsengine -port 31007
   rh1adm 28764 27863 1.1 2162344 302344 \_
   hdbwebdispatcher
   rh1adm 27763 1 0.2 502424 23376
   /usr/sap/RH1/HDB10/exe/sapstartsrvpf=/hana/shared/RH1/profile/RH1_HDB10_dc1hana01 -D -u rh1adm

Procedure

1. Install the hook on one node in each datacenter on a shared device. For more information, see Implementing a HA/DR Provider.

2. Create a directory in the hana shared folder to configure the hooks. This is configured to create additional data from the SAP HANA database. To enable it, you must stop the system and add two additional configuration parameters to the global.ini file. In this solution, the following example shows the configuration of ha_dr_provider_SAPHanaSr and trace.

   [root@dc1hana01]# su - rh1adm
   [rh1adm@dc1hana01]% sapcontrol -nr 10 -function StopSystem *[rh1adm@dc1hana01]% cat <<EOF >> /hana/shared/RH1/global/hdb/custom/config/global.ini
   
   [ha_dr_provider_SAPHanaSR]
   provider = SAPHanaSR
   path = /usr/share/SAPHanaSR-ScaleOut
   execution_order = 1
   
   [trace]
   ha_dr_saphanasr = info
   EOF

3. On each cluster node, create the file /etc/sudoers.d/20-saphana by running sudo visudo /etc/sudoers.d/20-saphana and add the contents below to allow the hook script to update the node attributes when the srConnectionChanged() hook is called:

   rh1adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_rh1_glob_srHook -v * -t crm_config -s SAPHanaSR
   
   rh1adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_rh1_gsh -v * -l reboot -t crm_config -s SAPHanaSR Defaults:rh1adm !requiretty

   For further information on why the defaults setting is needed, see The srHook attribute is set to SFAIL in a Pacemaker cluster managing SAP HANA system replication, even though replication is in a healthy state.

4. Start the SAP HANA database after the successful integration.

   # Execute the following commands on one HANA node in every datacenter
   [root]# su - rh1adm
   [rh1adm]% sapcontrol -nr 10 -function StartSystem

5. Verify that the hook script is working as expected. Perform an action to trigger the hook, such as stopping the HANA instance. Then, use the method given below to check if the hook logged anything:

   [rh1adm@dc1hana01]% cdtrace
   [rh1adm@dc1hana01]% awk '/ha_dr_SAPHanaSR.*crm_attribute/ \
   { printf "%s %s %s %s\n",$2,$3,$5,$16 }' nameserver_* 2018-05-04
   12:34:04.476445 ha_dr_SAPHanaSR SFAIL
   2018-05-04 12:53:06.316973 ha_dr_SAPHanaSR SOK

   For more information on how to verify that the SAP HANA hook is working, see Monitoring with M_HA_DR_PROVIDERS.

1. Create the constraint:

   [root@dc1hana01]# pcs constraint order start rsc_SAPHanaTopology_RH1_HDB10-clone then start rsc_SAPHana_RH1_HDB10-clone

2. Colocate the IPaddr2 resource with the promoted SAPHana resource:

   [root@dc1hana01]# pcs constraint colocation add rsc_ip_SAPHana_RH1_HDB10 with promoted rsc_SAPHana_RH1_HDB10-clone

3. Avoid the majoritymaker to use an active role in the cluster environment:

   [root@dc1hana01]# pcs constraint location add topology-avoids-majoritymaker rsc_SAPHanaTopology_RH1_HDB10-clone majoritymaker -INFINITY resource-discovery=never
   
   [root@dc1hana01]# pcs constraint location add hana-avoids-majoritymaker rsc_SAPHana_RH1_HDB10-clone majoritymaker -INFINITY resource-discovery=never

4. Disable maintenance-mode:

   Use maintenance-mode to start the resources after setting maintenance-mode to false. To avoid activities of pacemaker before all configurations are finished, we have used in the examples above --disabled. By default, the resources are started as soon as they are created. With --disabled, you can start the resources by using the command:

   [root@dc1hana01]# pcs resource enable <resource-name>

   To leave maintenance-mode, please use:

   [root@dc1hana01]# pcs property set maintenance-mode=false

5. Run the following 3 commands to verify that the cluster environment is working correctly:

   • pcs status provides an overview of every resource and if they are functioning correctly.
   • pcs status --full provides an overview of all resources and additional attribute information of the cluster environment.

   • SAPHanaSR-showAttr --sid=RH1 provides a readable overview that is based on the attribute information.

     The correct status is displayed a few minutes after deactivating the maintenance-mode.

     [root@dc1hana01]# pcs status
     Cluster name: hanascaleoutsr Stack: corosync
     Current DC: dc2hana01 (version 1.1.18-11.el7_5.4-2b07d5c5a9) - partition with quorum
     Last updated: Tue Mar 26 14:26:38 2019
     Last change: Tue Mar 26 14:25:47 2019 by root via crm_attribute on dc1hana01
     9 nodes configured
     20 resources configured
     Online: [ dc1hana01 dc1hana02 dc1hana03 dc1hana04 dc2hana01 dc2hana02 dc2hana03 dc2hana04 majoritymaker ]
     Full list of resources:
     
     fencing (stonith:fence_rhevm): Started dc1hana01
     Clone Set: rsc_SAPHanaTopology_RH1_HDB10-clone
     [rsc_SAPHanaTopology_RH1_HDB10]
     Started: [ dc1hana01 dc1hana02 dc1hana03 dc1hana04 dc2hana01 dc2hana02 dc2hana03 dc2hana04 ]
     Stopped: [ majoritymaker ] Clone Set: msl_rsc_SAPHana_RH1_HDB10 [rsc_SAPHana_RH1_HDB10] (promotable):
     Promoted: [ dc1hana01 ]
     Unpromoted: [ dc1hana02 dc1hana03 dc1hana04 dc2hana01 dc2hana02 dc2hana03 dc2hana04 ]
     Stopped: [ majoritymaker ]
     rsc_ip_SAPHana_RH1_HDB10	(ocf::heartbeat:IPaddr2): Started dc1hana01
     Daemon Status:
       corosync: active/enabled
       pacemaker: active/enabled
       pcsd: active/enabled
     
     
     [root@dc1hana01]# SAPHanaSR-showAttr --sid=RH1
     Global prim srHook sync_state
     ------------------------------
     global DC1 SOK SOK
     Sit lpt lss mns srr
     ---------------------------------
     DC1 1553607125 4 dc1hana01 P
     DC2 30 4 dc2hana01 S
     
     H clone_state roles score site
     --------------------------------------------------------
     1 PROMOTED promoted1 promoted:worker promoted 150 DC1
     2 DEMOTED promoted2:slave:worker:slave 110 DC1
     3 DEMOTED slave:slave:worker:slave -10000 DC1
     4 DEMOTED promoted3:slave:standby:standby 115 DC1
     5 DEMOTED promoted2 promoted:worker promoted 100 DC2
     6 DEMOTED promoted3:slave:worker:slave 80 DC2
     7 DEMOTED slave:slave:worker:slave -12200 DC2
     8 DEMOTED promoted1:slave:standby:standby 80 DC2
     9 :shtdown:shtdown:shtdown