Chapter 3. Installing SAP HANA scale-out for a 8-node HA cluster setup

Procedure

1. Create the directories for the shared filesystems:

   [root]# mkdir -p /hana/{shared,data,log}

2. Add the shared NFS filesystems to /etc/fstab to mount them automatically on system start. Configure the mount options that apply to your environment. The following is a basic example:

   [root]# vi /etc/fstab

   …
   <nfs_server>:/<site_path>/data /hana/data nfs4 defaults 0 0
   <nfs_server>:/<site_path>/log /hana/log nfs4 defaults 0 0
   <nfs_server>:/<site_path>/shared /hana/shared nfs4 defaults 0 0

   • Replace <nfs_server> with the NFS server DNS name or the IP address of each share, for example, nfs01-datacenter1a.example.com.
   • Replace <site_path> with the site specific root path, for example, dc1 on the nodes of one HANA site.

3. Reload the systemctl daemon to make the new /etc/fstab entries known to systemd:

   [root]# systemctl daemon-reload

4. Mount any new filesystems that you configured in the /etc/fstab:

   [root]# mount -a

5. Repeat the configuration steps on each cluster node.

Verification

1. Check that the filesystems are mounted, for example, on HANA site 1:

   [root]# df -hP | grep hana

   nfs01-datacenter1a.example.com:/dc1/hana/data    8.0E   32G  8.0E   1% /hana/data
   nfs01-datacenter1a.example.com:/dc1/hana/log     8.0E   32G  8.0E   1% /hana/log
   nfs01-datacenter1a.example.com:/dc1/hana/shared  8.0E   32G  8.0E   1% /hana/shared

2. Check that the systemd mount targets exist for the filesystems configured in the /etc/fstab:

   [root]# systemctl list-units --all | grep -e 'hana*.*mount' | column -t

   hana-data.mount    loaded  active  mounted  /hana/data
   hana-log.mount     loaded  active  mounted  /hana/log
   hana-shared.mount  loaded  active  mounted  /hana/shared

3. Repeat the verification steps on each cluster node.

Procedure

1. Create the sapsys group. Use the prepared group ID, for example, ID 10001:

   [root]# groupadd -g 10001 sapsys

2. Create the sapadm user as a member of the sapsys group. The user does not need a login shell. Use the prepared user ID, for example ID, 10200:

   [root]# useradd -u 10200 -g sapsys sapadm \
   -c 'SAP Local Administrator' -s /sbin/nologin

3. Create the <sid>adm user as a member of the sapsys group. Use the prepared user ID, for example, ID 10210 for user rh1adm:

   [root]# useradd -u 10210 -g sapsys rh1adm \
   -c 'SAP HANA Administrator' -s /bin/sh

   As the user shell, we recommend that you either use /bin/sh or /bin/csh. SAP installations provide user profiles and useful shell aliases in these shells.

4. Repeat the steps on all nodes.

Verification

1. Check that the users sapadm and <sid>adm exist and have the correct groups and IDs configured, for example:

   [root]# id sapadm rh1adm

   uid=10200(sapadm) gid=10001(sapsys) groups=10001(sapsys)
   uid=10210(rh1adm) gid=10001(sapsys) groups=10001(sapsys)

2. Check that the users have the correct description, home directory and shell defined:

   [root]# grep -E 'sapadm|rh1adm' /etc/passwd

   sapadm:x:10200:10001:SAP Local Administrator:/home/sapadm:/sbin/nologin
   rh1adm:x:10210:10001:SAP HANA Administrator:/home/rh1adm:/bin/sh

3. Repeat the check on all nodes and verify that the names and IDs are identical.

Procedure

1. Generate an ssh key pair. When no key type is defined, it creates an Ed25519 key by default, like in the following example for the root user:

   [root]# ssh-keygen

2. Option 1, if you have ssh PasswordAuthentication enabled on the remote system: Use the ssh-copy-id tool to add the ssh public key to the authorized_keys on the remote system. This automatically creates the .ssh/ directory and authorized_keys file with correct permissions for the target user on the remote system. Run it on the host on which you created the ssh key in the previous step and enter the target user password when prompted:

   [root]# ssh-copy-id <remote_system>

   In the case of the root user, this only works if the ssh config allows PermitRootLogin and you can provide the root user password in the prompt. Check the ssh configuration setting on the remote system if you face access permission issues even after you have enabled PasswordAuthentication. Consult your security policies before you enable these parameters on your HANA systems.

3. Option 2, if password login to the target user on the remote host is prohibited or otherwise not possible: Configure the ssh key access on the remote system manually.

   1. Create the .ssh/ directory in the target user’s home path on the remote system, if it does not exist yet. Run this on the remote system, for example, for the root user:

      [root]# mkdir /root/.ssh

   2. Change the permissions of the new .ssh/ directory. For security reasons the ssh key access does not work when the permissions are not correct. Run this on the remote system:

      [root]# chmod 0700 /root/.ssh

   3. Copy the ssh public key from the .pub file that was created by the previous ssh-keygen, for example, id_ed25519.pub in the default setting:

      [root]# cat /root/.ssh/id_ed25519.pub

   4. Add the public key to the authorized_keys file. The command creates the file if it does not exist yet, otherwise it appends the key to the existing content. Run this on the remote system, for example, on dc1hana2:

      [root]# cat << EOF >> /root/.ssh/authorized_keys
      ssh-ed25519 … root@<node1>
      EOF

   5. Ensure that the authorized_keys file has the correct permissions, otherwise the ssh key access is blocked for security reasons:

      [root]# chmod 0600 /root/.ssh/authorized_keys

4. Access each system and log in from any source host to any remote host that you require for the setup. On first login you must accept each new connection once in an interactive prompt. This saves each host and key in the ssh known_hosts file by default.

   1. Option 1: Log in from each host to each other host and accept the key fingerprint once to save it to the local known_hosts file. Subsequent logins to the same host will not require further interaction, unless the key changes. This is a security measure to prevent unsolicited changes of the ssh keys. The following example confirms the authenticity of host dc1hana2:

      [root]# ssh dc1hana2

      The authenticity of host 'dc1hana2 (pass:[***])' can't be established.
      ED25519 key fingerprint is SHA256:pass:[*********************************].
      This key is not known by any other names
      Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
      …

   2. Option 2: If you configure ssh key access between multiple systems you can use ssh-keyscan to collect the public host key from multiple hosts and save it to the local known_hosts file in a single step per host. Run this on each system for which you distributed the public key and list all remote hosts that you potentially access from this node and user, for example, for the root user on host dc1hana1:

      [root]# ssh-keyscan -f - >> /root/.ssh/known_hosts
      dc1hana1
      dc1hana2
      dc1hana3
      dc1hana4
      dc2hana1
      dc2hana2
      dc2hana3
      dc2hana4
      <Ctrl-d>

      # dc1hana1:22 SSH-2.0-OpenSSH_8.7
      dc1hana1 ssh-ed25519 …
      # dc1hana2:22 SSH-2.0-OpenSSH_8.7
      dc1hana2 ssh-ed25519 …
      # dc1hana3:22 SSH-2.0-OpenSSH_8.7
      dc1hana3 ssh-ed25519 …
      # dc1hana4:22 SSH-2.0-OpenSSH_8.7
      dc1hana4 ssh-ed25519 …
      # dc2hana1:22 SSH-2.0-OpenSSH_8.7
      dc2hana1 ssh-ed25519 …
      # dc2hana2:22 SSH-2.0-OpenSSH_8.7
      dc2hana2 ssh-ed25519 …
      # dc2hana3:22 SSH-2.0-OpenSSH_8.7
      dc2hana3 ssh-ed25519 …
      # dc2hana4:22 SSH-2.0-OpenSSH_8.7
      dc2hana4 ssh-ed25519 …

      • -f - allows you to provide a list of hosts on the standard input. Instead of the - you can use a file, which you prepare upfront with the list of hosts. You can also enter a single hostname instead of the -f parameter to collect the key of one host at a time.
      • In the case of the standard input list you end the input with Ctrl and d.
      • The >> shell redirection after the scan command directly appends the collected keys to the known_hosts file. If the file does not exist yet it is created in the process.

Procedure

1. Go to the directory which contains the installation media, for example, /sapmedia/hana:

   [root]# cd /sapmedia/hana

2. Unpack the installation media:

   [root]# unzip <sap_hana_software>.ZIP

3. Go into the path of the unpacked installation media:

   [root]# cd /sapmedia/hana/DATA_UNITS/HDB_LCM_LINUX_<arch>

4. Run the SAP HANA Lifecycle Management tool (HDBLCM) for an interactive installation:

   [root]# ./hdblcm

   In the interactive mode the installer asks you for all the required information, including the System ID (SID), Installation Number (instance), the filesystem location of data and log volumes, and more.

   In a scale-out installation you run the installer on the main node of one HANA site and provide any additional nodes of the same site as an installation parameter. For example, you run the installer for site 1 on dc1hana1 and add node dc1hana2 as an additional host name to add when the prompt asks for it.

   Optionally you can use the batch mode of the command-line installation tool and provide your configuration parameters in one step. For more details see Use Batch Mode to Perform Platform LCM Tasks in the SAP HANA Server Installation and Update Guide.

5. Repeat all steps on the main node of the second site. For the HANA system replication to work you must ensure that each HANA site consists of the same amount of systems with an identical HANA configuration.

Verification

1. Switch to the <sid>adm user:

   [root]# su - rh1adm

2. Check the HANA instance runtime information as user <sid>adm:

   rh1adm$HDB info

   USER          PID     PPID  %CPU        VSZ        RSS COMMAND
   rh1adm      12525    12524   0.2       8836       5568 -sh
   rh1adm      12584    12525   0.0       7520       3968  \_ /bin/sh /usr/sap/RH1/HDB02/HDB info
   rh1adm      12613    12584   0.0      10104       3484      \_ ps fx -U rh1adm -o user:8,pid:8,ppid:8,pcpu:5,vsz:10,rss:10,args
   rh1adm       8813        1   0.0     566804      41000 hdbrsutil  --start --port 30203 --volume 3 …
   rh1adm       8124        1   0.0     566724      40972 hdbrsutil  --start --port 30201 --volume 1 …
   rh1adm       7947        1   0.0       9312       3352 sapstart pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_dc1hana1
   rh1adm       7955     7947   0.0     460036      89176  \_ /usr/sap/RH1/HDB02/dc1hana1/trace/hdb.sapRH1_HDB02 -d -nw -f /usr/sap/RH1/HDB02/dc1hana1/daemon.ini pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_dc1hana1
   rh1adm       7981     7955  26.1   18612328   14092076      \_ hdbnameserver
   rh1adm       8642     7955   0.5    1465380     212048      \_ hdbcompileserver
   rh1adm       8645     7955   294    6616736    6049012      \_ hdbpreprocessor
   rh1adm       8687     7955  33.9   18931580   14929092      \_ hdbindexserver -port 30203
   rh1adm       8690     7955   2.0    5073572    1390440      \_ hdbxsengine -port 30207
   rh1adm       9202     7955   0.8    2772836     482088      \_ hdbwebdispatcher
   rh1adm       7782        1   0.1     566772      58444 /usr/sap/RH1/HDB02/exe/sapstartsrv pf=/usr/sap/RH1/SYS/profile/RH1_HDB02_dc1hana1
   root        11868     7782   0.1      10464       4644  \_ sapuxuserchk 0 128

3. Verify as <sid>adm on all sites that the HANA instances are running on all nodes in the site and their status is GREEN in the instance list, for example, on site 1:

   rh1adm$sapcontrol -nr ${TINSTANCE} -function GetSystemInstanceList

   hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
   dc1hana4, 2, 50213, 50214, 0.3, HDB|HDB_STANDBY, GREEN
   dc1hana1, 2, 50213, 50214, 0.3, HDB|HDB_WORKER, GREEN
   dc1hana3, 2, 50213, 50214, 0.3, HDB|HDB_WORKER, GREEN
   dc1hana2, 2, 50213, 50214, 0.3, HDB|HDB_WORKER, GREEN

4. Additionally, you can verify the landscapeHostConfiguration.py output for status ok:

   rh1adm$cdpy; python landscapeHostConfiguration.py

   | Host     | Host   | Host   | Failover | Remove | Storage   | Storage   | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host    | Host    | Worker  | Worker  |
   |          | Active | Status | Status   | Status | Config    | Actual    | Config   | Actual   | Config     | Actual     | Config      | Actual      | Config  | Actual  | Config  | Actual  |
   |          |        |        |          |        | Partition | Partition | Group    | Group    | Role       | Role       | Role        | Role        | Roles   | Roles   | Groups  | Groups  |
   | -------- | ------ | ------ | -------- | ------ | --------- | --------- | -------- | -------- | ---------- | ---------- | ----------- | ----------- | ------- | ------- | ------- | ------- |
   | dc1hana1 | yes    | ok     |          |        |         1 |         1 | default  | default  | master 1   | master     | worker      | master      | worker  | worker  | default | default |
   | dc1hana2 | yes    | ok     |          |        |         2 |         2 | default  | default  | master 2   | slave      | worker      | slave       | worker  | worker  | default | default |
   | dc1hana3 | yes    | ok     |          |        |         3 |         3 | default  | default  | slave      | slave      | worker      | slave       | worker  | worker  | default | default |
   | dc1hana4 | yes    | ignore |          |        |         0 |         0 | default  | default  | master 3   | slave      | standby     | standby     | standby | standby | default | -       |
   
   overall host status: ok

5. Check that the systemd units are installed for the HANA instance and the SAP Host Agent:

   [root]# systemctl list-unit-files --all sap* SAP*

   UNIT FILE            STATE     PRESET
   sapmedia.mount       generated -
   saphostagent.service enabled   disabled
   sapinit.service      generated -
   SAPRH1_02.service    enabled   disabled
   SAP.slice            static    -
   
   5 unit files listed.

6. Repeat the steps on all nodes. Note that the HANA profiles contain the individual node name in the format <SID>_HDB<instance>_<node>.

Procedure

1. Go to the HANA instance profile directory:

   [root]# cd /hana/shared/<SID>/profile

2. Edit the instance profile:

   [root]# vi <SID>_HDB<instance>_<hostname>

   Ensure that Autostart is set to 0.

3. Repeat the previous steps for each HANA instance that will be managed as part of the HA cluster.