Red Hat Summitこのコンテンツは選択した言語では利用できません。
Configuring HA clusters to manage SAP NetWeaver or SAP S/4HANA Application server instances using the RHEL HA Add-On
Abstract
Making open source more inclusive
Red Hat is committed to replacing problematic language in our code and documentation. We are beginning with these four terms: master, slave, blacklist, and whitelist. Due to the enormity of this endeavor, these changes will be gradually implemented over upcoming releases. For more details on making our language more inclusive, see our CTO Chris Wright’s message.
Providing feedback on Red Hat documentation
We appreciate your feedback on our documentation. Let us know how we can improve it.
Submitting feedback through Jira (account required)
- Make sure you are logged in to the Jira website.
- Click on this link to provide feedback.
- Enter a descriptive title in the Summary field.
- Enter your suggestion for improvement in the Description field. Include links to the relevant parts of the documentation.
- Click Create at the bottom of the dialogue.
Chapter 1. Overview
1.1. Introduction
SAP NetWeaver or SAP S/4HANA based systems play an important role in many business processes; thus, it is critical to ensure the continuous and reliable availability of those systems to the business. This can be achieved by using HA clustering for managing the instances of such SAP NetWeaver or SAP S/4HANA systems.
The underlying idea of HA clustering is a fairly simple one: not a single large machine bears all of the load and risk, but rather one or more machines automatically drop in as an instant full replacement for the service or the machine that has failed. In the best case, this replacement process causes no interruption to the systems' users.
1.2. Audience
Designing highly available solutions and implementing them based on SAP NetWeaver or SAP S/4HANA can be very complex, so deep knowledge about each layer of the infrastructure and every aspect of the deployment is needed to ensure reliable, repeatable, accurate, and quick automated actions.
This document is intended for SAP and Red Hat certified or trained administrators and consultants who already have experience setting up SAP NetWeaver or S/4HANA application server instances and HA clusters using the RHEL HA add-on or other clustering solutions. Access to both the SAP Support Portal and the Red Hat Customer Portal is required to be able to download software and additional documentation.
Red Hat Consulting is highly recommended to set up the cluster and customize the solution to meet customers' data center requirements, which are normally more complex than the solution presented in this document.
1.3. Concepts
1.3.1. SAP NetWeaver or S/4HANA High Availability
A typical SAP NetWeaver or S/4HANA environment consists of three distinctive components:
-
SAP
(A)SCSinstance - SAP application server instances (Primary Application Server (PAS) and Additional Application Server (AAS) instances)
- Database instance
The (A)SCS instance and the database instance are single points of failure (SPOF); therefore, it is important to ensure they are protected by an HA solution to avoid data loss or corruption and unnecessary outages of the SAP system. For more information on SPOF, please refer to Single point of failure.
For the application servers, the enqueue lock table that is managed by the enqueue server is the most critical component. To protect it, SAP has developed the “Enqueue Replication Server” (ERS), which maintains a backup copy of the enqueue lock table. While the (A)SCS is running on one server, the ERS always needs to maintain a copy of the current enqueue table on another server.
This document describes how to set up a two-node or three-node HA cluster solution for managing (A)SCS and ERS instances that conforms to the guidelines for high availability that have been established by both SAP and Red Hat. The HA solution can either be used for the “Standalone Enqueue Server” (ENSA1) that is typically used with SAP NetWeaver or the “Standalone Enqueue Server 2” (ENSA2) that is used by SAP S/4HANA.
Additionally, it also provides guidelines for setting up HA cluster resources for managing other SAP instance types, like Primary Application Server (PAS) or Additional Application Server (AAS) instances that can either be managed as part of the same HA cluster or on a separate HA cluster.
1.3.2. ENSA1 vs. ENSA2
1.3.2.1. Standalone Enqueue Server (ENSA1)
In case there is an issue with the (A)SCS instance, for the Standalone Enqueue Server (ENSA1), it is required that the (A)SCS instance "follows” the ERS instance. That is, an HA cluster has to start the (A)SCS instance on the host where the ERS instance is currently running. Until the host where the (A)SCS instance was running has been fenced, it can be noticed that both instances stay running on that same node. When the HA cluster node where the (A)SCS instance was previously running is back online, the HA cluster should move the ERS instance to that HA cluster node so that Enqueue Replication can resume.
The following diagram shows the typical architecture of a Pacemaker HA cluster for managing SAP NetWeaver setups with the Standalone Enqueue Server (ENSA1).
Even though the diagram shows that it is optionally possible to also have Primary and Additional Application Server (PAS/AAS) instances managed on separate servers, it is also supported to have these instances running on the same HA cluster nodes as the (A)SCS and ERS instances and have them managed by the cluster.
Please see the following SAP documentation for more information on how the Standalone Enqueue Server (ENSA1) works: Standalone Enqueue Server.
1.3.2.2. Standalone Enqueue Server 2 (ENSA2)
As shown above with ENSA1, if there is a failover, the Standalone Enqueue Server is required to "follow" the Enqueue Replication Server. That is, the HA software had to start the (A)SCS instance on the host where the ERS instance is currently running.
In contrast to the Standalone Enqueue Server (ENSA1), the new Standalone Enqueue Server 2 (ENSA2) and Enqueue Replicator 2 no longer have these restrictions, which means that the ASCS instance can either be restarted on the same cluster node in case of a failure. Or it can also be moved to another HA cluster node, which doesn’t have to be the HA cluster node where the ERS instance is running. This makes it possible to use a multi-node HA cluster setup with more than two HA cluster nodes when Standalone Enqueue Server 2 (ENSA2) is used.
When using more than two HA cluster nodes, the ASCS will failover to a spare node, as illustrated in the following picture:
For more information on ENSA2, please refer to SAP Note 2630416 - Support for Standalone Enqueue Server 2.
The following diagram shows the architecture of a three-node cluster that can be used for managing SAP S/4HANA setups with the Standalone Enqueue Server 2 (ENSA2).
Even though the diagram shows that it is optionally possible to also have Primary and Additional Application Server (PAS/AAS) instances managed on separate servers, it is also supported to have these instances running on the same HA cluster nodes as the ASCS and ERS instances and have them managed by the cluster.
For SAP S/4HANA, it is also possible to use a “cost-optimized” HA cluster setup, where the cluster nodes used for managing the HANA System Replication setup are also used for managing the ASCS and ERS instances. Please see Configuring a Cost-Optimized SAP S/4HANA HA cluster (HANA System Replication + ENSA2) using the RHEL HA Add-On, for more information.
1.4. Resource Agents
The following resource agents are provided on RHEL 8 for managing different instance types of SAP environments via the resource-agents-sap RPM package.
1.4.1. SAPInstance resource agent
The SAPInstance resource agent can be used for managing SAP application server instances using the SAP Start Service that is part of the SAP Kernel. In addition to the (A)SCS, ERS, PAS, and AAS instances, it can also be used for managing other SAP instance types, like standalone SAP Web Dispatcher or standalone SAP Gateway instances (see How to manage standalone SAP Web Dispatcher instances using the RHEL HA Add-On for information on how to configure a pacemaker resource for managing such instances).
All operations of the SAPInstance resource agent are done by using commands provided by the SAP Startup Framework, which communicate with the sapstartsrv process of each SAP instance. sapstartsrv knows 4 status colors:
| Color | Meaning |
|---|---|
| GREEN | Everything is fine. |
| YELLOW | Something is wrong, but the service is still working. |
| RED | The service does not work. |
| GRAY | The service has not been started. |
The SAPInstance resource agent will interpret GREEN and YELLOW as OK, while statuses RED and GRAY are reported as NOT_RUNNING to the cluster.
The versions of the SAPInstance resource agent shipped with RHEL 8 also support SAP instances that are managed by the systemd-enabled SAP Startup Framework (see The Systemd-Based SAP Startup Framework for further details).
1.4.1.1. Important SAPInstance resource agent parameters
| Attribute Name | Required | Default value | Description |
|---|---|---|---|
|
| yes | null |
The full SAP instance profile name ( |
|
| no | null | The full path to the SAP Start Profile (with SAP NetWeaver 7.1 and newer, the SAP Start profile is identical to the instance profile). |
|
| no | false |
Only used for |
|
| no | null |
The full qualified path where to find |
|
| no | null | The full qualified path where to find the SAP START profile (only needed if the default location for the instance profiles has been changed). |
|
| no | false |
The SAPInstance resource agent tries to recover a failed start attempt automatically one time. This is done by killing running instance processes, removing the |
|
| no |
|
The list of services of an SAP instance that need to be monitored to determine the health of the instance. To monitor more/less, or other services that |
The full list of parameters can be obtained by running pcs resource describe SAPInstance.
1.4.2. SAPDatabase resource agent
The SAPDatabase resource agent can be used to manage single Oracle, IBM DB2, SAP ASE, or MaxDB database instances as part of a SAP NetWeaver based HA cluster setup. For more information, refer to Support Policies for RHEL High Availability Clusters - Management of SAP NetWeaver in a Cluster for the list of supported database versions on RHEL 8.
The SAPDatabase resource agent does not run any database commands directly. It uses the SAP Host Agent to control the database. Therefore, the SAP Host Agent must be installed on each cluster node.
Since the SAPDatabase resource agent only provides basic functionality for managing database instances, it is recommended to use the HA features of the databases instead (for example, Oracle RAC and IBM DB2 HA/DR) if more HA capabilities are required for the database instance.
For S/4HANA HA setups, it is recommended to use HANA System Replication to make the HANA instance more robust against failures. The HANA System Replication HA setup can either be done using a separate cluster, or alternatively, it is also possible to use a “cost-optimized” S/4HANA HA setup where the ASCS and ERS instances are managed by the same HA cluster that is used for managing the HANA System Replication setup.
1.4.2.1. Important SAPDatabase resource agent parameters
| Attribute Name | Required | Default value | Description |
|---|---|---|---|
| SID | yes | null | The unique database system identifier (usually identical to the SAP SID). |
| DBTYPE | yes | null |
The type of database to manage. Valid values are: |
| DBINSTANCE | no | null | Must be used for special database implementations when the database instance name is not equal to the SID (e.g., Oracle DataGuard). |
| DBOSUSER | no |
ADA=taken from | The parameter can be set if the database processes on the operating system level are not executed with the default user of the used database type. |
| STRICT_MONITORING | no | false |
This controls how the resource agent monitors the database. If set to |
| MONITOR_SERVICES | no |
|
Defines which services are monitored by the |
| AUTOMATIC_RECOVER | no | false |
If you set this to |
The full list of parameters can be obtained by running pcs resource describe SAPDatabase.
1.5. Multi-SID Support (optional)
The setup described in this document can also be used to manage the (A)SCS/ERS instances for multiple SAP environments (Multi-SID) within the same HA cluster. For example, SAP products that contain both ABAP and Java application server instances (like SAP Solution Manager) could be candidates for a Multi-SID cluster.
However, some additional considerations need to be taken into account for such setups.
1.5.1. Unique SID and Instance Number
To avoid conflicts, each pair of (A)SCS/ERS instances must use a different SID, and each instance must use a unique Instance Number even if they belong to a different SID.
1.5.2. Sizing
Each HA cluster node must meet the SAP requirements for sizing to support multiple instances.
1.5.3. Installation
For each (A)SCS/ERS pair, please repeat all the steps documented in sections 4.5, 4.6, and 4.7. Each (A)SCS/ERS pair will failover independently, following the configuration rules.
With the default pacemaker configuration for RHEL 8, certain failures of resource actions (for example, the stop of a resource fails) will cause the cluster node to be fenced. This means that, for example, if the stop of the resource for one (A)SCS instance on a HA cluster node fails, it would cause an outage for all other resources running on the same HA cluster node. Please see the description of the on-fail property for monitoring operations in Configuring and managing high availability clusters - Chapter 21. Resource monitoring operations for options on how to modify this behavior.
1.6. Support policies
Chapter 2. Requirements
2.1. Subscriptions and repositories
It is important to keep the subscription, kernel, and patch level identical on all cluster nodes and to ensure that the correct repositories are enabled.
Check out the following documentation for guidelines on how to enable the required subscriptions and repositories for running SAP NetWeaver or SAP S/4HANA application servers on RHEL 8 and have them managed by the RHEL HA Add-On: RHEL for SAP Subscriptions and Repositories.
2.2. Storage requirements
The directories used by a SAP S/4HANA installation that is managed by the cluster must be set up according to the guidelines provided by SAP. See SAP Directories for more information.
2.2.1. Local directories
As per SAP’s guidance, the /usr/sap/, /usr/sap/SYS/, and /usr/sap/<SAPSID>/ directories should be created locally on each node. While /usr/sap/ will contain some additional files and directories after the installation of the SAP system that are specific to the node (for example, /usr/sap/sapservices, and /usr/sap/hostctrl), /usr/sap/SYS/ only contains symlinks to other files and directories, and /usr/sap/<SAPSID>/ is primarily used as a mountpoint for the instance-specific directories.
2.2.2. Instance Specific Directories
For the (A)SCS, ERS, and any other application server instance that is managed by the cluster, the instance-specific directory must be created on a separate SAN LUN or NFS export that can be mounted by the cluster as a local directory on the node where an instance is supposed to be running. For example:
-
(A)SCS:/usr/sap/<SAPSID>/ASCS<Ins#>/ -
ERS:/usr/sap/<SAPSID>/ERS<Ins#>/ -
App Server:
/usr/sap/<SAPSID>/D<Ins#>/
The cluster configuration must include resources for managing the filesystems for the instance directories as part of the resource group that is used to manage the instance and the virtual IP so that the cluster can automatically mount the filesystem on the node where the instance should be running.
When using SAN LUNs for instance-specific directories, customers must use HA-LVM to ensure that the instance directories can only be mounted on one node at a time.
The resources for managing the logical volumes (if SAN LUNS are used) and the filesystems must always be configured before the resource that is used for managing the SAP instance to ensure that the filesystem is mounted when the cluster attempts to start the instance itself.
Using a shared file system (for example, GFS2) to host all the instance-specific directories and make them available on all cluster nodes at the same time is not supported for the solution described in this document.
When using NFS exports for specific directories, if the directories are created on the same directory tree on an NFS file server, such as Azure NetApp Files (ANF) or Amazon EFS, the option force_unmount=safe must be used when configuring the Filesystem resource. This option will ensure that the cluster only stops the processes running on the specific NFS export instead of stopping all processes running on the directory tree where the exports have been created (see During failover of a pacemaker resource, a Filesystem resource kills processes not using the filesystem for more information).
2.3. Fencing/STONITH
As documented at Support Policies for RHEL High Availability Clusters - General Requirements for Fencing/STONITH, a working Fencing/STONITH device must be enabled on each cluster node in order for an HA cluster setup using the RHEL HA Add-on to be fully supported.
Which Fencing/STONITH device to use depends on the platform the cluster is running on. Please check out the Fencing/STONITH section in the Support Policies for RHEL High Availability Clusters for recommendations on fencing agents, or consult with your hardware or cloud provider to find out which fence device to use on their platform.
Using fence_scsi/fence_mpath as the fencing device for HA cluster setups for managing SAP NetWeaver/S/4HANA application server instances is not a supported option since, as documented in Support Policies for RHEL High Availability Clusters - fence_scsi and fence_mpath these fence devices can only be used for cluster setups that manage shared storage, which is simultaneously accessed by multiple clients for reading and writing. Since the main purpose of a HA cluster for managing SAP NetWeaver/S/4HANA is to manage the SAP application server instances and not the shared directories that are needed in such environments, using fence_scsi/fence_mpath could result in the SAP instances not being stopped in case a node needs to be fenced (since fence_scsi/fence_mpath normally only block access to the storage devices managed by the cluster).
2.4. Quorum
While pacemaker provides some built-in mechanisms to determine if a cluster is quorate or not, in some situations it might be desirable to add additional “quorum devices” in the cluster setup to help the cluster determine which side of the cluster should stay up and running in case a “split-brain” situation occurs.
For HA cluster setups that are used for managing SAP Application server instances, a quorum device is not required by default, but it is possible to add quorum devices to such setups if needed.
The options for setting up quorum devices vary depending on the configuration. Please review the following guidelines for more information:
Chapter 3. Installing SAP application server instances
3.1. Configuration options used in this document
Below are the configuration options that will be used for instances in this document. Please adapt these options according to your local requirements.
For the HA cluster nodes and the (A)SCS and ERS instances managed by the HA cluster, the following values are used:
For the optional primary application server (PAS) and additional application server (AAS) instances, the following values are used:
3.2. Preparing the cluster nodes for installation of the SAP instances
Before starting the installation, ensure that:
- RHEL 8 is installed and configured on all HA cluster nodes according to the recommendations from SAP and Red Hat for running SAP application server instances on RHEL 8.
- The RHEL for SAP Applications or RHEL for SAP Solutions subscriptions are activated, and the required repositories are enabled on all HA cluster nodes, as documented in RHEL for SAP Subscriptions and Repositories.
- Shared storage and instance directories are present at the correct mount points.
- The virtual hostnames and IP addresses used by the SAP instances can be resolved in both directions, and the virtual IP addresses must be accessible.
- The SAP installation media are accessible on each HA cluster node where a SAP instance will be installed.
These setup steps can be partially automated using Ansible and rhel-system-roles-sap system roles. For more information on this, please check out Red Hat Enterprise Linux System Roles for SAP.
3.3. Installing SAP instances
Using software provisioning manager (SWPM), install instances in the following order:
-
(A)SCSinstance -
ERSinstance - DB instance
- PAS instance
- AAS instances
The following sections just provide some specific recommendations that should be followed when installing SAP instances that will be managed by the HA cluster setup described in this document. Please check the official SAP installation guides for detailed instructions on how to install SAP NetWeaver or S/4HANA application server instances.
3.3.1. Installing (A)SCS on node1
The local directories and mount points required by the SAP instance must be created on the HA cluster node where the (A)SCS instance will be installed:
/sapmnt/ /usr/sap/ /usr/sap/SYS/ /usr/sap/trans/ /usr/sap/S4H/ASCS20/
/sapmnt/
/usr/sap/
/usr/sap/SYS/
/usr/sap/trans/
/usr/sap/S4H/ASCS20/The shared directories and the instance directory must be manually mounted before starting the installation.
Also, the virtual IP address for the (A)SCS instance must be enabled on node 1, and it must have been verified that the virtual hostname for the ERS instance resolves to the virtual IP address.
When running the SAP installer, please make sure to specify the virtual hostname that should be used for the (A)SCS instance:
./sapinst SAPINST_USE_HOSTNAME=s4ascs
[root@node1]# ./sapinst SAPINST_USE_HOSTNAME=s4ascs
Select the High-Availability System option for the installation of the (A)SCS instance:
3.3.2. Installing ERS on node2
The local directories and mount points required by the SAP instance must be created on the HA cluster node where the ERS instance will be installed:
/sapmnt/ /usr/sap/ /usr/sap/SYS/ /usr/sap/trans/ /usr/sap/S4H/ERS29
/sapmnt/
/usr/sap/
/usr/sap/SYS/
/usr/sap/trans/
/usr/sap/S4H/ERS29The shared directories and the instance directory must be manually mounted before starting the installation.
Also, the virtual IP address for the ERS instance must be enabled on node 2, and it must have been verified that the virtual hostname for the ERS instance resolves to the virtual IP address.
Make sure to specify the virtual hostname for the ERS instance when starting the installation:
./sapinst SAPINST_USE_HOSTNAME=s4ers
[root@node2]# ./sapinst SAPINST_USE_HOSTNAME=s4ers
Select the High-Availability System option for the installation of the ERS instance:
3.3.3. Installing primary/additional application server instances
The local directories and mount points required by the SAP instance must be created on the HA cluster node where the primary or additional application server instance(s) will be installed:
The shared directories and the instance directory must be manually mounted before starting the installation.
Also, the virtual IP address for the application server instance must be enabled on the HA cluster node, and it must have been verified that the virtual hostname for the application server instance resolves to the virtual IP address.
Specify the virtual hostname for the instance when starting the installer:
[root@node<X>]# ./sapinst SAPINST_USE_HOSTNAME=<virtual hostname of instance>
[root@node<X>]# ./sapinst SAPINST_USE_HOSTNAME=<virtual hostname of instance>Select the High-Availability System option for the installation of the application server instance.
3.4. Post Installation
3.4.1. (A)SCS profile modification
The (A)SCS instance profile has to be modified to prevent the automatic restart of the enqueue server process by the sapstartsrv process of the instance, since the instance will be managed by the cluster.
To modify the (A)SCS instance profile, run the following command:
sed -i -e 's/Restart_Program_01/Start_Program_01/' /sapmnt/S4H/profile/S4H_ASCS20_s4ascs
[root@node1]# sed -i -e 's/Restart_Program_01/Start_Program_01/' /sapmnt/S4H/profile/S4H_ASCS20_s4ascs3.4.2. ERS profile modification
The ERS instance profile has to be modified to prevent the automatic restart of the enqueue replication server process by the sapstartsrv of the instance since the ERS instance will be managed by the cluster.
To modify the ERS instance profile, run the following command:
sed -i -e 's/Restart_Program_00/Start_Program_00/' /sapmnt/S4H/profile/S4H_ERS29_s4ers
[root@node2]# sed -i -e 's/Restart_Program_00/Start_Program_00/' /sapmnt/S4H/profile/S4H_ERS29_s4ers3.4.3. Updating the /usr/sap/sapservices file
To prevent the SAP instances that will be managed by the HA cluster to be started outside of the control of the HA cluster, make sure the following lines are commented out in the /usr/sap/sapservices file on all cluster nodes:
3.4.4. Creating mount points for the instance specific directories on the failover node
The mount points where the instance-specific directories will be mounted have to be created and the user and group ownership must be set to root:root on all HA cluster nodes:
3.4.5. Verifying that the SAP instances can be started and stopped on all cluster nodes
Stop the (A)SCS and ERS instances using sapcontrol`, unmount the instance specific directories and then mount them on the other node:
/usr/sap/S4H/ASCS20/ /usr/sap/S4H/ERS29/ /usr/sap/S4H/D<Ins#>/
/usr/sap/S4H/ASCS20/
/usr/sap/S4H/ERS29/
/usr/sap/S4H/D<Ins#>/
Verify that manual starting and stopping of all SAP instances using sapcontrol works on all HA cluster nodes and that the SAP instances are running correctly using the tools provided by SAP.
3.4.6. Verifying that the correct version of SAP Host Agent is installed on all HA cluster nodes
Run the following command on each cluster node to verify that SAP Host Agent has the same version and meets the minimum version requirement:
[root@node<x>]# /usr/sap/hostctrl/exe/saphostexec -version
[root@node<x>]# /usr/sap/hostctrl/exe/saphostexec -versionPlease check SAP Note 1031096-Installing Package SAPHOSTAGENT in case SAP Host Agent needs to be updated.
3.4.7. Installing permanent SAP license keys
To ensure that the SAP instances continue to run after a failover, it might be necessary to install several SAP license keys based on the hardware key of each cluster node. Please see SAP Note 1178686 - Linux: Alternative method to generate a SAP hardware key for more information.
3.4.8. Additional changes required when using systemd enabled SAP instances
If the SAP instances that will be managed by the cluster are systemd enabled, additional configuration changes are required to ensure that systemd does not interfere with the management of the SAP instances by the HA cluster. Please check out section 2. Red Hat HA Solutions for SAP in The Systemd-Based SAP Startup Framework for information.
Chapter 4. Setting up the cluster
4.1. Performing the basic cluster installation on each node
Please refer to Configuring and managing high availability clusters, to first set up a pacemaker cluster.
Please make sure to follow the guidelines in Support Policies for RHEL High Availability Clusters - General Requirements for Fencing/STONITH for the fencing/STONITH setup. Information about the fencing/STONITH agents supported for different platforms is available at Cluster Platforms and Architectures.
The rest of this guide will assume that following things are working properly:
- Pacemaker cluster is configured according to documentation and has proper and working fencing (see How to test fence devices and fencing configuration in a Red Hat High Availability cluster?, for information on procedures for verifying that fencing is working properly).
- Enqueue replication between the (A)SCS and ERS instances has been manually tested, as explained in Setting up Enqueue Replication Server failover.
- All HA cluster nodes are subscribed to RHEL for SAP Applications or RHEL for SAP Solutions, and the required repositories are enabled as explained in RHEL for SAP Subscriptions and Repositories.
4.2. Configuring general cluster properties
To avoid unnecessary failovers of the resources set the default values for the resource-stickiness and migration-threshold parameters by running the following commands on one cluster node:
pcs resource defaults update resource-stickiness=1 pcs resource defaults update migration-threshold=3
[root@node1]# pcs resource defaults update resource-stickiness=1
[root@node1]# pcs resource defaults update migration-threshold=3
The resource-stickiness=1 will encourage the resource to stay running where it is, while migration-threshold=3 will cause the resource to move to a new node after 3 failures. 3 is generally sufficient in preventing the resource from prematurely failing over to another node. This also ensures that the resource failover time stays within a controllable limit.
4.3. Installing the resource-agents-sap package on all cluster nodes
The SAPInstance and SAPDatabase resource agents are provided via a separate resource-agents-sap package. Run the following command to install it on each HAcluster node:
[root@node<x>]# dnf install resource-agents-sap
[root@node<x>]# dnf install resource-agents-sap4.5. Configuring the (A)SCS resource group
4.5.1. Creating resource for managing the virtual IP address of the (A)SCS instance
To allow application servers and other clients to connect to the (A)SCS instance on the HA cluster node where the instance is currently running, the virtual IP address (VIP) that has been assigned to the instance needs to be moved by the cluster when the (A)SCS instance is moving from one HA cluster node to another.
For this, a resource that manages the VIP needs to be created as part of the resource group that is used for managing the (A)SCS instance.
Please use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running.
On physical servers or VMs the resource can be created using the IPaddr2 resource agent:
pcs resource create s4h_vip_ascs20 IPaddr2 ip=192.168.200.101 --group s4h_ASCS20_group
[root@node1]# pcs resource create s4h_vip_ascs20 IPaddr2 ip=192.168.200.101 --group s4h_ASCS20_group4.5.2. Creating resource for managing the (A)SCS instance directory
Since SAP requires that the instance directory be only available on the HA cluster node where the instance is supposed to be running it is necessary to set up HA cluster resources for managing the filesystems that are used for the instance directories.
Even if the instance directories are stored on NFS it is still necessary to create the resource to allow the HA cluster to only mount the NFS export on the HA cluster node where the SAP instance should be running.
4.5.2.1. NFS
If the instance directory for the (A)SCS instance is located on NFS, the resource to have it managed as part of the resource group for managing the (A)SCS instance can be created with the following command:
pcs resource create s4h_fs_ascs20 Filesystem device='<NFS_Server>:<s4h_ascs20_nfs_share>' directory=/usr/sap/S4H/ASCS20 fstype=nfs force_unmount=safe fast_stop=no --group s4h_ASCS20_group \ op start interval=0 timeout=60 \ op stop interval=0 timeout=120 \ op monitor interval=200 timeout=40
[root@node1]# pcs resource create s4h_fs_ascs20 Filesystem device='<NFS_Server>:<s4h_ascs20_nfs_share>' directory=/usr/sap/S4H/ASCS20 fstype=nfs force_unmount=safe fast_stop=no --group s4h_ASCS20_group \
op start interval=0 timeout=60 \
op stop interval=0 timeout=120 \
op monitor interval=200 timeout=404.5.2.2. HA-LVM
When using HA-LVM to manage the instance directory for the (A)SCS instance, it must be configured according to the guidelines in the article What is a Highly Available LVM (HA-LVM) configuration and how do I implement it?.
First, an LVM-activate cluster resource must be added, followed by a Filesystem cluster resource:
pcs resource create s4h_fs_ascs20_lvm LVM-activate volgrpname='<ascs_volume_group>' vg_access_mode=system_id --group s4h_ASCS20_group pcs resource create s4h_fs_ascs20 Filesystem device='/dev/mapper/<ascs_logical_volume>' directory=/usr/sap/S4H/ASCS20 fstype=ext4 fast_stop=no --group s4h_ASCS20_group
[root@node1]# pcs resource create s4h_fs_ascs20_lvm LVM-activate volgrpname='<ascs_volume_group>' vg_access_mode=system_id --group s4h_ASCS20_group
[root@node1]# pcs resource create s4h_fs_ascs20 Filesystem device='/dev/mapper/<ascs_logical_volume>' directory=/usr/sap/S4H/ASCS20 fstype=ext4 fast_stop=no --group s4h_ASCS20_group4.5.3. Creating resource for managing the (A)SCS instance
resource-stickiness=5000 is used to balance out the failover constraint with the ERS resource so the resource stays on the node where it started and doesn’t migrate around the cluster uncontrollably.
migration-threshold=1 ensures that the (A)SCS instance fails over to another node when an issue is detected instead of restarting on the same HA cluster node. For ENSA2 setups this option is not required, since with ENSA2 restarting the (A)SCS instance on the same HA cluster node is allowed.
When all resources for the resource group have been created, add a resource stickiness to the group to ensure that the (A)SCS instance will stay on a HA cluster node if possible:
pcs resource meta s4h_ASCS20_group resource-stickiness=3000
[root@node1]# pcs resource meta s4h_ASCS20_group resource-stickiness=30004.6. Configuring the ERS resource group
4.6.1. Creating resource for managing the virtual IP address of the ERS instance
Even though the ERS instance is not directly accessed by application servers, it still requires a virtual IP to allow SAP management tools to connect to the ERS instance on the HA cluster node where the instance is currently running. Therefore, the virtual IP address (VIP) that has been assigned to the instance needs to be moved by the cluster when the (A)SCS instance is moving from one HA cluster node to another.
For this, a resource that manages the VIP needs to be created as part of the resource group that is used for managing the ERS instance.
Please use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running.
On physical servers or VMs the resource can be created using the IPaddr2 resource agent:
pcs resource create s4h_vip_ers29 IPaddr2 ip=192.168.200.102 --group s4h_ERS29_group
[root@node1]# pcs resource create s4h_vip_ers29 IPaddr2 ip=192.168.200.102 --group s4h_ERS29_group4.6.2. Creating resource for managing the ERS instance directory
Since SAP requires that the instance directory be only available on the HA cluster node where the instance is supposed to be running it is necessary to set up HA cluster resources for managing the filesystems that are used for the instance directories.
Even if the instance directories are stored on NFS it is still necessary to create the resource to allow the HA cluster to only mount the NFS export on the HA cluster node where the SAP instance should be running.
4.6.2.1. NFS
If the instance directory for the ERS instance is located on NFS, the resource to have it managed as part of the resource group for managing the ERS instance can be created with the following command:
pcs resource create s4h_fs_ers29 Filesystem device='<NFS_Server>:<s4h_ers29_nfs_share>' directory=/usr/sap/S4H/ERS29 fstype=nfs force_unmount=safe fast_stop=no --group s4h_ERS29_group \ op start interval=0 timeout=60 \ op stop interval=0 timeout=120 \ op monitor interval=200 timeout=40
[root@node1]# pcs resource create s4h_fs_ers29 Filesystem device='<NFS_Server>:<s4h_ers29_nfs_share>' directory=/usr/sap/S4H/ERS29 fstype=nfs force_unmount=safe fast_stop=no --group s4h_ERS29_group \
op start interval=0 timeout=60 \
op stop interval=0 timeout=120 \
op monitor interval=200 timeout=404.6.2.2. HA-LVM
When using HA-LVM to manage the instance directory for the ERS instance it must be configured according to the guidelines in the article What is a Highly Available LVM (HA-LVM) configuration and how do I implement it?.
First, an LVM-activate cluster resource must be added, followed by a Filesystem cluster resource:
pcs resource create s4h_fs_ers29_lvm LVM-activate volgrpname='<ers_volume_group>' vg_access_mode=system_id --group s4h_ERS29_group # pcs resource create s4h_fs_ers29 Filesystem device='/dev/mapper/<ers_logical_volume>' directory=/usr/sap/S4H/ERS29 fstype=ext4 fast_stop=no --group s4h_ERS29_group
[root@node1]# pcs resource create s4h_fs_ers29_lvm LVM-activate volgrpname='<ers_volume_group>' vg_access_mode=system_id --group s4h_ERS29_group
# pcs resource create s4h_fs_ers29 Filesystem device='/dev/mapper/<ers_logical_volume>' directory=/usr/sap/S4H/ERS29 fstype=ext4 fast_stop=no --group s4h_ERS29_group4.6.3. Creating resource for managing the ERS instance
Create the ERS instance cluster resource:
pcs resource create s4h_ers29 SAPInstance InstanceName="S4H_ERS29_rhers" START_PROFILE=/sapmnt/S4H/profile/S4H_ERS29_rhers AUTOMATIC_RECOVER=false IS_ERS=true --group s4h_ERS29_group \ op monitor interval=20 on-fail=restart timeout=60 \ op start interval=0 timeout=600 \ op stop interval=0 timeout=600
[root@node1]# pcs resource create s4h_ers29 SAPInstance InstanceName="S4H_ERS29_rhers" START_PROFILE=/sapmnt/S4H/profile/S4H_ERS29_rhers AUTOMATIC_RECOVER=false IS_ERS=true --group s4h_ERS29_group \
op monitor interval=20 on-fail=restart timeout=60 \
op start interval=0 timeout=600 \
op stop interval=0 timeout=600
IS_ERS=true attribute is mandatory for ENSA1 deployment. More information about IS_ERS can be found in How does the IS_ERS attribute work on a SAP NetWeaver cluster with Standalone Enqueue Server (ENSA1 and ENSA2)?.
4.7. Creating constraints
4.7.1. Creating colocation constraint for (A)SCS and ERS resource groups
Resource groups s4h_ASCS20_group and s4h_ERS29_group should try to avoid running on the same node. The order of the groups matters.
pcs constraint colocation add s4h_ERS29_group with s4h_ASCS20_group -5000
[root@node1]# pcs constraint colocation add s4h_ERS29_group with s4h_ASCS20_group -50004.7.2. Creating location constraint for (A)SCS resource (ENSA1 only)
When using ENSA1, it must be ensured that the (A)SCS instance moves to the node where the ERS instance is running when failover is happening.
pcs constraint location s4h_ascs20 rule score=2000 runs_ers_S4H eq 1
[root@node1]# pcs constraint location s4h_ascs20 rule score=2000 runs_ers_S4H eq 14.7.3. Creating order constraint for (A)SCS and ERS resource groups
Stop the resource group for managing the ERS instance after the resource group for managing the (A)SCS instance has been started, if pacemaker decides to start the resource group for managing the (A)SCS instance at the same time as stopping the resource group for managing the ERS instance:
pcs constraint order start s4h_ASCS20_group then stop s4h_ERS29_group symmetrical=false kind=Optional
[root@node1]# pcs constraint order start s4h_ASCS20_group then stop s4h_ERS29_group symmetrical=false kind=Optional
Since symmetrical=false and kind=Optional are used, there can be situation where this constraint won’t take effect. For more information, refer to Determining the order in which cluster resources are run.
4.7.4. Creating order constraints for /sapmnt resource managed by cluster
If the shared filesystem /sapmnt is managed by the cluster, then the following constraints ensure that resource groups used for managing the (A)SCS and ERS instances are started only after the /sapmnt filesystem is available:
pcs constraint order s4h_fs_sapmnt-clone then s4h_ASCS20_group pcs constraint order s4h_fs_sapmnt-clone then s4h_ERS29_group
[root@node1]# pcs constraint order s4h_fs_sapmnt-clone then s4h_ASCS20_group
[root@node1]# pcs constraint order s4h_fs_sapmnt-clone then s4h_ERS29_group4.8. Configuring cluster resource group for managing database instances (optional)
When using the HA cluster for managing a SAP NetWeaver based SAP product that is still using a legacy database like Oracle, IBM DB2, SAP ASE or SAP MaxDB, it is possible to also have the database instance managed by the cluster.
This chapter shows an example of how to set up a resource group for managing a single database instance using the SAPDatabase resource agent and the virtual IP address and the file system required by it.
The example setup described in this chapter uses the SAPSID RH1 instead of S4H, since the SAPDatabase resource agent cannot be used with S/4HANA setups.
4.8.1. Creating resource for managing the virtual IP address of the database instance
To create the resource for managing the virtual IP address used for accessing the database instance that will be part of the rh1_SAPDatabase_group use the following command:
pcs resource create rh1_vip_db IPaddr2 ip=192.168.200.115 --group rh1_SAPDatabase_group
[root]# pcs resource create rh1_vip_db IPaddr2 ip=192.168.200.115 --group rh1_SAPDatabase_group4.8.2. Creating resource for managing the directories used by the database instance
The directories used by the database instance can only be mounted on the HA cluster node where the database instance is supposed to run to avoid that the database instance can accidentally be started on another system at the same time which would cause data corruption.
Depending on the way the storage for managing the directories used by the database instance is set up, different methods for creating the resources for managing the database directories have to be used.
Even if the instance directories are stored on NFS it is still necessary to create the resource to allow the HA cluster to only mount the NFS export on the HA cluster node where the database instance should be running.
4.8.2.1. NFS
If the directories used by the database instance are located on NFS, a resource to have them managed as part of the resource group for managing the database instance must be created for each directory with the following command:
pcs resource create rh1_fs_db Filesystem device='<NFS_Server>:<rh1_db_nfs_share>' directory=/sapdb/RH1 fstype=nfs force_unmount=safe --group rh1_SAPDatabase_group \ op start interval=0 timeout=60 \ op stop interval=0 timeout=120 \ op monitor interval=200 timeout=40
[root@node1]# pcs resource create rh1_fs_db Filesystem device='<NFS_Server>:<rh1_db_nfs_share>' directory=/sapdb/RH1 fstype=nfs force_unmount=safe --group rh1_SAPDatabase_group \
op start interval=0 timeout=60 \
op stop interval=0 timeout=120 \
op monitor interval=200 timeout=404.8.2.2. HA-LVM
When using HA-LVM to manage the directories used by the database instance it must be configured according to the guidelines in the article What is a Highly Available LVM (HA-LVM) configuration and how do I implement it?.
First, an LVM-activate cluster resource must be added followed by a Filesystem cluster resource:
pcs resource create rh1_lvm_db LVM-activate volgrpname=vg_db vg_access_mode=system_id --group rh1_SAPDatabase_group pcs resource create rh1_fs_db Filesystem device=/dev/vg_db/lv_db directory=/sapdb/RH1 fstype=xfs --group rh1_SAPDatabase_group
[root]# pcs resource create rh1_lvm_db LVM-activate volgrpname=vg_db vg_access_mode=system_id --group rh1_SAPDatabase_group
[root]# pcs resource create rh1_fs_db Filesystem device=/dev/vg_db/lv_db directory=/sapdb/RH1 fstype=xfs --group rh1_SAPDatabase_groupIf multiple file systems are used for the database directories, then a separate Filesystem cluster resource must be created for each one.
4.8.3. Configuring SAPDatabase cluster resource
After the resources for the virtual IP address and the filesystems required by the database instance have been added, the SAPDatabase cluster resource that will manage the database instance can be added to the resource group:
pcs resource create rh1_SAPDatabase SAPDatabase DBTYPE="ADA" SID="RH1" STRICT_MONITORING="TRUE" AUTOMATIC_RECOVER="TRUE" --group rh1_SAPDatabase_group
[root]# pcs resource create rh1_SAPDatabase SAPDatabase DBTYPE="ADA" SID="RH1" STRICT_MONITORING="TRUE" AUTOMATIC_RECOVER="TRUE" --group rh1_SAPDatabase_group4.9. Configuring Primary/Additional Application Server (PAS/AAS) resource group (Optional)
This section describes how to configure a resource group for managing the Primary Application Server (PAS) instance and the associated VIP and filesystem for the instance directory, in case the PAS instance should also be managed by the HA cluster. The same configuration can also be used for Additional Application Server (AAS) instances that should be managed by the HA cluster.
4.9.1. Creating resource for managing the Virtual IP address (VIP) for the PAS/AAS instance
To allow other application servers and clients to PAS/AAS instances managed by the HA cluster, the virtual IP address (VIP) that has been assigned to the instance needs to be moved by the cluster when a PAS/AAS instance is moving from one HA cluster node to another.
For this, a resource that manages the VIP needs to be created as part of the resource group that is used for managing a PAS/AAS instance.
Please use the appropriate resource agent for managing the virtual IP address based on the platform on which the HA cluster is running.
On physical servers or VMs the resource can be created using the IPaddr2 resource agent:
pcs resource create s4h_vip_pas_d21 IPaddr2 ip=192.168.200.103 --group s4h_PAS_D21_group
[root@node1]# pcs resource create s4h_vip_pas_d21 IPaddr2 ip=192.168.200.103 --group s4h_PAS_D21_group4.9.2. Creating resource for managing the filesystem for the PAS/AAS instance directory
Since SAP requires that the instance directory be only available on the HA cluster node where the instance is supposed to be running it is necessary to set up HA cluster resources for managing the filesystems that are used for the instance directories.
Even if the instance directories are stored on NFS it is still necessary to create the resource to allow the HA cluster to only mount the NFS export on the HA cluster node where the SAP instance should be running.
4.9.2.1. NFS
If the instance directory for the PAS/AAS instance is located on NFS, the resource to have it managed as part of the resource group for managing the PAS/AAS instance can be created with the following command:
pcs resource create s4h_fs_pas_d21 Filesystem device='<NFS_Server>:<s4h_pas_d21_nfs_share>' directory=/usr/sap/S4H/D21 fstype=nfs force_unmount=safe --group s4h_PAS_D21_group \ op start interval=0 timeout=60 \ op stop interval=0 timeout=120 \ op monitor interval=200 timeout=40
[root@node1]# pcs resource create s4h_fs_pas_d21 Filesystem device='<NFS_Server>:<s4h_pas_d21_nfs_share>' directory=/usr/sap/S4H/D21 fstype=nfs force_unmount=safe --group s4h_PAS_D21_group \
op start interval=0 timeout=60 \
op stop interval=0 timeout=120 \
op monitor interval=200 timeout=404.9.2.2. HA-LVM
When using HA-LVM to manage the instance directory for the PAS/AAS instance it must be configured according to the guidelines in the article What is a Highly Available LVM (HA-LVM) configuration and how do I implement it?.
First, an LVM-activate cluster resource must be added followed by a Filesystem cluster resource:
pcs resource create s4h_lvm_pas_d21 LVM-activate volgrpname=vg_d21 vg_access_mode=system_id --group s4h_PAS_D21_group pcs resource create s4h_fs_pas_d21 Filesystem device=/dev/vg_d21/lv_d21 directory=/usr/sap/S4H/D21 fstype=xfs --group s4h_PAS_D21_group
[root@node1]# pcs resource create s4h_lvm_pas_d21 LVM-activate volgrpname=vg_d21 vg_access_mode=system_id --group s4h_PAS_D21_group
[root@node1]# pcs resource create s4h_fs_pas_d21 Filesystem device=/dev/vg_d21/lv_d21 directory=/usr/sap/S4H/D21 fstype=xfs --group s4h_PAS_D21_group4.9.3. Configuring PAS/AAS SAPInstance cluster resource
To enable pacemakers to manage a PAS or AAS instance, the same SAPInstance resource agent as for (A)SCS/ERS instance can be used. The PAS/AAS instance is, compared to the (A)SCS/ERS instance setup, a simple instance and requires less attributes to be configured.
Check the command below for an example on how to create a PAS instance for the D21 instance and place it at the end of the s4h_PAS_D21_group resource group.
pcs resource create s4h_pas_d21 SAPInstance InstanceName="S4H_D21_s4h-pas" DIR_PROFILE=/sapmnt/S4H/profile START_PROFILE=/sapmnt/S4H/profile/S4H_D21_s4h-pas --group s4h_PAS_D21_group
[root@node1]# pcs resource create s4h_pas_d21 SAPInstance InstanceName="S4H_D21_s4h-pas" DIR_PROFILE=/sapmnt/S4H/profile START_PROFILE=/sapmnt/S4H/profile/S4H_D21_s4h-pas --group s4h_PAS_D21_group4.9.4. Configuring constraints
4.9.4.1. Configuring order constraints for the PAS/AAS resource group(s)
The PAS/AAS instance(s) require the (A)SCS and database instance to be running before it can start properly. The following sections show how to set up the required constraints based on the different types of database instances that can be used by SAP NetWeaver / S/4HANA.
4.9.4.1.1. Deployments with s4h_SAPDatabase_group
For configurations that have one cluster resource group, it will start all resources needed by the database. For example, here the SAPDatabase resource agent is used to manage the database and is part of the database group rh1_SAPDatabase_group. Commands below will create constraints that will start the whole rh1_PAS_D21_group only once the (A)SCS instance was promoted and when the database group rh1_SAPDatabase_group is running.
pcs constraint order rh1_SAPDatabase_group then rh1_PAS_D21_group kind=Optional symmetrical=false pcs constraint order start rh1_ASCS20_group then rh1_PAS_D21_group kind=Optional symmetrical=false
[root@node1]# pcs constraint order rh1_SAPDatabase_group then rh1_PAS_D21_group kind=Optional symmetrical=false
[root@node1]# pcs constraint order start rh1_ASCS20_group then rh1_PAS_D21_group kind=Optional symmetrical=false4.9.4.1.2. Deployments with SAP HANA with System Replication as database
When using SAP HANA database that is configured for system replication (SR) that is managed by cluster, the following constraints will ensure that the whole s4h_PAS_D21_group group will start only once the (A)SCS instance was promoted and when the SAP HANA SAPHana_S4H_02-master was promoted.
pcs constraint order promote SAPHana_S4H_02-master then s4h_PAS_D21_group Kind=Optional symmetrical=false pcs constraint order start s4h_ASCS20_group then s4h_PAS_D21_group Kind=Optional symmetrical=false
[root@node1]# pcs constraint order promote SAPHana_S4H_02-master then s4h_PAS_D21_group Kind=Optional symmetrical=false
[root@node1]# pcs constraint order start s4h_ASCS20_group then s4h_PAS_D21_group Kind=Optional symmetrical=false4.9.4.2. Configuring colocation constraint for PAS and AAS SAPInstance cluster resources (Optional)
To ensure that PAS and AAS instances do not run on the same nodes whenever both nodes are running, you can add a negative colocation constraint with the command below:
pcs constraint colocation add s4h_AAS_D22_group with s4h_PAS_D21_group score=-1000
[root@node1]# pcs constraint colocation add s4h_AAS_D22_group with s4h_PAS_D21_group score=-1000
The score of -1000 is to ensure that if only 1 node is available, the PAS/AAS instance will continue to run on the remaining 1 node. In such a situation, if you would like to keep the AAS instance down, then you can use the score=-INFINITY which will enforce this condition.
4.9.4.3. Creating order constraint for /sapmnt resource managed by cluster
If the shared filesystem /sapmnt is managed by the cluster, then the following constraint ensures that resource groups used for managing the PAS/AAS instance(s) are started only after the /sapmnt filesystem is available:
pcs constraint order s4h_fs_sapmnt-clone then s4h_PAS_D21_group
[root@node1]# pcs constraint order s4h_fs_sapmnt-clone then s4h_PAS_D21_group4.10. Standalone Enqueue Server 2 (ENSA2) Multi-Node Cluster (optional)
For SAP S/4HANA with ENSA2, more than two HA cluster nodes can be used for managing the ASCS and ERS instances. Please use the guidelines in the following section in case an additional cluster node should be added to allow more flexibility for the instances to failover in case there is an issue with the node they were running on.
4.10.1. OS Configuration
Create a node that’s identical to the first two nodes, in terms of resources, subscriptions, OS configuration, etc.
In the example, the hostname of the node is node3. Make sure the /etc/hosts file on each cluster node contains the hostnames and IP addresses of all cluster nodes and also the virtual hostnames and virtual IP addresses of all SAP instances that are managed by the HA cluster.
Make sure to copy the SAP related entries in /etc/services from one of the first two nodes to the third node.
4.10.2. Creating users and groups
Create the users and groups required by the SAP instances identical to the ones used on the other nodes. For example:
4.10.3. Creating local directories and mount points for the shared file systems
Create all mount points required for all instances that should be able to run on the additional HA cluster node:
Make sure to set the user and group ownership for all directories to the same user and group as on the other cluster nodes and copy the contents of the local directories (e. g., /usr/sap/SYS) from one of the other cluster nodes.
If /sapmnt and /usr/sap/trans are statically mounted on the existing HA cluster nodes via /etc/fstab then these file systems must also be added to the /etc/fstab on the additional HA cluster node and the file systems must be mounted afterwards.
If /sapmnt and /usr/sap/trans are managed by the cluster then the cluster configuration must be updated so that the file systems will also be mounted on the additional HA cluster node.
4.10.4. Installing the RHEL HA Add-On and the resource agents for managing SAP instances
For the node to be part of the cluster and to be able to manage the SAP instances, install the required packages:
dnf install pcs pacemaker resource-agents-sap
[root@node3]# dnf install pcs pacemaker resource-agents-sap4.10.5. Adding the node to the cluster
On one node of the existing cluster add the third node:
pcs cluster auth node3 Username: hacluster Password: pcs cluster node add node3
[root@node1]# pcs cluster auth node3
Username: hacluster
Password:
[root@node1]# pcs cluster node add node34.10.6. Updating fencing/STONITH configuration to include the 3rd node
Depending on the STONITH setup, you may need to update the STONITH resource to include the 3rd HA cluster node.
Before moving any resources to the new HA cluster node, please use the following command to verify that it is possible to fence the HA cluster new node from one of the existing HA cluster nodes:
pcs stonith fence node3
[root@node1]# pcs stonith fence node34.10.7. Updating ERS resource configuration
To ensure that the ERS instance stays on the node where it started and doesn’t migrate around the cluster uncontrollably, set resource-stickiness for the resource:
pcs resource meta s4h_ers29 \ resource-stickiness=3000
[root@node1]# pcs resource meta s4h_ers29 \ resource-stickiness=3000To allow SAP admins to manage the SAP application server instances that are controlled by the HA cluster setup described in this documentation using tools like SAP Landscape Management (LaMa), the SAP HA interface must be enabled for each SAP application server instance managed by the HA cluster to ensure that the HA cluster is aware of any action performed by the SAP management tools that will affect the cluster resources used to manage the SAP instances (for example, the HA cluster needs to be notified if a SAP app server instance it manages is being started or stopped via SAP LaMa).
Please check How to enable the SAP HA Interface for SAP ABAP application server instances managed by the RHEL HA Add-On?, for instructions on how to configure the SAP HA interface.
4.12. Enabling cluster to auto-start at boot (optional)
By default the HA cluster is not enabled to auto-start at the boot of the OS, and it needs to be manually started after a cluster node is fenced and rebooted.
The automatic start of all cluster components on all cluster nodes can be enabled with the following command:
pcs cluster enable --all
[root@node1]# pcs cluster enable --allIn some situations it can be beneficial not to have the cluster auto-start after a node has been rebooted. For example, if there is an issue with a filesystem that is required by a cluster resource, the filesystem needs to be repaired first before it can be used again. Having the cluster auto-start, but then fail because the filesystem doesn’t work, can cause even more trouble.
Chapter 5. Testing the cluster configuration
Before the HA cluster setup is put in production, it is recommended to perform the following tests to ensure that the HA cluster setup works as expected.
These tests should also be repeated later on as part of regular HA/DR drills to ensure that the cluster still works as expected and that admins stay familiar with the procedures required to bring the setup back to a healthy state in case an issue occurs during normal operation, or if manual maintenance of the setup is required.
5.1. Manually moving ASCS instance using pcs command
To verify that the pacemaker cluster is able to move the instances to the other HA cluster node on demand.
Test Preconditions
Both cluster nodes are up, with the resource groups for the
ASCSandERSrunning on different HA cluster nodes:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - All failures for the resources and resource groups have been cleared and the failcounts have been reset.
Test Procedure
Run the following command from any node to initiate the move of the
ASCSinstance to the other HA cluster node:pcs resource move S4H_ascs20
[root@node1]# pcs resource move S4H_ascs20Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Monitoring
Run the following command in a separate terminal during the test:
watch -n 1 pcs status
[root@node2]# watch -n 1 pcs statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Expected behavior
-
The
ASCSresource group is moved to the other node. -
The
ERSresource group stops after that and moves to the node where theASCSresource group was running before.
-
The
Test Result
ASCSresource group moves to other node, in this scenario node node2 andERSresource group moves to node node1:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Recovery Procedure:
Remove the location constraints, if any:
pcs resource clear S4H_ascs20
[root@node1]# pcs resource clear S4H_ascs20Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.2. Manually moving of the ASCS instance using sapcontrol (with SAP HA interface enabled)
To verify that the sapcontrol command is able to move the instances to the other HA cluster node when the SAP HA interface is enabled for the instance.
Test Preconditions
- The SAP HA interface is enabled for the SAP instance.
Both cluster nodes are up with the resource groups for the
ASCSandERSrunning.[root@node2: ~]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1
[root@node2: ~]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1Copy to Clipboard Copied! Toggle word wrap Toggle overflow - All failures for the resources and resource groups have been cleared and the failcounts have been reset.
Test Procedure
-
As the
<sid>admuser, run theHAFailoverToNodefunction ofsapcontrolto move theASCSinstance to the other node.
-
As the
Monitoring
Run the following command in a separate terminal during the test:
watch -n 1 pcs status
[root@node2]# watch -n 1 pcs statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Expected behavior
-
ASCSinstances should move to the other HA cluster node, creating a temporary location constraint for the move to complete.
-
Test
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Test result
ASCSandERSboth move to the other node:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2
[root@node2]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2Copy to Clipboard Copied! Toggle word wrap Toggle overflow Constraints are created as shown below:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Recovery Procedure
-
The constraint shown above is cleared automatically when the
date ltmentioned in the Expression is reached. Alternatively, the constraint can be removed with the following command:
pcs resource clear S4H_ascs20
[root@node1]# pcs resource clear S4H_ascs20Copy to Clipboard Copied! Toggle word wrap Toggle overflow
-
The constraint shown above is cleared automatically when the
5.3. Testing failure of the ASCS instance
To verify that the pacemaker cluster takes necessary action when the enqueue server of the ASCS instance or the whole ASCS instance fails.
Test Preconditions
Both cluster nodes are up with the resource groups for the
ASCSandERSrunning:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2
[root@node2]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2Copy to Clipboard Copied! Toggle word wrap Toggle overflow - All failures for the resources and resource groups have been cleared and the failcounts have been reset.
Test Procedure
-
Identify the
PIDof the enqueue server on the node whereASCSis running. -
Send a
SIGKILLsignal to the identified process.
-
Identify the
Monitoring
Run the following command in a separate terminal during the test:
watch -n 1 pcs status
[root@node2]# watch -n 1 pcs statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Expected behavior
- Enqueue server process gets killed.
-
The pacemaker cluster takes the required action as per configuration, in this case moving the
ASCSto the other node.
Test
Switch to the
<sid>adm useron the node whereASCSis running:su - s4hadm
[root@node1]# su - s4hadmCopy to Clipboard Copied! Toggle word wrap Toggle overflow Identify the PID of en.sap(NetWeaver) enq.sap(S/4HANA):
node1:s4hadm 51> pgrep -af "(en|enq).sap" 31464 enq.sapS4H_ASCS20 pf=/usr/sap/S4H/SYS/profile/S4H_ASCS20_s4ascs
node1:s4hadm 51> pgrep -af "(en|enq).sap" 31464 enq.sapS4H_ASCS20 pf=/usr/sap/S4H/SYS/profile/S4H_ASCS20_s4ascsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Kill the identified process:
node1:s4hadm 52> kill -9 31464
node1:s4hadm 52> kill -9 31464Copy to Clipboard Copied! Toggle word wrap Toggle overflow Notice the cluster
Failed Resource Actions:pcs status | grep "Failed Resource Actions" -A1 Failed Resource Actions: * S4H_ascs20 2m-interval monitor on node1 returned 'not running' at Wed Dec 6 15:37:24 2023
[root@node2]# pcs status | grep "Failed Resource Actions" -A1 Failed Resource Actions: * S4H_ascs20 2m-interval monitor on node1 returned 'not running' at Wed Dec 6 15:37:24 2023Copy to Clipboard Copied! Toggle word wrap Toggle overflow ASCSandERSmove to the other node:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ascs20 2m-interval monitor on node1 returned 'not running' at Wed Dec 6 15:37:24 2023
[root@node2]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ascs20 2m-interval monitor on node1 returned 'not running' at Wed Dec 6 15:37:24 2023Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Recovery Procedure
Clear the failed action:
pcs resource cleanup S4H_ascs20 … Waiting for 1 reply from the controller ... got reply (done)
[root@node2]# pcs resource cleanup S4H_ascs20 … Waiting for 1 reply from the controller ... got reply (done)Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.4. Testing failure of the ERS instance
To verify that the pacemaker cluster takes necessary action when the enqueue replication server (ERS) of the ASCS instance fails.
Test Preconditions
Both cluster nodes are up with the resource groups for the
ASCSandERSrunning:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1[root@node1]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1Copy to Clipboard Copied! Toggle word wrap Toggle overflow - All failures for the resources and resource groups have been cleared and the failcounts have been reset.
Test Procedure
-
Identify the PID of the enqueue replication server process on the node where the
ERSinstance is running. - Send a SIGKILL signal to the identified process.
-
Identify the PID of the enqueue replication server process on the node where the
Monitoring
Run the following command in a separate terminal during the test:
watch -n 1 pcs status
[root@node2]# watch -n 1 pcs statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Expected behavior
- Enqueue Replication server process gets killed.
-
Pacemaker cluster takes the required action as per configuration, in this case, restarting the
ERSinstance on the same node.
Test
Switch to the
<sid>admuser:su - s4hadm
[root@node1]# su - s4hadmCopy to Clipboard Copied! Toggle word wrap Toggle overflow Identify the PID of
enqr.sap:node1:s4hadm 56> pgrep -af enqr.sap 532273 enqr.sapS4H_ERS29 pf=/usr/sap/S4H/SYS/profile/S4H_ERS29_s4ers
node1:s4hadm 56> pgrep -af enqr.sap 532273 enqr.sapS4H_ERS29 pf=/usr/sap/S4H/SYS/profile/S4H_ERS29_s4ersCopy to Clipboard Copied! Toggle word wrap Toggle overflow Kill the identified process:
node1:s4hadm 58> kill -9 532273
node1:s4hadm 58> kill -9 532273Copy to Clipboard Copied! Toggle word wrap Toggle overflow Notice the cluster “Failed Resource Actions”:
pcs status | grep "Failed Resource Actions" -A1 Failed Resource Actions: * S4H_ers29 2m-interval monitor on node1 returned 'not running' at Thu Dec 7 13:15:02 2023
[root@node1]# pcs status | grep "Failed Resource Actions" -A1 Failed Resource Actions: * S4H_ers29 2m-interval monitor on node1 returned 'not running' at Thu Dec 7 13:15:02 2023Copy to Clipboard Copied! Toggle word wrap Toggle overflow ERSrestarts on the same node without disturbing theASCSalready running on the other node:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 2m-interval monitor on node1 returned 'not running' at Thu Dec 7 13:15:02 2023[root@node1]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 2m-interval monitor on node1 returned 'not running' at Thu Dec 7 13:15:02 2023Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Recovery Procedure
Clear the failed action:
pcs resource cleanup S4H_ers29 … Waiting for 1 reply from the controller ... got reply (done)
[root@node1]# pcs resource cleanup S4H_ers29 … Waiting for 1 reply from the controller ... got reply (done)Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.5. Failover of ASCS instance due to node crash
To verify that the ASCS instance moves correctly in case of a node crash.
Test Preconditions
Both cluster nodes are up with the resource groups for the
ASCSandERSrunning:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1[root@node1]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node2 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1Copy to Clipboard Copied! Toggle word wrap Toggle overflow - All failures for the resources and resource groups have been cleared and the failcounts have been reset.
Test Procedure
-
Crash the node where
ASCSis running.
-
Crash the node where
Monitoring
Run the following command in a separate terminal on the other node during the test:
watch -n 1 pcs status
[root@node1]# watch -n 1 pcs statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Expected behavior
-
Node where
ASCSis running gets crashed and shuts down or restarts as per configuration. -
Meanwhile
ASCSmoves to the other node. -
ERSstarts on the previously crashed node, after it comes back online.
-
Node where
Test
Run the following command as the root user on the node where
ASCSis running:echo c > /proc/sysrq-trigger
[root@node2]# echo c > /proc/sysrq-triggerCopy to Clipboard Copied! Toggle word wrap Toggle overflow ASCSmoves to the other node:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1[root@node1]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node1Copy to Clipboard Copied! Toggle word wrap Toggle overflow ERSstops and moves to the previously crashed node once it comes back online:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Recovery Procedure
Clean up failed actions, if any:
pcs resource cleanup
[root@node1]# pcs resource cleanupCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.6. Failure of ERS instance due to node crash
To verify that the ERS instance restarts on the same node.
Test Preconditions
Both cluster nodes are up with the resource groups for the
ASCSandERSrunning:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2[root@node1]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2Copy to Clipboard Copied! Toggle word wrap Toggle overflow - All failures for the resources and resource groups have been cleared and the failcounts have been reset.
Test Procedure
-
Crash the node where
ERSis running.
-
Crash the node where
Monitoring
Run the following command in a separate terminal on the other node during the test:
watch -n 1 pcs status
[root@nod1]# watch -n 1 pcs statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Expected behavior
-
Node where
ERSis running gets crashed and shuts down or restarts as per configuration. -
Meanwhile
ASCScontinues to run to the other node.ERSrestarts on the crashed node, after it comes back online.
-
Node where
Test
Run the following command as the root user on the node where
ERSis running:echo c > /proc/sysrq-trigger
[root@node2]# echo c > /proc/sysrq-triggerCopy to Clipboard Copied! Toggle word wrap Toggle overflow ERSrestarts on the crashed node, after it comes back online, without disturbing theASCSinstance throughout the test:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2[root@node1]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node1 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Recovery Procedure
Clean up failed actions if any:
pcs resource cleanup
[root@node2]# pcs resource cleanupCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.7. Failure of ASCS Instance due to node crash (ENSA2)
In case of 3 node ENSA 2 cluster environment, the third node is considered during failover events of any instance.
Test Preconditions
-
A 3 node SAP S/4HANA cluster with the resource groups for the
ASCSandERSrunning. - The 3rd node has access to all the file systems and can provision the required instance specific IP addresses the same way as the first 2 nodes.
In the example setup, the underlying shared
NFSfilesystems are as follows:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - All failures for the resources and resource groups have been cleared and the failcounts have been reset.
-
A 3 node SAP S/4HANA cluster with the resource groups for the
Test Procedure
-
Crash the node where
ASCSis running.
-
Crash the node where
Monitoring
Run the following command in a separate terminal on one of the nodes where the
ASCSgroup is currently not running during the test:watch -n 1 pcs status
[root@node2]# watch -n 1 pcs statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Expected behavior
-
ASCSmoves to the 3rd node. -
ERScontinues to run on the same node where it is already running.
-
Test
Crash the node where the
ASCSgroup is currently running:echo c > /proc/sysrq-trigger
[root@node1]# echo c > /proc/sysrq-triggerCopy to Clipboard Copied! Toggle word wrap Toggle overflow ASCSmoves to the 3rd node without disturbing the already runningERSinstance on 2nd node:pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node3 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2[root@node2]# pcs status | egrep -e "S4H_ascs20|S4H_ers29" * S4H_ascs20 (ocf:heartbeat:SAPInstance): Started node3 * S4H_ers29 (ocf:heartbeat:SAPInstance): Started node2Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Recovery Procedure
Clean up failed actions if any:
pcs resource cleanup
[root@node2]# pcs resource cleanupCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Chapter 6. Maintenance procedures
6.1. Update RHEL and the RHEL HA Add-On
Please refer to Recommendations: Applying package updates in a RHEL High Availability cluster, for more information.
For two-node cluster setups it is not necessary to manually move the resources to the other HA cluster node before placing the HA cluster node in standby mode (putting the HA cluster node in standby mode will take care of moving or stopping the resources running on the HA cluster node based on the HA cluster configuration).
Also, to minimize downtime of the SAP system it is recommended to first update the HA cluster node running the “less critical” resources, like the ERS instance. When the HA cluster node has been updated and it has been verified that the resources that were running on the node before the update was started are running again, the other HA cluster node running the “critical” resources, like the (A)SCS instance, can be updated as well.
Chapter 7. References
7.1. Red Hat
- Configuring and managing high availability clusters
- Support Policies for RHEL High Availability Clusters
- Support Policies for RHEL High Availability Clusters - Fencing/STONITH
- Support Policies for RHEL High Availability Clusters - Management of SAP S/4HANA
- Support Policies for RHEL High Availability Clusters - Management of SAP NetWeaver in a Cluster
- Red Hat HA Solutions for SAP HANA, S/4HANA and NetWeaver based SAP Applications
- How to enable the SAP HA Interface for SAP ABAP application server instances managed by the RHEL HA Add-On?
- How to manage standalone SAP Web Dispatcher instances using the RHEL HA Add-On
- The Systemd-Based SAP Startup Framework
7.2. SAP
- SAP Note 1552925 - Linux: High Availability Cluster Solutions
- SAP Note 1693245 - SAP HA Script Connector Library
- SAP Note 1908655 - Support details for Red Hat Enterprise Linux HA Add-On
- SAP Note 2630416 - Support for Standalone Enqueue Server 2
- SAP Note 2641322 - Installation of ENSA2 and update from ENSA1 to ENSA2 when using the Red Hat HA solutions for SAP
- SAP Note 2772999 - Red Hat Enterprise Linux 8.x: Installation and Configuration
- Standalone Enqueue Server | SAP Help Portal
- Setting up Enqueue Replication Server Fail over | SAP Blogs
- High Availability with the Standalone Enqueue Server
- Evolution of ENSA2 and ERS2… | SAP Blogs
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}