Red Hat SummitChapter 7. Useful commands
Below are 3 sections of useful commands. In most cases, it should help to verify successful operation or configuration. Examples are listed together with the response. In some cases, the output has been adjusted for formatting reasons.
-
All commands listed in this document when executed by the
<sid>admuser start with>. -
All commands run by the
root userstart with a#. -
To execute the commands, omit the prefix
>or#.
7.1. SAP HANA commands
The SAP HANA commands are executed by the <sid>adm user. Example:
7.1.1. SAP HANA installation using hdbclm
The installation of the third site is similar to the installation of the second site. The installation can be done with hdblcm as user root. To ensure that nothing is installed before, run hdbuninst to check if SAP HANA is not already installed on this node.
Example output of HANA uninstallation:
cd /software/DATA_UNITS/HDB_SERVER_LINUX_X86_64 root@DC3/software/DATA_UNITS/HDB_SERVER_LINUX_X86_64# ./hdbuninst Option 0 will remove an already existing HANA Installation No SAP HANA Installation found is the expected answer
[root@remotehost3]# cd /software/DATA_UNITS/HDB_SERVER_LINUX_X86_64
root@DC3/software/DATA_UNITS/HDB_SERVER_LINUX_X86_64# ./hdbuninst
Option 0 will remove an already existing HANA Installation
No SAP HANA Installation found is the expected answerExample output of HANA installation on DC3:
Before the installation starts, a summary is listed:
Enter y to start the installation.
7.1.2. Using hdbsql to check Inifile contents
7.1.3. Check database
Check if the database is running and discover the current primary node.
List database instances
If the output is green the instance is running.
List database processes
Usually, all database processes have the status GREEN.
List SAP HANA processes
Display SAP HANA landscape configuration
Returncodes:
- 0: Fatal
- 1: Error
- 2: Warning
- 3: Info
- 4: OK
Discover primary database
clusternode1:rh2adm> hdbnsutil -sr_state | egrep -e "primary masters|^mode"
clusternode1:rh2adm> hdbnsutil -sr_state | egrep -e "primary masters|^mode"Example of check on a secondary:
clusternode1:rh2adm> hdbnsutil -sr_state | egrep -e "primary masters|^mode" mode: syncmem primary masters: clusternode1
clusternode1:rh2adm> hdbnsutil -sr_state | egrep -e "primary masters|^mode"
mode: syncmem
primary masters: clusternode1Example of check on the current primary:
Display the database version
Example using SQL query:
hdbsql RH2=> select * from m_database SYSTEM_ID,DATABASE_NAME,HOST,START_TIME,VERSION,USAGE "RH2","RH2","node1","2023-06-22 15:33:05.235000000","2.00.059.02.1647435895","CUSTOM" 1 row selected (overall time 29.107 msec; server time 927 usec)
hdbsql RH2=> select * from m_database
SYSTEM_ID,DATABASE_NAME,HOST,START_TIME,VERSION,USAGE
"RH2","RH2","node1","2023-06-22 15:33:05.235000000","2.00.059.02.1647435895","CUSTOM"
1 row selected (overall time 29.107 msec; server time 927 usec)
Example using systemOverview.py:
7.1.4. Start and stop SAP HANA
Option 1: HDB command
clusternode1:rh2adm> HDB help
Usage: /usr/sap/RH2/HDB02/HDB { start|stop|reconf|restart|version|info|proc|admin|kill|kill-<sig>|term }
kill or kill-9 should never be used in a productive environment!
clusternode1:rh2adm> HDB help
Usage: /usr/sap/RH2/HDB02/HDB { start|stop|reconf|restart|version|info|proc|admin|kill|kill-<sig>|term }
kill or kill-9 should never be used in a productive environment!Start the Database
clusternode1:rh2adm> HDB start
clusternode1:rh2adm> HDB startCopy to Clipboard Copied! Toggle word wrap Toggle overflow Stop the database
clusternode1:rh2adm> HDB stop
clusternode1:rh2adm> HDB stopCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Option 2 (recommended): Use sapcontrol
clusternode1:rh2adm> sapcontrol -nr ${TINSTANCE} -function StartSystem HDB
03.07.2023 14:08:30
StartSystem
OK
clusternode1:rh2adm> sapcontrol -nr ${TINSTANCE} -function StartSystem HDB
03.07.2023 14:08:30
StartSystem
OKclusternode1:rh2adm> sapcontrol -nr ${TINSTANCE} -function StopSystem HDB
03.07.2023 14:09:33
StopSystem
OK
clusternode1:rh2adm> sapcontrol -nr ${TINSTANCE} -function StopSystem HDB
03.07.2023 14:09:33
StopSystem
OKUse the GetProcessList to monitor the starting and stopping of HANA services:
clusternode1:rh2adm> sapcontrol -nr ${TINSTANCE} -function GetProcessList
clusternode1:rh2adm> sapcontrol -nr ${TINSTANCE} -function GetProcessList7.1.5. Check SAP HANA System Replication status
There are many ways to check the SAP HANA System Replication status:
- `clusternode1:rh2adm> python systemReplicationStatus.py ` on the primary node
-
clusternode1:rh2adm> echo $? #(Return code of systemReplicationStatus) -
clusternode1:rh2adm> hdbnsutil -sr_state -
clusternode1:rh2adm> hdbnsutil -sr_stateConfiguration
Example of systemReplicationStatus.py output running as a monitor:
The expected results for the return codes are:
- 10: NoHSR
- 11: Error
- 12: Unknown
- 13: Initializing
- 14: Syncing
- 15: Active
In most cases the System Replication check will return with return code 15. Another display option is to use -t (printLandscapeTree).
Example for the output on the current primary:
clusternode1:rh2adm> python systemReplicationStatus.py -t
HANA System Replication landscape:
DC1 ( primary )
| --- DC3 ( syncmem )
| --- DC2 ( syncmem )
clusternode1:rh2adm> python systemReplicationStatus.py -t
HANA System Replication landscape:
DC1 ( primary )
| --- DC3 ( syncmem )
| --- DC2 ( syncmem )
Example of hdbnsutil -sr_state:
Example of sr_stateConfiguation on the primary:
Example of sr_stateConfiguration on the secondary:
You can also check in the secondary database which node is the current primary. During the failover it happens to have two primary databases and this information is needed to decide which potential primary database is wrong and needs to be re-registered as secondary.
For additional information, refer to Example: Checking the Status on the Primary and Secondary Systems.
7.1.6. Register secondary node
Preconditions to register a secondary database for a SAP HANA System Replication environment:
- Create SAP HANA backup
- Enable SAP HANA System Replication on the primary node
- Copy database keys
- Register Secondary Node
Registration example:
With the registration the global.ini file will be automatically updated
… from:
… to:
7.1.7. sapcontrol GetProcessList
Check the processes of an active SAP HANA database
7.1.8. sapcontrol GetInstanceList
This will list the status of instances of a SAP HANA database. It will also show the ports. There are three different status names:
- GREEN (running)
- GRAY (stopped)
- YELLOW ( status is currently changing)
Example of an active instance:
Example of a stopped instance:
7.1.9. hdbcons examples
You can also use the HDB Console to display information about the database:
-
hdbcons -e hdbindexserver 'replication info' -
hdbcons -e hdbindexserver helpfor more options
Example of ‘replication info’:
Example of help:
7.1.10. Create SAP HANA backup
If you want to use SAP HANA System Replication, a backup must first be created on the primary system.
Example of how to perform this is as user <sid>adm:
clusternode1:rh2adm> hdbsql -i ${TINSTANCE} -u system -d SYSTEMDB "BACKUP DATA USING FILE ('/hana/backup/')"
clusternode1:rh2adm> hdbsql -i ${TINSTANCE} -u system -d ${SAPSYSTEMNAME} "BACKUP DATA USING FILE ('/hana/backup/')"
clusternode1:rh2adm> hdbsql -i ${TINSTANCE} -u system -d SYSTEMDB "BACKUP DATA USING FILE ('/hana/backup/')"
clusternode1:rh2adm> hdbsql -i ${TINSTANCE} -u system -d ${SAPSYSTEMNAME} "BACKUP DATA USING FILE ('/hana/backup/')"7.1.11. Enable SAP HANA System Replication on the primary database
SAP HANA System Replication has to be enabled on the primary node. This requires a backup to be done first.
clusternode1:rh2adm> hdbnsutil -sr_enable --name=DC1 nameserver is active, proceeding ... successfully enabled system as system replication source site done.
clusternode1:rh2adm> hdbnsutil -sr_enable --name=DC1
nameserver is active, proceeding ...
successfully enabled system as system replication source site
done.7.1.12. Copy database keys to the secondary nodes
The database keys need to be copied from the primary to the secondary database before it can be registered as a secondary.
For example:
clusternode1:rh2adm> scp -rp /usr/sap/${SAPSYSTEMNAME}/SYS/global/security/rsecssfs/data/SSFS_${SAPSYSTEMNAME}.DAT remotehost3:/usr/sap/${SAPSYSTEMNAME}/SYS/global/security/rsecssfs/data/SSFS_${SAPSYSTEMNAME}.DAT
clusternode1:rh2adm> scp -rp /usr/sap/${SAPSYSTEMNAME}/SYS/global/security/rsecssfs/key/SSFS_${SAPSYSTEMNAME}.KEY remotehost3:/usr/sap/${SAPSYSTEMNAME}/SYS/global/security/rsecssfs/key/SSFS_${SAPSYSTEMNAME}.KEY
clusternode1:rh2adm> scp -rp /usr/sap/${SAPSYSTEMNAME}/SYS/global/security/rsecssfs/data/SSFS_${SAPSYSTEMNAME}.DAT remotehost3:/usr/sap/${SAPSYSTEMNAME}/SYS/global/security/rsecssfs/data/SSFS_${SAPSYSTEMNAME}.DAT
clusternode1:rh2adm> scp -rp /usr/sap/${SAPSYSTEMNAME}/SYS/global/security/rsecssfs/key/SSFS_${SAPSYSTEMNAME}.KEY remotehost3:/usr/sap/${SAPSYSTEMNAME}/SYS/global/security/rsecssfs/key/SSFS_${SAPSYSTEMNAME}.KEY7.1.13. Register a secondary node for SAP HANA System Replication
Please ensure that the database keys have been copied to the secondary nodes first. Then run the registration command:
clusternode1:rh2adm> hdbnsutil -sr_register --remoteHost=remotehost3 --remoteInstance=${TINSTANCE} --replicationMode=syncmem --name=DC1 --remoteName=DC3 --operationMode=logreplay --online
clusternode1:rh2adm> hdbnsutil -sr_register --remoteHost=remotehost3 --remoteInstance=${TINSTANCE} --replicationMode=syncmem --name=DC1 --remoteName=DC3 --operationMode=logreplay --onlineParameter description:
-
remoteHost: hostname of the active node running the source (primary) database -
remoteInstance: the instance number of the database replicationMode: one of the following options-
sync: hard disk synchronization -
async: asynchronous replication -
syncmem: memory synchronization
-
-
name: this is an alias for this replication site -
remoteName: alias name of the source database operationMode: one of the following options-
delta_datashipping: data is periodically transmitted. Takeovers take a little bit longer. -
logreplay: logs are redone immediately on the remote site. Takeover is faster. -
logreplay_readaccess: additional logreplay read-only access to the second site is possible.
-
7.1.14. Check the log_mode of the SAP HANA database
There are two options for setting the log_mode:
-
log_mode=overwrite -
log_mode=normal: This is the default value and is also required when the database instance is running as primary. Using SAP HANA Multitarget System Replication, you have to uselog_mode=normal. The best way to check thelog_modeis by usinghdbsql:
Example including a wrong overwrite entry:
In this case, we have two global.ini files:
DEFAULT-
/usr/sap/${SAPSYSTEMNAME}/SYS/global/hdb/custom/config/global.ini
-
HOST-
/hana/shared/${SAPSYSTEMNAME}/HDB${TINSTANCE}/${HOSTNAME}/global.iniTheHOSTvalues overwrite theDEFAULTvalues. You can also check both files before the database is started and then usehdbsqlagain to verify the right settings. You can change thelog_modeby editing theglobal.inifile.
-
Example:
clusternode1:rh2adm> vim /hana/shared/${SAPSYSTEMNAME}/HDB${TINSTANCE}/${HOSTNAME}/global.ini
# global.ini last modified 2023-04-06 16:15:03.521715 by hdbnameserver
[persistence]
log_mode = overwrite
clusternode1:rh2adm> vim /hana/shared/${SAPSYSTEMNAME}/HDB${TINSTANCE}/${HOSTNAME}/global.ini
# global.ini last modified 2023-04-06 16:15:03.521715 by hdbnameserver
[persistence]
log_mode = overwriteglobal.ini last modified 2023-04-06 16:15:03.521715 by hdbnameserver [persistence] log_mode = normal
# global.ini last modified 2023-04-06 16:15:03.521715 by hdbnameserver
[persistence]
log_mode = normal
After having checked or updated the global.ini file(s), verify the log_mode values:
The section also shows that this parameter needs to be set in the [persistence] section. When you change the log mode from overwrite to normal, it is recommended that you create a full data backup to ensure that the database can be recovered.
7.1.15. Discover primary database
There are several ways to identify the primary node, for instance:
-
pcs status | grep Promoted -
hdbnsutil -sr_stateConfiguration -
systemReplicationStatus.py
Option 1 - The following example of the systemReplicationStatus.py script and filter will return the primary database location on all nodes:
clusternode1:rh2adm>
/usr/sap/${SAPSYSTEMNAME}/HDB${TINSTANCE}/exe/Python/bin/python
/usr/sap/${SAPSYSTEMNAME}/HDB${TINSTANCE}/exe/python_support/systemReplicationStatus.py --sapcontrol=1 | egrep -e
"3${TINSTANCE}01/HOST|PRIMARY_MASTERS"| head -1 | awk -F"=" '{ print $2 }'
clusternode1:rh2adm>
/usr/sap/${SAPSYSTEMNAME}/HDB${TINSTANCE}/exe/Python/bin/python
/usr/sap/${SAPSYSTEMNAME}/HDB${TINSTANCE}/exe/python_support/systemReplicationStatus.py --sapcontrol=1 | egrep -e
"3${TINSTANCE}01/HOST|PRIMARY_MASTERS"| head -1 | awk -F"=" '{ print $2 }'Output:
clusternode2
clusternode2
Option 2 - The following example displays the systemReplicationStatus in a similar way for all nodes:
rh2adm>hdbnsutil -sr_state --sapcontrol=1 | grep site.*Mode
rh2adm>hdbnsutil -sr_state --sapcontrol=1 | grep site.*ModeOutput:
7.1.16. Takeover primary
Please refer to Check the Replication status section for check on the primary and the secondary nodes. Also:
-
Put cluster into
maintenance-mode - Initiate the takeover on the secondary node
Example for enabling maintenance-mode for the cluster:
pcs property set maintenance-mode=true
[root@clusternode1]# pcs property set maintenance-mode=true
On the secondary that is to become the new primary, run as <sidadm> user:
clusternode1:rh2adm> hdbnsutil -sr_takeover
clusternode1:rh2adm> hdbnsutil -sr_takeoverThis secondary becomes the primary, other active secondary databases get re-registered to the new primary and the old primary needs to be manually re-registered as secondary.
7.1.17. Re-register former primary as secondary
Please ensure that the cluster is stopped or put in maintenance-mode. Example:
clusternode2:rh2adm> hdbnsutil -sr_register --remoteHost=remotehost3 --remoteInstance=${TINSTANCE} --replicationMode=syncmem --name=DC2 --online --remoteName=DC3 --operationMode=logreplay --force_full_replica --online
clusternode2:rh2adm> hdbnsutil -sr_register --remoteHost=remotehost3 --remoteInstance=${TINSTANCE} --replicationMode=syncmem --name=DC2 --online --remoteName=DC3 --operationMode=logreplay --force_full_replica --onlineIn our examples, we are using full replication. Your SAP HANA system administrator should know when full replication is required.
7.1.18. Recover from failover
Please refer to Check the SAP HANA System Replication status and Discover the primary node. It is important that the information is consistent. If a node is not part of the systemReplicationStatus.py output and has a different system replication state, please check with your database administrator if this node needs to be re-registered.
One way of solving this is to re-register this site as a new secondary.
Sometimes a secondary instance will still not come up. Then unregister this site before you re-register it again. Example of unregistering the secondary DC1:
clusternode1:rh2adm> hdbnsutil -sr_unregister --name=DC1
clusternode1:rh2adm> hdbnsutil -sr_unregister --name=DC1Example of re-registering DC1:
clusternode1:rh2adm> hdbnsutil -sr_register --name=DC1 --remoteHost=node2 --remoteInstance=02 --replicationMode=sync --operationMode=logreplay --online
clusternode1:rh2adm> hdbnsutil -sr_register --name=DC1 --remoteHost=node2 --remoteInstance=02 --replicationMode=sync --operationMode=logreplay --onlineThe database needs to be started and checked if it is running. Finally check the replication status.
7.2. Pacemaker commands
7.2.1. Start and stop the cluster
To start the cluster on all nodes execute the following command:
pcs cluster start -all
# pcs cluster start -allAfter a reboot, the cluster will be started automatically only if the service is enabled. The command will help to know if the cluster has started and if the daemons are enabled to be autostarted.
pcs cluster status
# pcs cluster statusThe cluster auto-start can be enabled with:
pcs cluster enable --all
# pcs cluster enable --allOther options are:
- Stop the cluster.
- Put a node into standby.
-
Put the cluster into
maintenance-mode.
For more details, please check the pcs cluster help:
pcs cluster stop --all pcs cluster help
# pcs cluster stop --all
# pcs cluster help7.2.2. Put the cluster into maintenance-mode
If you want to make changes and you want to avoid interference bythe pacemaker cluster, you can "freeze" the cluster by putting it into maintenance-mode:
pcs property set maintenance-mode=true
# pcs property set maintenance-mode=true
An easy way to verify maintenance-mode is to check if the resources are unmanaged:
Refresh cluster resources to detect the resource state while the cluster is in maintenance-mode and does not update resource status changes:
pcs resource refresh
# pcs resource refresh
This will indicate if anything is not yet correct and will cause remediation action by the cluster, as soon as it is taken out of maintenance-mode.
Remove the maintenance-mode by running:
pcs property set maintenance-mode=false
# pcs property set maintenance-mode=falseNow the cluster will continue to work. If something is configured wrong, it will react now.
7.2.3. Check cluster status
Following are several ways to check the cluster status:
Check if the cluster is running:
pcs cluster status
# pcs cluster statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the cluster and all resources:
pcs status
# pcs statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the cluster, all resources and all node attributes:
pcs status --full
# pcs status --fullCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the resources only:
pcs resource status --full
# pcs resource status --fullCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check
Stonithhistory:pcs stonith history
# pcs stonith historyCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check location constraints:
pcs constraint location
# pcs constraint locationCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Fencing must be configured and tested. In order to obtain a solution that is as automated as possible, the cluster must be constantly activated, which will then enable the cluster to automatically start after a reboot. In a production environment, disabling the restart allows manual intervention, for instance after a crash. Please also check the daemon status.
Example:
7.2.4. Check resource states
Use pcs resource to check the status of all resources. This prints the list and the current status of the resources.
Example:
7.2.5. Check resource config
The following displays the current resource configuration:
This lists all the parameters which are used to configure the installed and configured resource agent.
7.2.6. SAPHana resource option AUTOMATED_REGISTER=true
If this option is used in the SAPHana resource, pacemaker will automatically re-register the secondary database.
It is recommended to use this option for the first tests. When using AUTOMATED_REGISTER=false the administrator needs to re-register the secondary node manually.
7.2.7. Resource handling
There are several options for managing resources. For more information, please check out the help available:
pcs resource help
# pcs resource helpList the used resource agents:
pcs resource config | grep "type=" | awk -F"type=" '{ print $2 }' | sed -e "s/)//g"
# pcs resource config | grep "type=" | awk -F"type=" '{ print $2 }' | sed -e "s/)//g"Example output:
IPaddr2 SAPHanaTopology SAPHana
IPaddr2
SAPHanaTopology
SAPHanaDisplay specific resource agent description and configuration parameters:
pcs resource describe <resource agent>
# pcs resource describe <resource agent>Example (without output):
pcs resource describe IPaddr2
# pcs resource describe IPaddr2
Example of resource agent IPaddr2 (with output):
If the cluster is stopped, all the resources will be stopped as well; if the cluster is put into maintenance-mode, all resources remain in their current status but will not be monitored or managed.
7.2.8. Cluster property handling for maintenance-mode
List all defined properties:
To reconfigure the database, the cluster must be instructed to ignore any changes until the configuration is complete. You can put the cluster into maintenance-mode using:
pcs property set maintenance-mode=true
# pcs property set maintenance-mode=true
Check the maintenance-mode:
Verify that all resources are "unmanaged":
The resources will switch back to managed if you unset the maintenance-mode:
pcs property set maintenance-mode=false
# pcs property set maintenance-mode=false7.2.9. Failover the SAPHana resource using Move
A simple example of how to failover the SAP HANA database is to use the pcs resource move command. You need to use the clone resource name and move the resource as shown below:
pcs resource move <SAPHana-clone-resource>
# pcs resource move <SAPHana-clone-resource>
In this example, the clone resource is SAPHana_RH2_02-clone:
Move the resource:
Check if there are remaining constraints:
pcs constraint location
# pcs constraint locationYou can remove those location constraints created during the failover by clearing the resource. Example:
pcs resource clear SAPHana_RH2_02-clone
[root@clusternode1]# pcs resource clear SAPHana_RH2_02-cloneCheck if there are any remaining warnings or entries in the "Migration Summary":
pcs status --full
# pcs status --full
Check the stonith history:
pcs stonith history
# pcs stonith historyIf desired, clear the stonith history:
pcs stonith history cleanup
# pcs stonith history cleanupIf you are using a pacemaker version earlier than 2.1.5, please refer to Is there a way to manage constraints when running pcs resource move? and check the remaining constraints.
7.2.10. Monitor failover and sync state
All pacemaker activities are logged in the /var/log/messages file on the cluster nodes. Since there are many other messages, it is sometimes difficult to read the messages related to the SAP resource agent. You can configure a command alias that filters out only the messages related to SAP resource agent.
Example alias tmsl:
alias tmsl='tail -1000f /var/log/messages | egrep -s "Setting master-rsc_SAPHana_${SAPSYSTEMNAME}_HDB${TINSTANCE}|sr_register|WAITING4LPA|PROMOTED|DEMOTED|UNDEFINED|master_walk|SWAIT|WaitforStopped|FAILED|LPT"'
# alias tmsl='tail -1000f /var/log/messages | egrep -s "Setting master-rsc_SAPHana_${SAPSYSTEMNAME}_HDB${TINSTANCE}|sr_register|WAITING4LPA|PROMOTED|DEMOTED|UNDEFINED|master_walk|SWAIT|WaitforStopped|FAILED|LPT"'
Example output of tsml:
The filter makes it easier to understand what status changes are happening. If details are missing, you can open the whole message file to read all the information.
After a failover, you can clear the resource. Please also check that there are no remaining location constraints.
7.2.11. Check cluster consistency
During the installation the resources are sometimes started before the configuration is finally completed. This can lead to entries in the Cluster Information Base (CIB), which can result in incorrect behavior. This can easily be checked and also manually corrected after the configuration has been completed.
If you start the SAPHana resources the missing entries will be recreated. Wrong entries cannot be addressed by pcs commands and need to be removed manually.
Check CIB entries:
cibadmin --query
# cibadmin --queryDC3 and SFAIL are entries that should not be present in the Cluster Information Base, when the cluster members are DC1 and DC2, and when the sync state between the nodes is reported as SOK.
Example to check for corresponding entries:
cibadmin --query |grep '"DC3"' cibadmin --query |grep '"SFAIL"'
# cibadmin --query |grep '"DC3"'
# cibadmin --query |grep '"SFAIL"'The command can be executed on any node in the cluster as user root. Usually the output of the command is empty. If there is still an error in the configuration the output could look like this:
<nvpair id="SAPHanaSR-hana_rh1_glob_sec" name="hana_rh1_glob_sec" value="DC3"/>
<nvpair id="SAPHanaSR-hana_rh1_glob_sec" name="hana_rh1_glob_sec" value="DC3"/>These entries can be removed with the following command:
cibadmin --delete --xml-text '<...>'
# cibadmin --delete --xml-text '<...>'To remove the entries in the example above you have to enter the following. Please note that the output contains double quotes, so the text must be embedded in single quotes:
cibadmin --delete --xml-text ' <nvpair id="SAPHanaSR-hana_rh1_glob_sec" name="hana_rh1_glob_sec" value="DC3"/>'
# cibadmin --delete --xml-text ' <nvpair id="SAPHanaSR-hana_rh1_glob_sec" name="hana_rh1_glob_sec" value="DC3"/>'Verify the absence of the removed CIB entries. The returned output should be empty.
cibadmin --query |grep 'DC3"'
# cibadmin --query |grep 'DC3"'7.2.12. Cluster cleanup
During the failover tests there might be left behind constraints and other remains from previous tests. The cluster needs to be cleared from these before starting the next test.
Check the cluster status for failure events:
pcs status --full
# pcs status --fullIf you see cluster warnings or entries in the "Migration Summary", you should clear and cleanup the resources:
pcs resource clear SAPHana_RH2_02-clone pcs resource cleanup SAPHana_RH2_02-clone
# pcs resource clear SAPHana_RH2_02-clone
# pcs resource cleanup SAPHana_RH2_02-cloneOutput:
Cleaned up SAPHana_RH2_02:0 on clusternode1 Cleaned up SAPHana_RH2_02:1 on clusternode2
Cleaned up SAPHana_RH2_02:0 on clusternode1
Cleaned up SAPHana_RH2_02:1 on clusternode2Check if there are unwanted location constraints, for example from a previous failover:
pcs constraint location
# pcs constraint locationCheck the existing constraints in more detail:
pcs constraint --full
# pcs constraint --fullExample of a location constraint after a resource move:
Node: hana08 (score:-INFINITY) (role:Started) (id:cli-ban-SAPHana_RH2_02-clone-on-hana08)
Node: hana08 (score:-INFINITY) (role:Started) (id:cli-ban-SAPHana_RH2_02-clone-on-hana08)Clear this location constraint:
pcs resource clear SAPHana_RH2_02-clone
# pcs resource clear SAPHana_RH2_02-cloneVerify the constraint is gone from the constraints list. If it persists, explicitly delete it using its constraint id:
pcs constraint delete cli-ban-SAPHana_RH2_02-clone-on-hana08
# pcs constraint delete cli-ban-SAPHana_RH2_02-clone-on-hana08
If you run several tests with fencing you might also clear the stonith history:
pcs stonith history cleanup
# pcs stonith history cleanupAll pcs commands are executed as user root. Please also check Discover leftovers.
7.2.13. Other cluster commands
Various cluster command examples
7.3. RHEL and general commands
7.3.1. Discover current status
You have to follow several steps to know what the current status of the environment is. Please refer to Monitor the environment. Also, we recommend to do the following:
-
Check
/var/log/messages, use Aliases for monitoring for easier log reviews. - Sometimes a cluster must be cleaned up from previous activity to continue proper operation. Discover leftovers and clear them if necessary.
7.3.2. yum info
7.3.3. RPM display version
rpm -q resource-agents-sap-hana resource-agents-sap-hana-0.162.1-2.el9_2.noarch
# rpm -q resource-agents-sap-hana
resource-agents-sap-hana-0.162.1-2.el9_2.noarch7.3.4. Aliases for monitoring
You can add this to your shell profile. In the example the root aliases depend on the <sid>adm aliases, which must therefore also already be defined.
root ( add to
~/.bashrc):Copy to Clipboard Copied! Toggle word wrap Toggle overflow <sid>adm( add to~/.customer.sh):Copy to Clipboard Copied! Toggle word wrap Toggle overflow
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}