5.5. Test 3:Failover of the primary node to the third site


Subject of the test

Failover the primary to the third site.. Third site becomes primary.

Secondary will be re-registered to third site.

Test preconditions

  • SAP HANA on DC1, DC2, DC3 is running.
  • Cluster is up and running without errors or warnings.
  • System Replication is in place and in sync (check % python systemReplicationStatus.py).

Test steps

Put the cluster into maintenance-mode to be able to recover.

Takeover the HANA database form the third node using: % hdbnsuttil -sr_takeover

Starting the test

Execute the SAP HANA command on remotehost3:rh2adm>: hdbnsutil -sr_takeover

Monitoring the test

On the third site run as sidadm% watch hdbnsutil -sr_state

Expected result

  • Third node will become primary.
  • Secondary node will change the primary master to remotehost3. Former primary node needs to be re-registered to the new primary.

Ways to return to an initial state

Run Test 4: Failback of the primary node to the first site.

Detailed description

  • Check if the databases are running using Check database and check the replication status:

    clusternode2:rh2adm> hdbnsutil -sr_state | egrep -e "^mode:|primary masters"

    The output is, for example:

    mode: syncmem
    primary masters: clusternode1

    In this case, the primary database is clusternode1. If you run this command on clusternode1, you will get:

    mode: primary

    On this primary node, you can also display the system replication status. It should look like this:

    clusternode1:rh2adm> cdpy
    clusternode1:rh2adm> python systemReplicationStatus.py
    |Database |Host   |Port  |Service Name |Volume ID |Site ID |Site Name |Secondary |Secondary |Secondary |Secondary |Secondary     |Replication |Replication |Replication    |Secondary    |
    |         |       |      |             |          |        |          |Host      |Port      |Site ID   |Site Name |Active Status |Mode        |Status      |Status Details |Fully Synced |
    |-------- |------ |----- |------------ |--------- |------- |--------- |--------- |--------- |--------- |--------- |------------- |----------- |----------- |-------------- |------------ |
    |SYSTEMDB |clusternode1 |30201 |nameserver   |        1 |      1 |DC1       |remotehost3    |    30201 |        3 |DC3       |YES           |SYNCMEM     |ACTIVE      |               |        True |
    |RH2      |clusternode1 |30207 |xsengine     |        2 |      1 |DC1       |remotehost3    |    30207 |        3 |DC3       |YES           |SYNCMEM     |ACTIVE      |               |        True |
    |RH2      |clusternode1 |30203 |indexserver  |        3 |      1 |DC1       |remotehost3    |    30203 |        3 |DC3       |YES           |SYNCMEM     |ACTIVE      |               |        True |
    |SYSTEMDB |clusternode1 |30201 |nameserver   |        1 |      1 |DC1       |clusternode2    |    30201 |        2 |DC2       |YES           |SYNCMEM     |ACTIVE      |               |        True |
    |RH2      |clusternode1 |30207 |xsengine     |        2 |      1 |DC1       |clusternode2    |    30207 |        2 |DC2       |YES           |SYNCMEM     |ACTIVE      |               |        True |
    |RH2      |clusternode1 |30203 |indexserver  |        3 |      1 |DC1       |clusternode2    |    30203 |        2 |DC2       |YES           |SYNCMEM     |ACTIVE      |               |        True |
    
    status system replication site "3": ACTIVE
    status system replication site "2": ACTIVE
    overall system replication status: ACTIVE
    
    Local System Replication State
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    mode: PRIMARY
    site id: 1
    site name: DC1
  • Now we have a proper environment, and we can start monitoring the system replication status on all 3 nodes in separate windows. The 3 monitors should be started before the test is started. The output will change when the test is executed. So keep them running as long as the test is not completed.

    On the old primary node, clusternode1 ran in a separate window during the test:

    clusternode1:rh2adm> watch -n 5 'python /usr/sap/${SAPSYSTEMNAME}/HDB${TINSTANCE}/exe/python_support/systemReplicationStatus.py ; echo Status $?'

    The output on clusternode1 will be:

    Every 5.0s: python /usr/sap/${SAPSYSTEMNAME}/HDB${TINSTANCE}/exe/python_support/systemReplicati...  clusternode1: Tue XXX XX HH:MM:SS 2023
    
    |Database |Host   |Port  |Service Name |Volume ID |Site ID |Site Name |Secondary |Secondary |Secondary |Secondary |Secondary     |
    Replication |Replication |Replication    |Secondary    |
    |         |	  |	 |             |          |        |          |Host	 |Port      |Site ID   |Site Name |Active Status |
    Mode        |Status	 |Status Details |Fully Synced |
    |-------- |------ |----- |------------ |--------- |------- |--------- |--------- |--------- |--------- |--------- |------------- |
    ----------- |----------- |-------------- |------------ |
    |SYSTEMDB |clusternode1 |30201 |nameserver   |        1 |	 1 |DC1       |remotehost3    |    30201 |        3 |DC3	  |YES		 |
    ASYNC       |ACTIVE	 |               |        True |
    |RH2	  |clusternode1 |30207 |xsengine     |        2 |	 1 |DC1       |remotehost3    |    30207 |        3 |DC3	  |YES		 |
    ASYNC       |ACTIVE	 |               |        True |
    |RH2	  |clusternode1 |30203 |indexserver  |        3 |	 1 |DC1       |remotehost3    |    30203 |        3 |DC3	  |YES		 |
    ASYNC       |ACTIVE	 |               |        True |
    |SYSTEMDB |clusternode1 |30201 |nameserver   |        1 |	 1 |DC1       |clusternode2    |    30201 |        2 |DC2	  |YES		 |
    SYNCMEM     |ACTIVE	 |               |        True |
    |RH2	  |clusternode1 |30207 |xsengine     |        2 |	 1 |DC1       |clusternode2    |    30207 |        2 |DC2	  |YES		 |
    SYNCMEM     |ACTIVE	 |               |        True |
    |RH2	  |clusternode1 |30203 |indexserver  |        3 |	 1 |DC1       |clusternode2    |    30203 |        2 |DC2	  |YES		 |
    SYNCMEM     |ACTIVE	 |               |        True |
    
    status system replication site "3": ACTIVE
    status system replication site "2": ACTIVE
    overall system replication status: ACTIVE
    
    Local System Replication State
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    mode: PRIMARY
    site id: 1
    site name: DC1
    Status 15

    On remotehost3, run the same command:

    remotehost3:rh2adm> watch -n 5 'python /usr/sap/${SAPSYSTEMNAME}/HDB${TINSTANCE}/exe/python_support/systemReplicationStatus.py ; echo Status $?'

    The response will be:

    this system is either not running or is not primary system replication site

    This will change after the test initiates the failover. The output looks similar to the example of the primary node before the test was started.

    On the second node, start:

    clusternode2:rh2adm> watch -n 10 'hdbnsutil -sr_state | grep masters'

    This will show the current master clusternode1 and will switch immediately after the failover is initiated.

  • To ensure that everything is configured correctly, please also check the global.ini.
  • Check global.ini on DC1, DC2, and DC3:

    On all three nodes, the global.ini should contain:

    [persistent]
    log_mode=normal
    [system_replication]
    register_secondaries_on_takeover=true

    You can edit the global.ini with:

    clusternode1:rh2adm>vim /usr/sap/${SAPSYSTEMNAME}/SYS/global/hdb/custom/config/global.ini
  • [Optional] Put the cluster into maintenance-mode:

    [root@clusternode1]# pcs property set maintenance-mode=true

    During the tests, you will find out that the failover will work with and without setting the maintenance-mode. So you can run the first test without it. While recovering, it should be done; I just want to show you that it works with and without. This is an option if the primary is not accessible.

  • Start the test: Failover to DC3. On remotehost3, please run:

    remotehost3:rh2adm> hdbnsutil -sr_takeover
    done.

    The test has started, and now please check the output of the previously started monitors. On the clusternode1, the system replication status will lose its relationship to remotehost3 and clusternode2 (DC2):

    Every 5.0s: python /usr/sap/RH2/HDB02/exe/python_support/systemReplicationStatus.py ; echo Status $?                               clusternode1: Mon Sep  4 11:52:16 2023
    
    |Database |Host   |Port  |Service Name |Volume ID |Site ID |Site Name |Secondary |Secondary |Secondary |Secondary |Secondary     |Replication |Replication |Replic
    ation                  |Secondary    |
    |         |       |      |             |          |        |          |Host      |Port      |Site ID   |Site Name |Active Status |Mode        |Status      |Status
     Details               |Fully Synced |
    |-------- |------ |----- |------------ |--------- |------- |--------- |--------- |--------- |--------- |--------- |------------- |----------- |----------- |------
    ---------------------- |------------ |
    |SYSTEMDB |clusternode1 |30201 |nameserver   |        1 |      1 |DC1       |clusternode2    |    30201 |        2 |DC2       |YES           |SYNCMEM     |ERROR       |Commun
    ication channel closed |       False |
    |RH2      |clusternode1 |30207 |xsengine     |        2 |      1 |DC1       |clusternode2    |    30207 |        2 |DC2       |YES           |SYNCMEM     |ERROR       |Commun
    ication channel closed |       False |
    |RH2      |clusternode1 |30203 |indexserver  |        3 |      1 |DC1       |clusternode2    |    30203 |        2 |DC2       |YES           |SYNCMEM     |ERROR       |Commun
    ication channel closed |       False |
    
    status system replication site "2": ERROR
    overall system replication status: ERROR
    
    Local System Replication State
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    mode: PRIMARY
    site id: 1
    site name: DC1
    Status 11

    The cluster still doesn’t notice this behavior. If you check the return code of the system replication status, Returncode 11 means error, which tells you something is wrong. If you have access, it is a good idea to enter maintenance-mode now.

    The remotehost3 becomes the new primary, and clusternode2 (DC2) gets automatically registered as the new primary on the remotehost3.

    Example output of the system replication state of remotehost3:

    Every 5.0s: python /usr/sap/RH2/HDB02/exe/python_support/systemReplicationStatus.py ; echo Status $?                               remotehost3: Mon Sep  4 13:55:29 2023
    
    |Database |Host   |Port  |Service Name |Volume ID |Site ID |Site Name |Secondary |Secondary |Secondary |Secondary |Secondary     |Replication |Replication |Replic
    ation    |Secondary    |
    |         |       |      |             |          |        |          |Host      |Port      |Site ID   |Site Name |Active Status |Mode        |Status      |Status
     Details |Fully Synced |
    |-------- |------ |----- |------------ |--------- |------- |--------- |--------- |--------- |--------- |--------- |------------- |----------- |----------- |------
    -------- |------------ |
    |SYSTEMDB |remotehost3 |30201 |nameserver   |        1 |      3 |DC3       |clusternode2    |    30201 |        2 |DC2       |YES           |SYNCMEM     |ACTIVE      |
             |        True |
    |RH2      |remotehost3 |30207 |xsengine     |        2 |      3 |DC3       |clusternode2    |    30207 |        2 |DC2       |YES           |SYNCMEM     |ACTIVE      |
             |        True |
    |RH2      |remotehost3 |30203 |indexserver  |        3 |      3 |DC3       |clusternode2    |    30203 |        2 |DC2       |YES           |SYNCMEM     |ACTIVE      |
             |        True |
    
    status system replication site "2": ACTIVE
    overall system replication status: ACTIVE
    
    Local System Replication State
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    mode: PRIMARY
    site id: 3
    site name: DC3
    Status 15

    The returncode 15 also says everything is okay, but clusternode1 is missing. This must be re-registered manually. The former primary clusternode1 is not listed, so the replication relationship is lost.

  • Set maintenance-mode.

    If not already done before, set maintenance-mode on the cluster on one node of the cluster with the command:

    [root@clusternode1]# pcs property  set maintenance-mode=true

    You can check if the maintenance-mode is active by running this command:

    [root@clusternode1]# pcs resource
      * Clone Set: SAPHanaTopology_RH2_02-clone [SAPHanaTopology_RH2_02] (unmanaged):
        * SAPHanaTopology_RH2_02    (ocf::heartbeat:SAPHanaTopology):        Started clusternode2node2 (unmanaged)
        * SAPHanaTopology_RH2_02    (ocf::heartbeat:SAPHanaTopology):        Started clusternode1node1 (unmanaged)
      * Clone Set: SAPHana_RH2_02-clone [SAPHana_RH2_02] (promotable, unmanaged):
        * SAPHana_RH2_02    (ocf::heartbeat:SAPHana):        Slave clusternode2node2 (unmanaged)
        * SAPHana_RH2_02    (ocf::heartbeat:SAPHana):        Master clusternode1node1 (unmanaged)
      * vip_RH2_02_MASTER   (ocf::heartbeat:IPaddr2):        Started clusternode1node1 (unmanaged)

    The resources are displaying unmanaged, this indicates that the cluster is in maintenance-mode=true. The virtual IP address is still started on clusternode1. If you want to use this IP on another node, please disable vip_RH2_02_MASTER before you set maintanence-mode=true.

    [root@clusternode1]# pcs resource disable vip_RH2_02_MASTER
  • Re-register clusternode1.

    When we check the sr_state on clusternode1, you will see a relationship only to DC2:

    clusternode1:rh2adm> hdbnsutil -sr_state
    
    System Replication State
    ~~~~~~~~~~~~~~~~~~~~~~~~
    
    online: true
    
    mode: primary
    operation mode: primary
    site id: 1
    site name: DC1
    
    is source system: true
    is secondary/consumer system: false
    has secondaries/consumers attached: true
    is a takeover active: false
    is primary suspended: false
    
    Host Mappings:
    ~~~~~~~~~~~~~~
    
    clusternode1 -> [DC2] clusternode2
    clusternode1 -> [DC1] clusternode1
    
    
    Site Mappings:
    ~~~~~~~~~~~~~~
    DC1 (primary/primary)
        |---DC2 (syncmem/logreplay)
    
    Tier of DC1: 1
    Tier of DC2: 2
    
    Replication mode of DC1: primary
    Replication mode of DC2: syncmem
    
    Operation mode of DC1: primary
    Operation mode of DC2: logreplay
    
    Mapping: DC1 -> DC2
    done.

    But when we check DC2, the primary database server is DC3. So the information from DC1 is not correct.

    clusternode2:rh2adm> hdbnsutil -sr_state

    If we check the system replication status on DC1, the returncode is 12, which is unknown. So DC1 needs to be re-registered.

    You can use this command to register the former primary clusternode1 as a new secondary of remotehost3.

    clusternode1:rh2adm> hdbnsutil -sr_register --remoteHost=remotehost3 --remoteInstance=${TINSTANCE} --replicationMode=asyncsyncmem --name=DC1 --remoteName=DC3 --operationMode=logreplay --online

    After the registration is done, you will see on remotehost3 all three sites replicated, and the status (return code) will change to 15.

    If this fails, you have to manually remove the replication relationships on DC1 and DC3. Please follow the instructions described in Register Secondary.

    For example, list the existing relationships with:

    clusternode1:rh2adm> hdbnsutil -sr_state

    To remove the existing relationships you can use:

    clusternode1:rh2adm> hdbnsutil -sr_unregister --name=DC2`

    This may not usually be necessary. We assume that test 4 will be performed after test 3. So the recovery step is to run test 4.

Red Hat logoGithubRedditYoutubeTwitter

学习

尝试、购买和销售

社区

關於紅帽

我们提供强化的解决方案,使企业能够更轻松地跨平台和环境(从核心数据中心到网络边缘)工作。

让开源更具包容性

红帽致力于替换我们的代码、文档和 Web 属性中存在问题的语言。欲了解更多详情,请参阅红帽博客.

关于红帽文档

Legal Notice

Theme

© 2026 Red Hat
返回顶部