This document provides information about installing, configuring and managing Red Hat High Availability Add-On components. Red Hat High Availability Add-On components allow you to connect a group of computers (called nodes or members) to work together as a cluster. In this document, the use of the word cluster or clusters is used to refer to a group of computers running the Red Hat High Availability Add-On.
The audience of this document should have advanced working knowledge of Red Hat Enterprise Linux and understand the concepts of clusters, storage, and server computing.
This document is organized as follows:
For more information about Red Hat Enterprise Linux 6, refer to the following resources:
Red Hat Enterprise Linux Installation Guide — Provides information regarding installation of Red Hat Enterprise Linux 6.
Red Hat Enterprise Linux Deployment Guide — Provides information regarding the deployment, configuration and administration of Red Hat Enterprise Linux 6.
For more information about the High Availability Add-On and related products for Red Hat Enterprise Linux 6, refer to the following resources:
High Availability Add-On Overview — Provides a high-level overview of the Red Hat High Availability Add-On.
Logical Volume Manager Administration — Provides a description of the Logical Volume Manager (LVM), including information on running LVM in a clustered environment.
Global File System 2: Configuration and Administration — Provides information about installing, configuring, and maintaining Red Hat GFS2 (Red Hat Global File System 2), which is included in the Resilient Storage Add-On.
DM Multipath — Provides information about using the Device-Mapper Multipath feature of Red Hat Enterprise Linux 6.
Load Balancer Administration — Provides information on configuring high-performance systems and services with the Load Balancer Add-On, a set of integrated software components that provide Linux Virtual Servers (LVS) for balancing IP load across a set of real servers.
Release Notes — Provides information about the current release of Red Hat products.
High Availability Add-On documentation and other Red Hat documents are available in HTML, PDF, and RPM versions on the Red Hat Enterprise Linux Documentation CD and online at
http://docs.redhat.com/docs/en-US/index.html.
This manual uses several conventions to highlight certain words and phrases and draw attention to specific pieces of information.
In PDF and paper editions, this manual uses typefaces drawn from the
Liberation Fonts set. The Liberation Fonts set is also used in HTML editions if the set is installed on your system. If not, alternative but equivalent typefaces are displayed. Note: Red Hat Enterprise Linux 5 and later includes the Liberation Fonts set by default.
1.1. Typographic Conventions
Four typographic conventions are used to call attention to specific words and phrases. These conventions, and the circumstances they apply to, are as follows.
Mono-spaced Bold
Used to highlight system input, including shell commands, file names and paths. Also used to highlight keycaps and key combinations. For example:
To see the contents of the file my_next_bestselling_novel in your current working directory, enter the cat my_next_bestselling_novel command at the shell prompt and press Enter to execute the command.
The above includes a file name, a shell command and a keycap, all presented in mono-spaced bold and all distinguishable thanks to context.
Key combinations can be distinguished from keycaps by the hyphen connecting each part of a key combination. For example:
Press Enter to execute the command.
Press Ctrl+Alt+F2 to switch to the first virtual terminal. Press Ctrl+Alt+F1 to return to your X-Windows session.
The first paragraph highlights the particular keycap to press. The second highlights two key combinations (each a set of three keycaps with each set pressed simultaneously).
If source code is discussed, class names, methods, functions, variable names and returned values mentioned within a paragraph will be presented as above, in mono-spaced bold. For example:
File-related classes include filesystem for file systems, file for files, and dir for directories. Each class has its own associated set of permissions.
Proportional Bold
This denotes words or phrases encountered on a system, including application names; dialog box text; labeled buttons; check-box and radio button labels; menu titles and sub-menu titles. For example:
Choose → → from the main menu bar to launch Mouse Preferences. In the Buttons tab, click the Left-handed mouse check box and click Close to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand).
To insert a special character into a gedit file, choose → → from the main menu bar. Next, choose → from the Character Map menu bar, type the name of the character in the Search field and click Next. The character you sought will be highlighted in the Character Table. Double-click this highlighted character to place it in the Text to copy field and then click the Copy button. Now switch back to your document and choose → from the gedit menu bar.
The above text includes application names; system-wide menu names and items; application-specific menu names; and buttons and text found within a GUI interface, all presented in proportional bold and all distinguishable by context.
Mono-spaced Bold ItalicProportional Bold Italic
Whether mono-spaced bold or proportional bold, the addition of italics indicates replaceable or variable text. Italics denotes text you do not input literally or displayed text that changes depending on circumstance. For example:
To connect to a remote machine using ssh, type ssh username@domain.name at a shell prompt. If the remote machine is example.com and your username on that machine is john, type ssh john@example.com.
The mount -o remount file-system command remounts the named file system. For example, to remount the /home file system, the command is mount -o remount /home.
To see the version of a currently installed package, use the rpm -q package command. It will return a result as follows: package-version-release
Note the words in bold italics above — username, domain.name, file-system, package, version and release. Each word is a placeholder, either for text you enter when issuing a command or for text displayed by the system.
Aside from standard usage for presenting the title of a work, italics denotes the first use of a new and important term. For example:
Publican is a DocBook publishing system.
1.2. Pull-quote Conventions
Terminal output and source code listings are set off visually from the surrounding text.
Output sent to a terminal is set in mono-spaced roman and presented thus:
books Desktop documentation drafts mss photos stuff svn
books_tests Desktop1 downloads images notes scripts svgs
Source-code listings are also set in mono-spaced roman but add syntax highlighting as follows:
package org.jboss.book.jca.ex1;
import javax.naming.InitialContext;
public class ExClient
{
public static void main(String args[])
throws Exception
{
InitialContext iniCtx = new InitialContext();
Object ref = iniCtx.lookup("EchoBean");
EchoHome home = (EchoHome) ref;
Echo echo = home.create();
System.out.println("Created Echo");
System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));
}
}
Finally, we use three visual styles to draw attention to information that might otherwise be overlooked.
Notes are tips, shortcuts or alternative approaches to the task at hand. Ignoring a note should have no negative consequences, but you might miss out on a trick that makes your life easier.
Important boxes detail things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply. Ignoring a box labeled 'Important' will not cause data loss but may cause irritation and frustration.
Warnings should not be ignored. Ignoring warnings will most likely cause data loss.
If you spot a typo, or if you have thought of a way to make this manual better, we would love to hear from you. Please submit a report in Bugzilla (
http://bugzilla.redhat.com/bugzilla/) against the component .
Be sure to mention the manual identifier:
Cluster_Administration(EN)-6 (2011-11-14T16:26)
By mentioning this manual's identifier, we know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If you have found an error, please include the section number and some of the surrounding text so we can find it easily.
Chapter 2. Before Configuring the Red Hat High Availability Add-On
This chapter describes tasks to perform and considerations to make before installing and configuring the Red Hat High Availability Add-On, and consists of the following sections.
Make sure that your deployment of Red Hat High Availability Add-On meets your needs and can be supported. Consult with an authorized Red Hat representative to verify your configuration prior to deployment. In addition, allow time for a configuration burn-in period to test failure modes.
2.1. General Configuration Considerations
You can configure the Red Hat High Availability Add-On in a variety of ways to suit your needs. Take into account the following general considerations when you plan, configure, and implement your deployment.
- Number of cluster nodes supported
The maximum number of cluster nodes supported by the High Availability Add-On is 16.
- Single site clusters
Only single site clusters are fully supported at this time. Clusters spread across multiple physical locations are not formally supported. For more details and to discuss multi-site clusters, please speak to your Red Hat sales or support representative.
- GFS2
Although a GFS2 file system can be implemented in a standalone system or as part of a cluster configuration, Red Hat does not support the use of GFS2 as a single-node file system. Red Hat does support a number of high-performance single-node file systems that are optimized for single node, and thus have generally lower overhead than a cluster file system. Red Hat recommends using those file systems in preference to GFS2 in cases where only a single node needs to mount the file system. Red Hat will continue to support single-node GFS2 file systems for existing customers.
When you configure a GFS2 file system as a cluster file system, you must ensure that all nodes in the cluster have access to the shared file system. Asymmetric cluster configurations in which some nodes have access to the file system and others do not are not supported.This does not require that all nodes actually mount the GFS2 file system itself.
- No-single-point-of-failure hardware configuration
Clusters can include a dual-controller RAID array, multiple bonded network channels, multiple paths between cluster members and storage, and redundant un-interruptible power supply (UPS) systems to ensure that no single failure results in application down time or loss of data.
Alternatively, a low-cost cluster can be set up to provide less availability than a no-single-point-of-failure cluster. For example, you can set up a cluster with a single-controller RAID array and only a single Ethernet channel.
Certain low-cost alternatives, such as host RAID controllers, software RAID without cluster support, and multi-initiator parallel SCSI configurations are not compatible or appropriate for use as shared cluster storage.
- Data integrity assurance
To ensure data integrity, only one node can run a cluster service and access cluster-service data at a time. The use of power switches in the cluster hardware configuration enables a node to power-cycle another node before restarting that node's HA services during a failover process. This prevents two nodes from simultaneously accessing the same data and corrupting it. Fence devices (hardware or software solutions that remotely power, shutdown, and reboot cluster nodes) are used to guarantee data integrity under all failure conditions.
- Ethernet channel bonding
Cluster quorum and node health is determined by communication of messages among cluster nodes via Ethernet. In addition, cluster nodes use Ethernet for a variety of other critical cluster functions (for example, fencing). With Ethernet channel bonding, multiple Ethernet interfaces are configured to behave as one, reducing the risk of a single-point-of-failure in the typical switched Ethernet connection among cluster nodes and other cluster hardware.
- IPv4 and IPv6
The High Availability Add-On supports both IPv4 and IPv6 Internet Protocols. Support of IPv6 in the High Availability Add-On is new for Red Hat Enterprise Linux 6.
Before configuring Red Hat High Availability Add-On software, make sure that your cluster uses appropriate hardware (for example, supported fence devices, storage devices, and Fibre Channel switches). Refer to the hardware configuration guidelines at
http://www.redhat.com/cluster_suite/hardware/ for the most current hardware compatibility information.
Before deploying the Red Hat High Availability Add-On, you must enable certain IP ports on the cluster nodes and on computers that run luci (the Conga user interface server). The following sections identify the IP ports to be enabled:
2.3.1. Enabling IP Ports on Cluster Nodes
Table 2.1. Enabled IP Ports on Red Hat High Availability Add-On Nodes
|
IP Port Number
|
Protocol
|
Component
|
|---|
|
5404, 5405
|
UDP
|
corosync/cman (Cluster Manager)
|
|
11111
|
TCP
|
ricci (propagates updated cluster information)
|
|
21064
|
TCP
|
dlm (Distributed Lock Manager)
|
|
16851
|
TCP
|
modclusterd
|
2.3.2. Enabling the IP Port for luci
If a cluster node is running luci, port 11111 should already have been enabled.
Table 2.2. Enabled IP Port on a Computer That Runs luci
|
IP Port Number
|
Protocol
|
Component
|
|---|
|
8084
|
TCP
|
luci (Conga user interface server)
|
As of the Red Hat Enterprise Linux 6.1 release, which enabled configuration by means of the /etc/sysconfig/luci file, you can specifically configure the only IP address luci is being served at. You can use this capability if your server infrastructure incorporates more than one network and you want to access luci from the internal network only. To do this, uncomment and edit the line in the file that specifies host. For example, to change the host setting in the file to 10.10.10.10, edit the host line as follows:
host = 10.10.10.10
2.4. Configuring luci with /etc/sysconfig/luci
As of the Red Hat Enterprise Linux 6.1 release, you can configure some aspects of luci's behavior by means of the /etc/sysconfig/luci file. The parameters you can change with this file include auxiliary settings of the running environment used by the init script as well as server configuration. In addition, you can edit this file to modify some application configuration parameters. There are instructions within the file itself describing which configuration parameters you can change by editing this file.
In order to protect the intended format, you should not change the non-configuration lines of the /etc/sysconfig/luci file when you edit the file. Additionally, you should take care to follow the required syntax for this file, particularly for the INITSCRIPT section which does not allow for white spaces around the equal sign and requires that you use quotation marks to enclose strings containing white spaces.
The following example shows how to change the port at which luci is being served by editing the /etc/sysconfig/luci file.
Uncomment the following line in the /etc/sysconfig/luci file:
#port = 4443
Replace 4443 with the desired port number, which must be higher than or equal to 1024 (not a privileged port). For example, you can edit that line of the file as follows to set the port at which luci is being served to 7084.
port = 7084
Restart the luci service for the changes to take effect.
When you modify a configuration parameter in the
/etc/sysconfig/luci file to redefine a default value, you should take care to use the new value in place of the documented default value. For example, when you modify the port at which
luci is being served, make sure that you specify the modified value when you enable an IP port for
luci, as described in
Section 2.3.2, “Enabling the IP Port for luci”.
Modified port and host parameters will automatically be reflected in the URL displayed when the
luci service starts, as described in
Section 3.2, “Starting luci”. You should use this URL to access
luci.
For more complete information on the parameters you can configure with the /etc/sysconfig/luci file, refer to the documentation within the file itself.
2.5. Configuring ACPI For Use with Integrated Fence Devices
If your cluster uses integrated fence devices, you must configure ACPI (Advanced Configuration and Power Interface) to ensure immediate and complete fencing.
If a cluster node is configured to be fenced by an integrated fence device, disable ACPI Soft-Off for that node. Disabling ACPI Soft-Off allows an integrated fence device to turn off a node immediately and completely rather than attempting a clean shutdown (for example, shutdown -h now). Otherwise, if ACPI Soft-Off is enabled, an integrated fence device can take four or more seconds to turn off a node (refer to note that follows). In addition, if ACPI Soft-Off is enabled and a node panics or freezes during shutdown, an integrated fence device may not be able to turn off the node. Under those circumstances, fencing is delayed or unsuccessful. Consequently, when a node is fenced with an integrated fence device and ACPI Soft-Off is enabled, a cluster recovers slowly or requires administrative intervention to recover.
The amount of time required to fence a node depends on the integrated fence device used. Some integrated fence devices perform the equivalent of pressing and holding the power button; therefore, the fence device turns off the node in four to five seconds. Other integrated fence devices perform the equivalent of pressing the power button momentarily, relying on the operating system to turn off the node; therefore, the fence device turns off the node in a time span much longer than four to five seconds.
To disable ACPI Soft-Off, use chkconfig management and verify that the node turns off immediately when fenced. The preferred way to disable ACPI Soft-Off is with chkconfig management: however, if that method is not satisfactory for your cluster, you can disable ACPI Soft-Off with one of the following alternate methods:
The following sections provide procedures for the preferred method and alternate methods of disabling ACPI Soft-Off:
2.5.1. Disabling ACPI Soft-Off with chkconfig Management
You can use chkconfig management to disable ACPI Soft-Off either by removing the ACPI daemon (acpid) from chkconfig management or by turning off acpid.
This is the preferred method of disabling ACPI Soft-Off.
Disable ACPI Soft-Off with chkconfig management at each cluster node as follows:
Run either of the following commands:
Reboot the node.
When the cluster is configured and running, verify that the node turns off immediately when fenced.
You can fence the node with the fence_node command or Conga.
2.5.2. Disabling ACPI Soft-Off with the BIOS
Disabling ACPI Soft-Off with the BIOS may not be possible with some computers.
You can disable ACPI Soft-Off by configuring the BIOS of each cluster node as follows:
Reboot the node and start the BIOS CMOS Setup Utility program.
Navigate to the menu (or equivalent power management menu).
The equivalents to , , and may vary among computers. However, the objective of this procedure is to configure the BIOS so that the computer is turned off via the power button without delay.
Exit the BIOS CMOS Setup Utility program, saving the BIOS configuration.
When the cluster is configured and running, verify that the node turns off immediately when fenced.
You can fence the node with the fence_node command or Conga.
Example 2.1. BIOS CMOS Setup Utility: set to
+---------------------------------------------|-------------------+
| ACPI Function [Enabled] | Item Help |
| ACPI Suspend Type [S1(POS)] |-------------------|
| x Run VGABIOS if S3 Resume Auto | Menu Level * |
| Suspend Mode [Disabled] | |
| HDD Power Down [Disabled] | |
| Soft-Off by PWR-BTTN [Instant-Off | |
| CPU THRM-Throttling [50.0%] | |
| Wake-Up by PCI card [Enabled] | |
| Power On by Ring [Enabled] | |
| Wake Up On LAN [Enabled] | |
| x USB KB Wake-Up From S3 Disabled | |
| Resume by Alarm [Disabled] | |
| x Date(of Month) Alarm 0 | |
| x Time(hh:mm:ss) Alarm 0 : 0 : | |
| POWER ON Function [BUTTON ONLY | |
| x KB Power ON Password Enter | |
| x Hot Key Power ON Ctrl-F1 | |
| | |
| | |
+---------------------------------------------|-------------------+
This example shows set to , and set to .
2.5.3. Disabling ACPI Completely in the grub.conf File
This method completely disables ACPI; some computers do not boot correctly if ACPI is completely disabled. Use this method only if the other methods are not effective for your cluster.
You can disable ACPI completely by editing the grub.conf file of each cluster node as follows:
Open /boot/grub/grub.conf with a text editor.
Reboot the node.
When the cluster is configured and running, verify that the node turns off immediately when fenced.
You can fence the node with the fence_node command or Conga.
Example 2.2. Kernel Boot Command Line with acpi=off Appended to It
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/mapper/vg_doc01-lv_root
# initrd /initrd-[generic-]version.img
#boot=/dev/hda
default=0
timeout=5
serial --unit=0 --speed=115200
terminal --timeout=5 serial console
title Red Hat Enterprise Linux Server (2.6.32-193.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-193.el6.x86_64 ro root=/dev/mapper/vg_doc01-lv_root console=ttyS0,115200n8 acpi=off
initrd /initramrs-2.6.32-131.0.15.el6.x86_64.img
In this example, acpi=off has been appended to the kernel boot command line — the line starting with "kernel /vmlinuz-2.6.32-193.el6.x86_64.img".
2.6. Considerations for Configuring HA Services
You can create a cluster to suit your needs for high availability by configuring HA (high-availability) services. The key component for HA service management in the Red Hat High Availability Add-On, rgmanager, implements cold failover for off-the-shelf applications. In the Red Hat High Availability Add-On, an application is configured with other cluster resources to form an HA service that can fail over from one cluster node to another with no apparent interruption to cluster clients. HA-service failover can occur if a cluster node fails or if a cluster system administrator moves the service from one cluster node to another (for example, for a planned outage of a cluster node).
To create an HA service, you must configure it in the cluster configuration file. An HA service comprises cluster resources. Cluster resources are building blocks that you create and manage in the cluster configuration file — for example, an IP address, an application initialization script, or a Red Hat GFS2 shared partition.
An HA service can run on only one cluster node at a time to maintain data integrity. You can specify failover priority in a failover domain. Specifying failover priority consists of assigning a priority level to each node in a failover domain. The priority level determines the failover order — determining which node that an HA service should fail over to. If you do not specify failover priority, an HA service can fail over to any node in its failover domain. Also, you can specify if an HA service is restricted to run only on nodes of its associated failover domain. (When associated with an unrestricted failover domain, an HA service can start on any cluster node in the event no member of the failover domain is available.)
Figure 2.1, “Web Server Cluster Service Example” shows an example of an HA service that is a web server named "content-webserver". It is running in cluster node B and is in a failover domain that consists of nodes A, B, and D. In addition, the failover domain is configured with a failover priority to fail over to node D before node A and to restrict failover to nodes only in that failover domain. The HA service comprises these cluster resources:
IP address resource — IP address 10.10.10.201.
An application resource named "httpd-content" — a web server application init script /etc/init.d/httpd (specifying httpd).
A file system resource — Red Hat GFS2 named "gfs2-content-webserver".
Clients access the HA service through the IP address 10.10.10.201, enabling interaction with the web server application, httpd-content. The httpd-content application uses the gfs2-content-webserver file system. If node B were to fail, the content-webserver HA service would fail over to node D. If node D were not available or also failed, the service would fail over to node A. Failover would occur with minimal service interruption to the cluster clients. For example, in an HTTP service, certain state information may be lost (like session data). The HA service would be accessible from another cluster node via the same IP address as it was before failover.
An HA service is a group of cluster resources configured into a coherent entity that provides specialized services to clients. An HA service is represented as a resource tree in the cluster configuration file, /etc/cluster/cluster.conf (in each cluster node). In the cluster configuration file, each resource tree is an XML representation that specifies each resource, its attributes, and its relationship among other resources in the resource tree (parent, child, and sibling relationships).
Because an HA service consists of resources organized into a hierarchical tree, a service is sometimes referred to as a resource tree or resource group. Both phrases are synonymous with HA service.
At the root of each resource tree is a special type of resource — a service resource. Other types of resources comprise the rest of a service, determining its characteristics. Configuring an HA service consists of creating a service resource, creating subordinate cluster resources, and organizing them into a coherent entity that conforms to hierarchical restrictions of the service.
There are two major considerations to take into account when configuring an HA service:
The types of resources needed to create a service
Parent, child, and sibling relationships among resources
The types of resources and the hierarchy of resources depend on the type of service you are configuring.
2.7. Configuration Validation
The cluster configuration is automatically validated according to the cluster schema at /usr/share/cluster/cluster.rng during startup time and when a configuration is reloaded. Also, you can validate a cluster configuration any time by using the ccs_config_validate command.
An annotated schema is available for viewing at /usr/share/doc/cman-X.Y.ZZ/cluster_conf.html (for example /usr/share/doc/cman-3.0.12/cluster_conf.html).
Configuration validation checks for the following basic errors:
XML validity — Checks that the configuration file is a valid XML file.
Configuration options — Checks to make sure that options (XML elements and attributes) are valid.
Option values — Checks that the options contain valid data (limited).
The following examples show a valid configuration and invalid configurations that illustrate the validation checks:
Example 2.3. cluster.conf Sample Configuration: Valid File
<cluster name="mycluster" config_version="1">
<logging debug="off"/>
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
</fencedevices>
<rm>
</rm>
</cluster>
Example 2.4. cluster.conf Sample Configuration: Invalid XML
<cluster name="mycluster" config_version="1">
<logging debug="off"/>
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
</fencedevices>
<rm>
</rm>
<cluster> <----------------INVALID
In this example, the last line of the configuration (annotated as "INVALID" here) is missing a slash — it is <cluster> instead of </cluster>.
Example 2.5. cluster.conf Sample Configuration: Invalid Option
<cluster name="mycluster" config_version="1">
<loging debug="off"/> <----------------INVALID
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
</fencedevices>
<rm>
</rm>
<cluster>
In this example, the second line of the configuration (annotated as "INVALID" here) contains an invalid XML element — it is loging instead of logging.
Example 2.6. cluster.conf Sample Configuration: Invalid Option Value
<cluster name="mycluster" config_version="1">
<loging debug="off"/>
<clusternodes>
<clusternode name="node-01.example.com" nodeid="-1"> <--------INVALID
<fence>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
</fencedevices>
<rm>
</rm>
<cluster>
In this example, the fourth line of the configuration (annotated as "INVALID" here) contains an invalid value for the XML attribute, nodeid in the clusternode line for node-01.example.com. The value is a negative value ("-1") instead of a positive value ("1"). For the nodeid attribute, the value must be a positive value.
2.8. Considerations for NetworkManager
The use of NetworkManager is not supported on cluster nodes. If you have installed NetworkManager on your cluster nodes, you should either remove it or disable it.
The cman service will not start if NetworkManager is either running or has been configured to run with the chkconfig command.
2.9. Considerations for Using Quorum Disk
Quorum Disk is a disk-based quorum daemon, qdiskd, that provides supplemental heuristics to determine node fitness. With heuristics you can determine factors that are important to the operation of the node in the event of a network partition. For example, in a four-node cluster with a 3:1 split, ordinarily, the three nodes automatically "win" because of the three-to-one majority. Under those circumstances, the one node is fenced. With qdiskd however, you can set up heuristics that allow the one node to win based on access to a critical resource (for example, a critical network path). If your cluster requires additional methods of determining node health, then you should configure qdiskd to meet those needs.
Configuring qdiskd is not required unless you have special requirements for node health. An example of a special requirement is an "all-but-one" configuration. In an all-but-one configuration, qdiskd is configured to provide enough quorum votes to maintain quorum even though only one node is working.
Overall, heuristics and other qdiskd parameters for your deployment depend on the site environment and special requirements needed. To understand the use of heuristics and other qdiskd parameters, refer to the qdisk(5) man page. If you require assistance understanding and using qdiskd for your site, contact an authorized Red Hat support representative.
If you need to use qdiskd, you should take into account the following considerations:
- Cluster node votes
When using Quorum Disk, each cluster node must have one vote.
- CMAN membership timeout value
The CMAN membership timeout value (the time a node needs to be unresponsive before CMAN considers that node to be dead, and not a member) should be at least two times that of the qdiskd membership timeout value. The reason is because the quorum daemon must detect failed nodes on its own, and can take much longer to do so than CMAN. The default value for CMAN membership timeout is 10 seconds. Other site-specific conditions may affect the relationship between the membership timeout values of CMAN and qdiskd. For assistance with adjusting the CMAN membership timeout value, contact an authorized Red Hat support representative.
- Fencing
To ensure reliable fencing when using qdiskd, use power fencing. While other types of fencing can be reliable for clusters not configured with qdiskd, they are not reliable for a cluster configured with qdiskd.
- Maximum nodes
A cluster configured with qdiskd supports a maximum of 16 nodes. The reason for the limit is because of scalability; increasing the node count increases the amount of synchronous I/O contention on the shared quorum disk device.
- Quorum disk device
A quorum disk device should be a shared block device with concurrent read/write access by all nodes in a cluster. The minimum size of the block device is 10 Megabytes. Examples of shared block devices that can be used by qdiskd are a multi-port SCSI RAID array, a Fibre Channel RAID SAN, or a RAID-configured iSCSI target. You can create a quorum disk device with mkqdisk, the Cluster Quorum Disk Utility. For information about using the utility refer to the mkqdisk(8) man page.
Using JBOD as a quorum disk is not recommended. A JBOD cannot provide dependable performance and therefore may not allow a node to write to it quickly enough. If a node is unable to write to a quorum disk device quickly enough, the node is falsely evicted from a cluster.
2.10. Red Hat High Availability Add-On and SELinux
The High Availability Add-On for Red Hat Enterprise Linux 6 supports SELinux in the enforcing state with the SELinux policy type set to targeted.
For more information about SELinux, refer to Deployment Guide for Red Hat Enterprise Linux 6.
2.11. Multicast Addresses
Red Hat High Availability Add-On nodes communicate among each other using multicast addresses. Therefore, each network switch and associated networking equipment in the Red Hat High Availability Add-On must be configured to enable multicast addresses and support IGMP (Internet Group Management Protocol). Ensure that each network switch and associated networking equipment in the Red Hat High Availability Add-On are capable of supporting multicast addresses and IGMP; if they are, ensure that multicast addressing and IGMP are enabled. Without multicast and IGMP, not all nodes can participate in a cluster, causing the cluster to fail; use UDP unicast in these environments, as described in
Section 2.13, “UDP Unicast Traffic”.
Procedures for configuring network switches and associated networking equipment vary according each product. Refer to the appropriate vendor documentation or other information about configuring network switches and associated networking equipment to enable multicast addresses and IGMP.
2.12. Configuring the iptables Firewall to Allow Cluster Components
You can use the following filtering to allow traffic through the iptables firewall for the various cluster components.
For corosync, use the following filtering. Port 5405 is used to receive multicast traffic.
iptables -I INPUT -p udp -m state --state NEW -m multiport --dports 5404,5405 -j ACCEPT
For ricci:
iptables -I INPUT -p tcp -m state --state NEW -m multiport --dports 11111 -j ACCEPT
For modcluster:
iptables -I INPUT -p tcp -m state --state NEW -m multiport --dports 16851 -j ACCEPT
For luci:
iptables -I INPUT -p tcp -m state --state NEW -m multiport --dports 8084 -j ACCEPT
For DLM:
iptables -I INPUT -p tcp -m state --state NEW -m multiport --dports 21064 -j ACCEPT
After executing these commands, run the following command.
service iptables save ; service iptables restart
In Red Hat Enterprise Linux 6, rgmanager does not access the network directly; rgmanager communication happens by means of corosync network transport. Enabling corosync allows rgmanager (or any corosync clients) to work automatically.
2.13. UDP Unicast Traffic
As of the Red Hat Enterprise Linux 6.2 release, Red Hat High-Availability Add-On nodes can communicate with each other using the UDP Unicast transport mechanism. It is recommended, however, that you use IP multicasting for the cluster network. UDP unicast is an alternative that can be used when IP multicasting is not available. For GFS2 deployments using UDP unicast is not recommended.
You can configure the Red Hat High-Availability Add-On to use UDP unicast by setting the
cman transport="udpu" parameter in the
cluster.conf configuration file. You can also specify Unicast from the page of the
Conga user interface, as described in
Section 3.5.3, “Network Configuration”.
2.14. Considerations for ricci
For Red Hat Enterprise Linux 6,
ricci replaces
ccsd. Therefore, it is necessary that
ricci is running in each cluster node to be able to propagate updated cluster configuration whether it is via the
cman_tool version -r command, the
ccs command, or the
luci user interface server. You can start
ricci by using
service ricci start or by enabling it to start at boot time via
chkconfig. For information on enabling IP ports for
ricci, refer to
Section 2.3.1, “Enabling IP Ports on Cluster Nodes”.
For the Red Hat Enterprise Linux 6.1 release and later, using ricci requires a password the first time you propagate updated cluster configuration from any particular node. You set the ricci password as root after you install ricci on your system with the passwd ricci command, for user ricci.
Chapter 3. Configuring Red Hat High Availability Add-On With Conga
Conga is a graphical user interface that you can use to administer the Red Hat High Availability Add-On. Note, however, that in order to use this interface effectively you need to have a good and clear understanding of the underlying concepts. Learning about cluster configuration by exploring the available features in the user interface is not recommended, as it may result in a system that is not robust enough to keep all services running when components fail.
This chapter consists of the following sections:
Configuring Red Hat High Availability Add-On software with Conga consists of the following steps:
Before starting
luci, ensure that the IP ports on your cluster nodes allow connections to port 11111 from the
luci server on any nodes that
luci will be communicating with. For information on enabling IP ports on cluster nodes, see
Section 2.3.1, “Enabling IP Ports on Cluster Nodes”.
To administer Red Hat High Availability Add-On with Conga, install and run luci as follows:
Select a computer to host luci and install the luci software on that computer. For example:
# yum install luci
Typically, a computer in a server cage or a data center hosts luci; however, a cluster computer can host luci.
Start luci using service luci start. For example:
# service luci start
Starting luci: generating https SSL certificates... done
[ OK ]
Please, point your web browser to https://nano-01:8084 to access luci
As of Red Hat Enterprise Linux release 6.1, you can configure some aspects of
luci's behavior by means of the
/etc/sysconfig/luci file, including the port and host parameters, as described in
Section 2.4, “Configuring luci with /etc/sysconfig/luci”. Modified port and host parameters will automatically be reflected in the URL displayed when the
luci service starts.
At a Web browser, place the URL of the luci server into the URL address box and click Go (or the equivalent). The URL syntax for the luci server is https://luci_server_hostname:luci_server_port. The default value of luci_server_port is 8084.
The first time you access luci, a web browser specific prompt regarding the self-signed SSL certificate (of the luci server) is displayed. Upon acknowledging the dialog box or boxes, your Web browser displays the luci login page.
Although any user able to authenticate on the system that is hosting
luci can log in to
luci, as of Red Hat Enterprise Linux 6.2 only the root user on the system that is running
luci can access any of the
luci components until an administrator (the root user or a user with administrator permission) sets permissions for that user. For information on setting
luci permissions for users, refer to
Section 3.3, “Controlling Access to luci”.
Figure 3.1. luci Homebase page
3.3. Controlling Access to luci
As of Red Hat Enterprise Linux 6.2, the root user or a user who has been granted luci administrator permissions on a system running luci can control access to the various luci components by setting permissions for the individual users on a system. To set the user permissions, log in to luci as root and click the selection in the upper right corner of the luci screen. This brings up the page, showing a dropdown menu from which you can select a user.
After selecting a user, you can check whether to allow the following permissions for that user:
- Luci Administrator
Grants the user the same permissions as the root user, with full permissions on all clusters and the ability to set or remove permissions on all other users except root, whose permissions cannot be restricted.
- Can Create Clusters
- Can Import Existing Clusters
For each cluster that has been created or imported to luci, you can set the following permissions for the indicated user:
- Can View This Cluster
Allows the user to view the specified cluster.
- Can Change the Cluster Configuration
Allows the user to modify the configuration for the specified cluster, with the exception of adding and removing cluster nodes.
- Can Enable, Disable, Relocate, and Migrate Service Groups
- Can Stop, Start, and Reboot Cluster Nodes
- Can Add and Delete Nodes
- Can Remove This Cluster from Luci
Click Submit for the permissions to take affect, or click Reset to return to the initial values.
Creating a cluster with luci consists of naming a cluster, adding cluster nodes to the cluster, entering the ricci passwords for each node, and submitting the request to create a cluster. If the node information and passwords are correct, Conga automatically installs software into the cluster nodes (if the appropriate software packages are not currently installed) and starts the cluster. Create a cluster as follows:
Enter the following parameters on the dialog box, as necessary:
At the text box, enter a cluster name. The cluster name cannot exceed 15 characters.
If each node in the cluster has the same ricci password, you can check to autofill the field as you add nodes.
Enter the node name for a node in the cluster in the column and enter the ricci password for the node in the column.
If your system is configured with a dedicated private network that is used only for cluster traffic, you may want to configure luci to communicate with ricci on an address that is different from the address to which the cluster node name resolves. You can do this by entering that address as the .
If you are using a different port for the ricci agent than the default of 11111, you can change that parameter.
Click on and enter the node name and ricci password for each additional node in the cluster.
If you do not want to upgrade the cluster software packages that are already installed on the nodes when you create the cluster, leave the option selected. If you want to upgrade all cluster software packages, select the option.
Whether you select the or the option, if any of the base cluster components are missing (cman, rgmanager, modcluster and all their dependencies), they will be installed. If they cannot be installed, the node creation will fail.
Select if desired.
Select if clustered storage is required; this downloads the packages that support clustered storage and enables clustered LVM. You should select this only when you have access to the Resilient Storage Add-On or the Scalable File System Add-On.
Click Create Cluster. Clicking Create Cluster causes the following actions:
If you have selected , the cluster software packages are downloaded onto the nodes.
Cluster software is installed onto the nodes (or it is verified that the appropriate software packages are installed).
The cluster configuration file is updated and propagated to each node in the cluster.
The added nodes join the cluster.
A message is displayed indicating that the cluster is being created. When the cluster is ready, the display shows the status of the newly created cluster, as shown in
Figure 3.4, “Cluster node display”. Note that if
ricci is not running on any of the nodes, the cluster creation will fail.
After clicking
Create Cluster to create the cluster, you can add or delete nodes from the cluster by clicking the or function from the menu at the top of the cluster node display page. Unless you are deleting an entire cluster, nodes must be stopped before being deleted. For information on deleting a node from an existing cluster that is currently in operation, see
Section 4.2.4, “Deleting a Member from a Cluster”.
Removing a cluster node from the cluster is a destructive operation that cannot be undone.
3.5. Global Cluster Properties
When you select a cluster to configure, a cluster-specific page is displayed. The page provides an interface for configuring cluster-wide properties. You can configure cluster-wide properties by clicking on along the top of the cluster display. This yields a tabbed interface which provides the following tabs: , , , and . To configure the parameters in those tabs, follow the steps in the following sections. If you do not need to configure parameters in a tab, skip the section for that tab.
3.5.1. Configuring General Properties
Clicking on the tab displays the page, which provides an interface for modifying the configuration version.
The text box displays the cluster name; it does not accept a cluster name change. The only way to change the name of a cluster is to create a new cluster configuration with the new name.
The value is set to 1 at the time of cluster creation and is automatically incremented each time you modify your cluster configuration. However, if you need to set it to another value, you can specify it at the text box.
If you have changed the value, click Apply for this change to take effect.
3.5.2. Configuring Fence Daemon Properties
Clicking on the tab displays the page, which provides an interface for configuring and . The values you configure for these parameters are general fencing properties for the cluster. To configure specific fence devices for the nodes of the cluster, use the menu item of the cluster display, as described in
Section 3.6, “Configuring Fence Devices”.
The parameter is the number of seconds the fence daemon (fenced) waits before fencing a node (a member of the fence domain) after the node has failed. The default value is 0. Its value may be varied to suit cluster and network performance.
The parameter is the number of seconds the fence daemon (fenced) waits before fencing a node after the node joins the fence domain. The default value is 3. A typical setting for is between 20 and 30 seconds, but can vary according to cluster and network performance.
Enter the values required and click Apply for changes to take effect.
For more information about and , refer to the fenced(8) man page.
3.5.3. Network Configuration
Clicking on the tab displays the page, which provides an interface for configuring the network transport type.
You can use this tab to select one of the following options:
This is the default setting. With this option selected, the Red Hat High Availability Add-On software creates a multicast address based on the cluster ID. It generates the lower 16 bits of the address and appends them to the upper portion of the address according to whether the IP protocol is IPv4 or IPv6:
For IPv4 — The address formed is 239.192. plus the lower 16 bits generated by Red Hat High Availability Add-On software.
For IPv6 — The address formed is FF15:: plus the lower 16 bits generated by Red Hat High Availability Add-On software.
The cluster ID is a unique identifier that cman generates for each cluster. To view the cluster ID, run the cman_tool status command on a cluster node.
If you need to use a specific multicast address, select this option enter a multicast address into the text box.
If you do specify a multicast address, you should use the 239.192.x.x series (or FF15:: for IPv6) that cman uses. Otherwise, using a multicast address outside that range may cause unpredictable results. For example, using 224.0.0.x (which is "All hosts on the network") may not be routed correctly, or even routed at all by some hardware.
If you specify a multicast address, make sure that you check the configuration of routers that cluster packets pass through. Some routers may take a long time to learn addresses, seriously impacting cluster performance.
As of the Red Hat Enterprise Linux 6.2 release, Red Hat High-Availability Add-On nodes can communicate each other using the UDP Unicast transport mechanism. It is recommended, however, that you use IP multicasting for the cluster network. UDP unicast is an alternative that can be used when IP multicasting is not available. For GFS2 deployments using UDP unicast is not recommended.
Click Apply. When changing the transport type, a cluster restart is necessary for the changes to take effect.
3.5.4. Quorum Disk Configuration
Clicking on the tab displays the page, which provides an interface for configuring quorum disk parameters to configure if you need to use a quorum disk.
Quorum disk parameters and heuristics depend on the site environment and the special requirements needed. To understand the use of quorum disk parameters and heuristics, refer to the qdisk(5) man page. If you require assistance understanding and using quorum disk, contact an authorized Red Hat support representative.
The parameter is enabled by default. If you need to use a quorum disk, click , enter quorum disk parameters, click Apply, and restart the cluster for the changes to take effect.
Table 3.1. Quorum-Disk Parameters
|
Parameter
|
Description
|
|---|
|
|
Specifies the quorum disk label created by the mkqdisk utility. If this field is used, the quorum daemon reads the /proc/partitions file and checks for qdisk signatures on every block device found, comparing the label against the specified label. This is useful in configurations where the quorum device name differs among nodes.
|
|
|
— The program used to determine if this heuristic is available. This can be anything that can be executed by /bin/sh -c. A return value of 0 indicates success; anything else indicates failure. This field is required. | | — The frequency (in seconds) at which the heuristic is polled. The default interval for every heuristic is 2 seconds. | | — The weight of this heuristic. Be careful when determining scores for heuristics. The default score for each heuristic is 1. | | — The number of consecutive failures required before this heuristic is declared unavailable. |
|
|
|
The minimum score for a node to be considered "alive". If omitted or set to 0, the default function, floor((n+1)/2), is used, where n is the sum of the heuristics scores. The value must never exceed the sum of the heuristic scores; otherwise, the quorum disk cannot be available.
|
3.5.5. Logging Configuration
Clicking on the tab displays the page, which provides an interface for configuring logging settings.
You can configure the following settings for global logging configuration:
Checking enables debugging messages in the log file.
Checking enables messages to syslog. You can select the and the . The setting indicates that messages at the selected level and higher are sent to syslog.
Checking enables messages to the log file. You can specify the name. The setting indicates that messages at the selected level and higher are written to the log file.
You can override the global logging settings for specific daemons by selecting one of the daemons listed beneath the heading at the bottom of the page. After selecting the daemon, you can check whether to log the debugging messages for that particular daemon. You can also specify the syslog and log file settings for that daemon.
Click Apply for the logging configuration changes you have specified to take effect.
3.6. Configuring Fence Devices
Configuring fence devices consists of creating, updating, and deleting fence devices for the cluster. You must configure the fence devices in a cluster before you can configure fencing for the nodes in the cluster.
Creating a fence device consists of selecting a fence device type and entering parameters for that fence device (for example, name, IP address, login, and password). Updating a fence device consists of selecting an existing fence device and changing parameters for that fence device. Deleting a fence device consists of selecting an existing fence device and deleting it.
This section provides procedures for the following tasks:
From the cluster-specific page, you can configure fence devices for that cluster by clicking on along the top of the cluster display. This displays the fence devices for the cluster and displays the menu items for fence device configuration: and . This is the starting point of each procedure described in the following sections.
If this is an initial cluster configuration, no fence devices have been created, and therefore none are displayed.
3.6.1. Creating a Fence Device
To create a fence device, follow these steps:
From the configuration page, click . Clicking displays the Add Fence Device (Instance) dialog box. From this dialog box, select the type of fence device to configure.
Click Submit.
After the fence device has been added, it appears on the configuration page.
3.6.2. Modifying a Fence Device
To modify a fence device, follow these steps:
From the configuration page, click on the name of the fence device to modify. This displays the dialog box for that fence device, with the values that have been configured for the device.
Click Apply and wait for the configuration to be updated.
3.6.3. Deleting a Fence Device
Fence devices that are in use cannot be deleted. To delete a fence device that a node is currently using, first update the node fence configuration for any node using the device and then delete the device.
To delete a fence device, follow these steps:
From the configuration page, check the box to the left of the fence device or devices to select the devices to delete.
Click Delete and wait for the configuration to be updated. A message appears indicating which devices are being deleted.
When the configuration has been updated, the deleted fence device no longer appears in the display.
3.7. Configuring Fencing for Cluster Members
Once you have completed the initial steps of creating a cluster and creating fence devices, you need to configure fencing for the cluster nodes. To configure fencing for the nodes after creating a new cluster and configuring the fencing devices for the cluster, follow the steps in this section. Note that you must configure fencing for each node in the cluster.
The following sections provide procedures for configuring a single fence device for a node, configuring a node with a backup fence device, and configuring a node with redundant power supplies:
3.7.1. Configuring a Single Fence Device for a Node
Use the following procedure to configure a node with a single fence device.
From the cluster-specific page, you can configure fencing for the nodes in the cluster by clicking on along the top of the cluster display. This displays the nodes that constitute the cluster. This is also the default page that appears when you click on the cluster name beneath from the menu on the left side of the luci page.
Click on a node name. Clicking a link for a node causes a page to be displayed for that link showing how that node is configured.
The node-specific page displays any services that are currently running on the node, as well as any failover domains of which this node is a member. You can modify an existing failover domain by clicking on its name. For information on configuring failover domains, see
Section 3.8, “Configuring a Failover Domain”.
On the node-specific page, under , click Add Fence Method. This displays the Add Fence Method to Node dialog box.
Enter a for the fencing method that you are configuring for this node. This is an arbitrary name that will be used by Red Hat High Availability Add-On; it is not the same as the DNS name for the device.
Click Submit. This displays the node-specific screen that now displays the method you have just added under .
Configure a fence instance for this method by clicking the
Add Fence Instance button that appears beneath the fence method. This displays the drop-down menu from which you can select a fence device you have previously configured, as described in
Section 3.6.1, “Creating a Fence Device”.
Select a fence device for this method. If this fence device requires that you configure node-specific parameters, the display shows the parameters to configure. For information on fencing parameters, refer to
Appendix A, Fence Device Parameters.
For non-power fence methods (that is, SAN/storage fencing), is selected by default on the node-specific parameters display. This ensures that a fenced node's access to storage is not re-enabled until the node has been rebooted. For information on unfencing a node, refer to the fence_node(8) man page.
Click Submit. This returns you to the node-specific screen with the fence method and fence instance displayed.
3.7.2. Configuring a Backup Fence Device
You can define multiple fencing methods for a node. If fencing fails using the first method, the system will attempt to fence the node using the second method, followed by any additional methods you have configured.
Use the following procedure to configure a backup fence device for a node.
Beneath the display of the primary method you defined, click Add Fence Method.
Enter a name for the backup fencing method that you are configuring for this node and click Submit. This displays the node-specific screen that now displays the method you have just added, below the primary fence method.
Configure a fence instance for this method by clicking
Add Fence Instance. This displays a drop-down menu from which you can select a fence device you have previously configured, as described in
Section 3.6.1, “Creating a Fence Device”.
Select a fence device for this method. If this fence device requires that you configure node-specific parameters, the display shows the parameters to configure. For information on fencing parameters, refer to
Appendix A, Fence Device Parameters.
Click Submit. This returns you to the node-specific screen with the fence method and fence instance displayed.
You can continue to add fencing methods as needed. You can rearrange the order of fencing methods that will be used for this node by clicking on and .
3.7.3. Configuring a Node with Redundant Power
If your cluster is configured with redundant power supplies for your nodes, you must be sure to configure fencing so that your nodes fully shut down when they need to be fenced. If you configure each power supply as a separate fence method, each power supply will be fenced separately; the second power supply will allow the system to continue running when the first power supply is fenced and the system will not be fenced at all. To configure a system with dual power supplies, you must configure your fence devices so that both power supplies are shut off and the system is taken completely down. When configuring your system using Conga, this requires that you configure two instances within a single fencing method.
To configure fencing for a node with dual power supplies, follow the steps in this section.
Before you can configure fencing for a node with redundant power, you must configure each of the power switches as a fence device for the cluster. For information on configuring fence devices, see
Section 3.6, “Configuring Fence Devices”.
From the cluster-specific page, click on along the top of the cluster display. This displays the nodes that constitute the cluster. This is also the default page that appears when you click on the cluster name beneath from the menu on the left side of the luci page.
Click on a node name. Clicking a link for a node causes a page to be displayed for that link showing how that node is configured.
On the node-specific page, click Add Fence Method.
Enter a name for the fencing method that you are configuring for this node.
Click Submit. This displays the node-specific screen that now displays the method you have just added under .
Configure the first power supply as a fence instance for this method by clicking
Add Fence Instance. This displays a drop-down menu from which you can select one of the power fencing devices you have previously configured, as described in
Section 3.6.1, “Creating a Fence Device”.
Select one of the power fence devices for this method and enter the appropriate parameters for this device.
Click Submit. This returns you to the node-specific screen with the fence method and fence instance displayed.
Under the same fence method for which you have configured the first power fencing device, click
Add Fence Instance. This displays a drop-down menu from which you can select the second power fencing devices you have previously configured, as described in
Section 3.6.1, “Creating a Fence Device”.
Select the second of the power fence devices for this method and enter the appropriate parameters for this device.
Click
Submit. This returns you to the node-specific screen with the fence methods and fence instances displayed, showing that each device will power the system off in sequence and power the system on in sequence. This is shown in
Figure 3.6, “Dual-Power Fencing Configuration”.
3.8. Configuring a Failover Domain
A failover domain is a named subset of cluster nodes that are eligible to run a cluster service in the event of a node failure. A failover domain can have the following characteristics:
Unrestricted — Allows you to specify that a subset of members are preferred, but that a cluster service assigned to this domain can run on any available member.
Restricted — Allows you to restrict the members that can run a particular cluster service. If none of the members in a restricted failover domain are available, the cluster service cannot be started (either manually or by the cluster software).
Unordered — When a cluster service is assigned to an unordered failover domain, the member on which the cluster service runs is chosen from the available failover domain members with no priority ordering.
Ordered — Allows you to specify a preference order among the members of a failover domain. The member at the top of the list is the most preferred, followed by the second member in the list, and so on.
Failback — Allows you to specify whether a service in the failover domain should fail back to the node that it was originally running on before that node failed. Configuring this characteristic is useful in circumstances where a node repeatedly fails and is part of an ordered failover domain. In that circumstance, if a node is the preferred node in a failover domain, it is possible for a service to fail over and fail back repeatedly between the preferred node and another node, causing severe impact on performance.
The failback characteristic is applicable only if ordered failover is configured.
Changing a failover domain configuration has no effect on currently running services.
Failover domains are not required for operation.
By default, failover domains are unrestricted and unordered.
In a cluster with several members, using a restricted failover domain can minimize the work to set up the cluster to run a cluster service (such as httpd), which requires you to set up the configuration identically on all members that run the cluster service. Instead of setting up the entire cluster to run the cluster service, you can set up only the members in the restricted failover domain that you associate with the cluster service.
To configure a preferred member, you can create an unrestricted failover domain comprising only one cluster member. Doing that causes a cluster service to run on that cluster member primarily (the preferred member), but allows the cluster service to fail over to any of the other members.
The following sections describe adding, modifying, and deleting a failover domain:
3.8.1. Adding a Failover Domain
To add a failover domain, follow the steps in this section.
From the cluster-specific page, you can configure failover domains for that cluster by clicking on along the top of the cluster display. This displays the failover domains that have been configured for this cluster.
In the Add Failover Domain to Cluster dialog box, specify a failover domain name at the text box.
The name should be descriptive enough to distinguish its purpose relative to other names used in your cluster.
To enable setting failover priority of the members in the failover domain, click the checkbox. With checked, you can set the priority value, , for each node selected as members of the failover domain.
To restrict failover to members in this failover domain, click the checkbox. With checked, services assigned to this failover domain fail over only to nodes in this failover domain.
To specify that a node does not fail back in this failover domain, click the checkbox. With checked, if a service fails over from a preferred node, the service does not fail back to the original node once it has recovered.
Configure members for this failover domain. Click the checkbox for each node that is to be a member of the failover domain. If is checked, set the priority in the text box for each member of the failover domain.
Click Create. This displays the Failover Domains page with the newly-created failover domain displayed. A message indicates that the new domain is being created. Refresh the page for an updated status.
3.8.2. Modifying a Failover Domain
To modify a failover domain, follow the steps in this section.
From the cluster-specific page, you can configure Failover Domains for that cluster by clicking on along the top of the cluster display. This displays the failover domains that have been configured for this cluster.
Click on the name of a failover domain. This displays the configuration page for that failover domain.
To modify the , , or properties for the failover domain, click or unclick the checkbox next to the property and click Update Properties.
To modify the failover domain membership, click or unclick the checkbox next to the cluster member. If the failover domain is prioritized, you can also modify the priority setting for the cluster member. Click Update Settings.
3.8.3. Deleting a Failover Domain
To delete a failover domain, follow the steps in this section.
From the cluster-specific page, you can configure Failover Domains for that cluster by clicking on along the top of the cluster display. This displays the failover domains that have been configured for this cluster.
Select the checkbox for the failover domain to delete.
Click on .
3.9. Configuring Global Cluster Resources
You can configure global resources that can be used by any service running in the cluster, and you can configure resources that are available only to a specific service.
From the cluster-specific page, you can add resources to that cluster by clicking on along the top of the cluster display. This displays the resources that have been configured for that cluster.
Click . This displays the drop-down menu.
Click the drop-down box under and select the type of resource to configure.
Click Submit. Clicking Submit returns to the resources page that displays the display of Resources, which displays the added resource (and other resources).
To modify an existing resource, perform the following steps.
From the luci page, click on the name of the resource to modify. This displays the parameters for that resource.
Edit the resource parameters.
Click Apply.
To delete an existing resource, perform the following steps.
From the luci page, click the checkbox for any resources to delete.
Click .
3.10. Adding a Cluster Service to the Cluster
To add a cluster service to the cluster, follow the steps in this section.
From the cluster-specific page, you can add services to that cluster by clicking on along the top of the cluster display. This displays the services that have been configured for that cluster. (From the page, you can also start, restart, and disable a service, as described in
Section 4.4, “Managing High-Availability Services”.)
Click . This displays the dialog box.
On the Add Service to Cluster dialog box, at the text box, type the name of the service.
Use a descriptive name that clearly distinguishes the service from other services in the cluster.
Check the checkbox if you want the service to start automatically when a cluster is started and running. If the checkbox is not checked, the service must be started manually any time the cluster comes up from the stopped state.
Check the checkbox to set a policy wherein the service only runs on nodes that have no other services running on them.
If you have configured failover domains for the cluster, you can use the drop-down menu of the parameter to select a failover domain for this service. For information on configuring failover domains, see
Section 3.8, “Configuring a Failover Domain”.
Use the drop-down box to select a recovery policy for the service. The options are to , , , or the service.
Selecting the option indicates that the system should attempt to restart the failed service before relocating the service. Selecting the option indicates that the system should attempt to restart the service in place if it fails, but if restarting the service fails the service will be disabled instead of being moved to another host in the cluster.
If you select or as the recovery policy for the service, you can specify the maximum number of restart failures before relocating or disabling the service, and you can specify the length of time in seconds after which to forget a restart.
To add a resource to the service, click Add resource. Clicking Add resource causes the display of the Add Resource To Service drop-down box that allows you to add an existing global resource or to add a new resource that is available only to this service.
To add an existing global resource, click on the name of the existing resource from the
Add Resource To Service drop-down box. This displays the resource and its parameters on the page for the service you are configuring. For information on adding or modifying global resources, see
Section 3.9, “Configuring Global Cluster Resources”).
To add a new resource that is available only to this service, select the type of resource to configure from the
Add Resource To Service drop-down box and enter the resource parameters for the resource you are adding.
Appendix B, HA Resource Parameters describes resource parameters.
When adding a resource to a service, whether it is an existing global resource or a resource available only to this service, you can specify whether the resource is an or a .
If you specify that a resource is an independent subtree, then if that resource fails only that resource is restarted (rather than the entire service) before the system attempting normal recovery. You can specify the maximum number of restarts to attempt for that resource on a node before implementing the recovery policy for the service. You can also specify the length of time in seconds after which the system will implement the recovery policy for the service.
If you specify that the resource is a non-critical resource, then if that resource fails only that resource is restarted, and if the resource continues to fail then only that resource is disabled, rather than the entire service. You can specify the maximum number of restarts to attempt for that resource on a node before disabling that resource. You can also specify the length of time in seconds after which the system will disable that resource.
If you want to add child resources to the resource you are defining, click Add a child resource. Clicking Add a child resource causes the display of the drop-down box, from which you can add an existing global resource or add a new resource that is available only to this service. You can continue adding children resources to the resource to suit your requirements.
If you are adding a Samba-service resource, add it directly to the service, not as a child of another resource.
When you have completed adding resources to the service, and have completed adding children resources to resources, click Submit. Clicking Submit returns to the page displaying the added service (and other services).
To verify the existence of the IP service resource used in a cluster service, you use the /sbin/ip addr list command on a cluster node. The following output shows the /sbin/ip addr list command executed on a node running a cluster service:
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1356 qdisc pfifo_fast qlen 1000
link/ether 00:05:5d:9a:d8:91 brd ff:ff:ff:ff:ff:ff
inet 10.11.4.31/22 brd 10.11.7.255 scope global eth0
inet6 fe80::205:5dff:fe9a:d891/64 scope link
inet 10.11.4.240/22 scope global secondary eth0
valid_lft forever preferred_lft forever
To modify an existing service, perform the following steps.
From the dialog box, click on the name of the service to modify. This displays the parameters and resources that have been configured for that service.
Edit the service parameters.
Click Submit.
To delete an existing service, perform the following steps.
From the luci page, click the checkbox for any services to delete.
Click .
Chapter 5. Configuring Red Hat High Availability Add-On With the ccs Command
As of the Red Hat Enterprise Linux 6.1 release and later, the Red Hat High Availability Add-On provides support for the ccs cluster configuration command. The ccs command allows an administrator to create, modify and view the cluster.conf cluster configuration file. You can use the ccs command to configure a cluster configuration file on a local file system or on a remote node. Using the ccs command, an administrator can also start and stop the cluster services on one or all of the nodes in a configured cluster.
This chapter consists of the following sections:
Make sure that your deployment of High Availability Add-On meets your needs and can be supported. Consult with an authorized Red Hat representative to verify your configuration prior to deployment. In addition, allow time for a configuration burn-in period to test failure modes.
This chapter references commonly used cluster.conf elements and attributes. For a comprehensive list and description of cluster.conf elements and attributes, refer to the cluster schema at /usr/share/cluster/cluster.rng, and the annotated schema at /usr/share/doc/cman-X.Y.ZZ/cluster_conf.html (for example /usr/share/doc/cman-3.0.12/cluster_conf.html).
5.1. Operational Overview
This section describes the following general operational aspects of using the ccs command to configure a cluster:
5.1.1. Creating the Cluster Configuration File on a Local System
Using the ccs command, you can create a cluster configuration file on a cluster node, or you can create a cluster configuration file on a local file system and then send that file to a host in a cluster. This allows you to work on a file from a local machine, where you can maintain it under version control or otherwise tag the file according to your needs. Using the ccs command does not require root privilege.
When you create and edit a cluster configuration file on a cluster node with the ccs command, you use the -h option to specify the name of the host. This creates and edits the cluster.conf file on the host:
ccs -h host [options]
To create and edit a cluster configuration file on a local system, use the -f option of the ccs command to specify the name of the configuration file when you perform a cluster operation. You can name this file anything you want.
ccs -f file [options]
After you have created the file locally you can send it to a cluster node using the --setconf option of the ccs command. On a host machine in a cluster, the file you send will be named cluster.conf and it will be placed in the /etc/cluster directory.
ccs -h host -f file --setconf
5.1.2. Viewing the Current Cluster Configuration
This chapter describes how to create a cluster configuration file. If at any time you want to print the current file for a cluster, use the following command, specifying a node in the cluster as the host:
ccs -h host --getconf
5.1.3. Specifying ricci Passwords with the ccs Command
Executing
ccs commands that distribute copies of the
cluster.conf file to the nodes of a cluster requires that
ricci be installed and running on the cluster nodes, as described in
Section 2.14, “Considerations for ricci”. Using
ricci requires a password the first time you interact with
ricci from any specific machine.
If you have not entered a password for an instance of ricci on a particular machine from the machine you are using, you will be prompted for that password when the ccs command requires it. Alternately, you can use the -p option to specify a ricci password on the command line.
ccs -h host -p password --sync --activate
When you propagate the cluster.conf file to all of the nodes in the cluster with the --sync option of the ccs command and you specify a ricci password for the command, the ccs command will use that password for each node in the cluster. If you need to set different passwords for ricci on individual nodes, you can use the --setconf option with the -p option to distribute the configuration file to one node at a time.
5.1.4. Modifying Cluster Configuration Components
You use the ccs command to configure cluster components and their attributes in the cluster configuration file. After you have added a cluster component to the file, in order to modify the attributes of that component you must remove the component you have defined and add the component again, with the modified attributes. Information on how to do this with each component is provided in the individual sections of this chapter.
The attributes of the cman cluster component provide an exception to this procedure for modifying cluster components. To modify these attributes, you execute the --setcman option of the ccs command, specifying the new attributes.
Configuring Red Hat High Availability Add-On software with the ccs consists of the following steps:
In order to create and distribute cluster configuration files on the nodes of the cluster, the ricci service must be running on each node. Before starting ricci, you should ensure that you have configured your system as follows:
After ricci has been installed and configured on each node, start the ricci service on each node:
# service ricci start
Starting ricci: [ OK ]
This section describes how to create, modify, and delete a skeleton cluster configuration with the ccs command without fencing, failover domains, and HA services. Subsequent sections describe how to configure those parts of the configuration.
To create a skeleton cluster configuration file, first create and name the cluster and then add the nodes to the cluster, as in the following procedure:
Create a cluster configuration file on one of the nodes in the cluster by executing the ccs command using the -h parameter to specify the node on which to create the file and the createcluster option to specify a name for the cluster:
ccs -h host --createcluster clustername
For example, the following command creates a configuration file on node-01.example.com named mycluster:
ccs -h node-01.example.com --createcluster mycluster
The cluster name cannot exceed 15 characters.
If a cluster.conf file already exists on the host that you specify, executing this command will replace that existing file.
To configure the nodes that the cluster contains, execute the following command for each node in the cluster:
ccs -h host --addnode node
For example, the following three commands add the nodes node-01.example.com, node-02.example.com, and node-03.example.com to the configuration file on node-01.example.com:
ccs -h node-01.example.com --addnode node-01.example.com
ccs -h node-01.example.com --addnode node-02.example.com
ccs -h node-01.example.com --addnode node-03.example.com
To view a list of the nodes that have been configured for a cluster, execute the following command:
ccs -h host --lsnodes
Example 5.1. cluster.conf File After Adding Three Nodes
<cluster name="mycluster" config_version="2">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
</fencedevices>
<rm>
</rm>
</cluster>
When you add a node to the cluster, you can specify the number of votes the node contributes to determine whether there is a quorum. To set the number of votes for a cluster node, use the following command:
ccs -h host --addnode host --votes votes
When you add a node, the ccs assigns the node a unique integer that is used as the node identifier. If you want to specify the node identifier manually when creating a node, use the following command:
ccs -h host --addnode host --nodeide nodeid
To remove a node from a cluster, execute the following command:
ccs -h host --rmnode node
5.5. Configuring Fence Devices
Configuring fence devices consists of creating, updating, and deleting fence devices for the cluster. You must create and name the fence devices in a cluster before you can configure fencing for the nodes in the cluster. For information on configuring fencing for the individual nodes in the cluster, see
Section 5.7, “Configuring Fencing for Cluster Members”.
Before configuring your fence devices, you may want to modify some of the fence daemon properties for your system from the default values. The values you configure for the fence daemon are general values for the cluster. The general fencing properties for the cluster you may want to modify are summarized as follows:
The post_fail_delay attribute is the number of seconds the fence daemon (fenced) waits before fencing a node (a member of the fence domain) after the node has failed. The post_fail_delay default value is 0. Its value may be varied to suit cluster and network performance.
To configure a value for the post_fail_delay attribute, execute the following command:
ccs -h host --setfencedaemon post_fail_delay=value
The post-join_delay attribute is the number of seconds the fence daemon (fenced) waits before fencing a node after the node joins the fence domain. The post_join_delay default value is 3. A typical setting for post_join_delay is between 20 and 30 seconds, but can vary according to cluster and network performance.
To configure a value for the post_join attribute, execute the following command:
ccs -h host --setfencedaemon post_join_delay=value
For more information about the post_join_delay and post_fail_delay attributes as well as the additional fence daemon properties you can modify, refer to the fenced(8) man page and refer to the cluster schema at /usr/share/cluster/cluster.rng, and the annotated schema at /usr/share/doc/cman-X.Y.ZZ/cluster_conf.html.
To configure a fence device for a cluster, execute the following command:
ccs -h host --addfencedev devicename [fencedeviceoptions]
For example, to configure an APC fence device in the configuration file on the cluster node node1 named myfence with an IP address of apc_ip_example, a login of login_example, and a password of password_example, execute the following command:
ccs -h node1 --addfencedev myfence agent=fence_apc ipaddr=apc_ip_example login=login_example passwd=password_example
The following example shows the fencedevices section of the cluster.conf configuration file after you have added this APC fence device:
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="myfence" passwd="password_example"/>
</fencedevices>
When configuring fence devices for a cluster, you may find it useful to see a listing of available devices for your cluster and the options available for each device. You may also find it useful to see a listing of fence devices currently configured for your cluster. For information on using the
ccs command to print a list of available fence devices and options or to print a list of fence devices currently configured for your cluster, refer to
Section 5.6, “Listing Fence Devices and Fence Device Options”.
To remove a fence device from your cluster configuration, execute the following command:
ccs -h host --rmfencedev fence_device_name
For example, to remove a fence device that you have named myfence from the cluster configuration file on cluster node node1, execute the following command:
ccs -h node1 --rmfencedev myfence
If you need to modify the attributes of a fence device you have already configured, you must first remove that fence device then add it again with the modified attributes.
5.6. Listing Fence Devices and Fence Device Options
You can use the ccs command to print a list of available fence devices and to print a list of options for each available fence type. You can also use the ccs command to print a list of fence devices currently configured for your cluster.
To print a list of fence devices currently available for your cluster, execute the following command:
ccs -h host --lsfenceopts
For example, the following command lists the fence devices available on the cluster node node1, showing sample output.
[root@ask-03 ~]# ccs -h node1 --lsfenceopts
fence_rps10 - RPS10 Serial Switch
fence_vixel - No description available
fence_egenera - No description available
fence_xcat - No description available
fence_na - Node Assassin
fence_apc - Fence agent for APC over telnet/ssh
fence_apc_snmp - Fence agent for APC over SNMP
fence_bladecenter - Fence agent for IBM BladeCenter
fence_bladecenter_snmp - Fence agent for IBM BladeCenter over SNMP
fence_cisco_mds - Fence agent for Cisco MDS
fence_cisco_ucs - Fence agent for Cisco UCS
fence_drac5 - Fence agent for Dell DRAC CMC/5
fence_eps - Fence agent for ePowerSwitch
fence_ibmblade - Fence agent for IBM BladeCenter over SNMP
fence_ifmib - Fence agent for IF MIB
fence_ilo - Fence agent for HP iLO
fence_ilo_mp - Fence agent for HP iLO MP
fence_intelmodular - Fence agent for Intel Modular
fence_ipmilan - Fence agent for IPMI over LAN
fence_kdump - Fence agent for use with kdump
fence_rhevm - Fence agent for RHEV-M REST API
fence_rsa - Fence agent for IBM RSA
fence_sanbox2 - Fence agent for QLogic SANBox2 FC switches
fence_scsi - fence agent for SCSI-3 persistent reservations
fence_virsh - Fence agent for virsh
fence_virt - Fence agent for virtual machines
fence_vmware - Fence agent for VMware
fence_vmware_soap - Fence agent for VMware over SOAP API
fence_wti - Fence agent for WTI
fence_xvm - Fence agent for virtual machines
To print a list of the options you can specify for a particular fence type, execute the following command:
ccs -h host --lsfenceopts fence_type
For example, the following command lists the fence options for the fence_wti fence agent.
[root@ask-03 ~]# ccs -h node1 --lsfenceopts fence_wti
fence_wti - Fence agent for WTI
Required Options:
Optional Options:
option: No description available
action: Fencing Action
ipaddr: IP Address or Hostname
login: Login Name
passwd: Login password or passphrase
passwd_script: Script to retrieve password
cmd_prompt: Force command prompt
secure: SSH connection
identity_file: Identity file for ssh
port: Physical plug number or name of virtual machine
inet4_only: Forces agent to use IPv4 addresses only
inet6_only: Forces agent to use IPv6 addresses only
ipport: TCP port to use for connection with device
verbose: Verbose mode
debug: Write debug information to given file
version: Display version information and exit
help: Display help and exit
separator: Separator for CSV created by operation list
power_timeout: Test X seconds for status change after ON/OFF
shell_timeout: Wait X seconds for cmd prompt after issuing command
login_timeout: Wait X seconds for cmd prompt after login
power_wait: Wait X seconds after issuing ON/OFF
delay: Wait X seconds before fencing is started
retry_on: Count of attempts to retry power on
To print a list of fence devices currently configured for your cluster, execute the following command:
ccs -h host --lsfencedev
5.7. Configuring Fencing for Cluster Members
Once you have completed the initial steps of creating a cluster and creating fence devices, you need to configure fencing for the cluster nodes. To configure fencing for the nodes after creating a new cluster and configuring the fencing devices for the cluster, follow the steps in this section. Note that you must configure fencing for each node in the cluster.
This section documents the following procedures:
5.7.1. Configuring a Single Power-Based Fence Device for a Node
Use the following procedure to configure a node with a single power-based fence device that uses a fence device named apc, which uses the fence_apc fencing agent.
Add a fence method for the node, providing a name for the fence method.
ccs -h host --addmethod method node
For example, to configure a fence method named APC for the node node-01.example.com in the configuration file on the cluster node node-01.example.com, execute the following command:
ccs -h node01.example.com --addmethod APC node01.example.com
Add a fence instance for the method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node:
ccs -h host --addfenceinst fencedevicename node method [options]
For example, to configure a fence instance in the configuration file on the cluster node node-01.example.com that uses the APC switch power port 1 on the fence device named apc to fence cluster node node-01.example.com using the method named APC, execute the following command:
ccs -h node01.example.com --addfenceinst apc node01.example.com APC port=1
You will need to add a fence method for each node in the cluster. The following commands configure a fence method for each node with the method name
APC. The device for the fence method specifies
apc as the device name, which is a device previously configured with the
--addfencedev option, as described in
Section 5.5, “Configuring Fence Devices”. Each node is configured with a unique APC switch power port number: The port number for
node-01.example.com is
1, the port number for
node-02.example.com is
2, and the port number for
node-03.example.com is
3.
ccs -h node01.example.com --addmethod APC node01.example.com
ccs -h node01.example.com --addmethod APC node02.example.com
ccs -h node01.example.com --addmethod APC node03.example.com
ccs -h node01.example.com --addfenceinst apc node01.example.com APC port=1
ccs -h node01.example.com --addfenceinst apc node02.example.com APC port=2
ccs -h node01.example.com --addfenceinst apc node03.example.com APC port=3
Example 5.2. cluster.conf After Adding Power-Based Fence Methods
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="APC">
<device name="apc" port="1"/>
</method>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="APC">
<device name="apc" port="2"/>
</method>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="APC">
<device name="apc" port="3"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/>
</fencedevices>
<rm>
</rm>
</cluster>
5.7.2. Configuring a Single Storage-Based Fence Device for a Node
When using non-power fencing methods (that is, SAN/storage fencing) to fence a node, you must configure unfencing for the fence device. This ensures that a fenced node is not re-enabled until the node has been rebooted. When you configure unfencing for a node, you specify a device that mirrors the corresponding fence device you have configured for the node with the notable addition of the explicit action of on or enable.
For more information about unfencing a node, refer to the fence_node(8) man page.
Use the following procedure to configure a node with a single storage-based fence device that uses a fence device named sanswitch1, which uses the fence_sanbox2 fencing agent.
Add a fence method for the node, providing a name for the fence method.
ccs -h host --addmethod method node
For example, to configure a fence method named SAN for the node node-01.example.com in the configuration file on the cluster node node-01.example.com, execute the following command:
ccs -h node01.example.com --addmethod SAN node01.example.com
Add a fence instance for the method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node:
ccs -h host --addfenceinst fencedevicename node method [options]
For example, to configure a fence instance in the configuration file on the cluster node node-01.example.com that uses the SAN switch power port 11 on the fence device named sanswitch1 to fence cluster node node-01.example.com using the method named SAN, execute the following command:
ccs -h node01.example.com --addfenceinst sanswitch1 node01.example.com SAN port=11
To configure unfencing for the storage based fence device on this node, execute the following command:
ccs -h host --addunfence fencedevicename node action=on|off
You will need to add a fence method for each node in the cluster. The following commands configure a fence method for each node with the method name
SAN. The device for the fence method specifies
sanswitch as the device name, which is a device previously configured with the --addfencedev option, as described in
Section 5.5, “Configuring Fence Devices”. Each node is configured with a unique SAN physical port number: The port number for
node-01.example.com is
11, the port number for
node-02.example.com is
12, and the port number for
node-03.example.com is
13.
ccs -h node01.example.com --addmethod SAN node01.example.com
ccs -h node01.example.com --addmethod SAN node02.example.com
ccs -h node01.example.com --addmethod SAN node03.example.com
ccs -h node01.example.com --addfenceinst sanswitch1 node01.example.com SAN port=11
ccs -h node01.example.com --addfenceinst sanswitch1 node02.example.com SAN port=12
ccs -h node01.example.com --addfenceinst sanswitch1 node03.example.com SAN port=13
ccs -h node01.example.com --addunfence sanswitch1 node01.example.com port=11 action=on
ccs -h node01.example.com --addunfence sanswitch1 node02.example.com port=12 action=on
ccs -h node01.example.com --addunfence sanswitch1 node03.example.com port=13 action=on
Example 5.3. cluster.conf After Adding Storage-Based Fence Methods
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="SAN">
<device name="sanswitch1" port="11"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="11" action="on"/>
</unfence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="SAN">
<device name="sanswitch1" port="12"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="12" action="on"/>
</unfence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="SAN">
<device name="sanswitch1" port="13"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="13" action="on"/>
</unfence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_sanbox2" ipaddr="san_ip_example"
login="login_example" name="sanswitch1" passwd="password_example"/>
</fencedevices>
<rm>
</rm>
</cluster>
5.7.3. Configuring a Backup Fence Device
You can define multiple fencing methods for a node. If fencing fails using the first method, the system will attempt to fence the node using the second method, followed by any additional methods you have configured. To configure a backup fencing method for a node, you configure two methods for a node, configuring a fence instance for each method.
The order in which the system will use the fencing methods you have configured follows their order in the cluster configuration file. The first method you configure with the ccs command is the primary fencing method, and the second method you configure is the backup fencing method. To change the order, you can remove the primary fencing method from the configuration file, then add that method back.
Note that at any time you can print a list of fence methods and instances currently configured for a node by executing the following command. If you do not specify a node, this command will list the fence methods and instances currently configured for all nodes.
ccs -h host --lsfenceinst [node]
Use the following procedure to configure a node with a primary fencing method that uses a fence device named apc, which uses the fence_apc fencing agent, and a backup fencing device that uses a fence device named sanswitch1, which uses the fence_sanbox2 fencing agent. Since the sanswitch1 device is a storage-based fencing agent, you will need to configure unfencing for that device as well.
Add a primary fence method for the node, providing a name for the fence method.
ccs -h host --addmethod method node
For example, to configure a fence method named APC as the primary method for the node node-01.example.com in the configuration file on the cluster node node-01.example.com, execute the following command:
ccs -h node01.example.com --addmethod APC node01.example.com
Add a fence instance for the primary method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node:
ccs -h host --addfenceinst fencedevicename node method [options]
For example, to configure a fence instance in the configuration file on the cluster node node-01.example.com that uses the APC switch power port 1 on the fence device named apc to fence cluster node node-01.example.com using the method named APC, execute the following command:
ccs -h node01.example.com --addfenceinst apc node01.example.com APC port=1
Add a backup fence method for the node, providing a name for the fence method.
ccs -h host --addmethod method node
For example, to configure a backup fence method named SAN for the node node-01.example.com in the configuration file on the cluster node node-01.example.com, execute the following command:
ccs -h node01.example.com --addmethod SAN node01.example.com
Add a fence instance for the backup method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node:
ccs -h host --addfenceinst fencedevicename node method [options]
For example, to configure a fence instance in the configuration file on the cluster node node-01.example.com that uses the SAN switch power port 11 on the fence device named sanswitch1 to fence cluster node node-01.example.com using the method named SAN, execute the following command:
ccs -h node01.example.com --addfenceinst sanswitch1 node01.example.com SAN port=11
Since the sanswitch1 device is a storage-based device, you must configure unfencing for this device.
ccs -h node01.example.com --addunfence sanswitch1 node01.example.com port=11 action=on
You can continue to add fencing methods as needed.
This procedure configures a fence device and a backup fence device for one node in the cluster. You will need to configure fencing for the other nodes in the cluster as well.
Example 5.4. cluster.conf After Adding Backup Fence Methods
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="APC">
<device name="apc" port="1"/>
</method>
<method name="SAN">
<device name="sanswitch1" port="11"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="11" action="on"/>
</unfence
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="APC">
<device name="apc" port="2"/>
</method>
<method name="SAN">
<device name="sanswitch1" port="12"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="12" action="on"/>
</unfence
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="APC">
<device name="apc" port="3"/>
</method>
<method name="SAN">
<device name="sanswitch1" port="13"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="13" action="on"/>
</unfence
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/>
<fencedevice agent="fence_sanbox2" ipaddr="san_ip_example" login="login_example" name="sanswitch1" passwd="password_example"/>
</fencedevices>
<rm>
</rm>
</cluster>
The order in which the system will use the fencing methods you have configured follows their order in the cluster configuration file. The first method you configure is the primary fencing method, and the second method you configure is the backup fencing method. To change the order, you can remove the primary fencing method from the configuration file, then add that method back.
5.7.4. Configuring a Node with Redundant Power
If your cluster is configured with redundant power supplies for your nodes, you must be sure to configure fencing so that your nodes fully shut down when they need to be fenced. If you configure each power supply as a separate fence method, each power supply will be fenced separately; the second power supply will allow the system to continue running when the first power supply is fenced and the system will not be fenced at all. To configure a system with dual power supplies, you must configure your fence devices so that both power supplies are shut off and the system is taken completely down. This requires that you configure two instances within a single fencing method, and that for each instance you configure both fence devices with an action attribute of off before configuring each of the devices with an action attribute of on.
To configure fencing for a node with dual power supplies, follow the steps in this section.
Before you can configure fencing for a node with redundant power, you must configure each of the power switches as a fence device for the cluster. For information on configuring fence devices, see
Section 5.5, “Configuring Fence Devices”.
To print a list of fence devices currently configured for your cluster, execute the following command:
ccs -h host --lsfencedev
Add a fence method for the node, providing a name for the fence method.
ccs -h host --addmethod method node
For example, to configure a fence method named APC-dual for the node node-01.example.com in the configuration file on the cluster node node-01.example.com, execute the following command:
ccs -h node01.example.com --addmethod APC-dual node01.example.com
Add a fence instance for the first power supply to the fence method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node. At this point you configure the action attribute as off.
ccs -h host --addfenceinst fencedevicename node method [options] action=off
For example, to configure a fence instance in the configuration file on the cluster node node-01.example.com that uses the APC switch power port 1 on the fence device named apc1 to fence cluster node node-01.example.com using the method named APC-dual, and setting the action attribute to off, execute the following command:
ccs -h node01.example.com --addfenceinst apc1 node01.example.com APC-dual port=1 action=off
Add a fence instance for the second power supply to the fence method. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node. At this point you configure the action attribute as off for this instance as well:
ccs -h host --addfenceinst fencedevicename node method [options] action=off
For example, to configure a second fence instance in the configuration file on the cluster node node-01.example.com that uses the APC switch power port 1 on the fence device named apc2 to fence cluster node node-01.example.com using the same method as you specified for the first instance named APC-dual, and setting the action attribute to off, execute the following command:
ccs -h node01.example.com --addfenceinst apc2 node01.example.com APC-dual port=1 action=off
At this point, add another fence instance for the first power supply to the fence method, configuring the action attribute as on. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node, and specifying the action attribute as on:
ccs -h host --addfenceinst fencedevicename node method [options] action=on
For example, to configure a fence instance in the configuration file on the cluster node node-01.example.com that uses the APC switch power port 1 on the fence device named apc1 to fence cluster node node-01.example.com using the method named APC-dual, and setting the action attribute to on, execute the following command:
ccs -h node01.example.com --addfenceinst apc1 node01.example.com APC-dual port=1 action=on
Add another fence instance for second power supply to the fence method, specifying the action attribute as on for this instance. You must specify the fence device to use for the node, the node this instance applies to, the name of the method, and any options for this method that are specific to this node as well as the action attribute of on.
ccs -h host --addfenceinst fencedevicename node method [options] action=on
For example, to configure a second fence instance in the configuration file on the cluster node node-01.example.com that uses the APC switch power port 1 on the fence device named apc2 to fence cluster node node-01.example.com using the same method as you specified for the first instance named APC-dual and setting the action attribute to on, execute the following command:
ccs -h node01.example.com --addfenceinst apc2 node01.example.com APC-dual port=1 action=on
Example 5.5. cluster.conf After Adding Dual-Power Fencing
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="APC-dual">
<device name="apc1" port="1"action="off"/>
<device name="apc2" port="1"action="off"/>
<device name="apc1" port="1"action="on"/>
<device name="apc2" port="1"action="on"/>
</method>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="APC-dual">
<device name="apc1" port="2"action="off"/>
<device name="apc2" port="2"action="off"/>
<device name="apc1" port="2"action="on"/>
<device name="apc2" port="2"action="on"/>
</method>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="APC-dual">
<device name="apc1" port="3"action="off"/>
<device name="apc2" port="3"action="off"/>
<device name="apc1" port="3"action="on"/>
<device name="apc2" port="3"action="on"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc1" passwd="password_example"/>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc2" passwd="password_example"/>
</fencedevices>
<rm>
</rm>
</cluster>
5.7.5. Removing Fence Methods and Fence Instances
To remove a fence method from your cluster configuration, execute the following command:
ccs -h host --rmmethod method node
For example, to remove a fence method that you have named APC that you have configured for node01.example.com from the cluster configuration file on cluster node node01.example.com, execute the following command:
ccs -h node01.example.com --rmmethod APC node01.example.com
To remove all fence instances of a fence device from a fence method, execute the following command:
ccs -h host --rmfenceinst fencedevicename node method
For example, to remove all instances of the fence device named apc1 from the method named APC-dual configured for node01.example.com from the cluster configuration file on cluster node node01.example.com, execute the following command:
ccs -h node01.example.com --rmfenceinst apc1 node01.example.com APC-dual
5.8. Configuring a Failover Domain
A failover domain is a named subset of cluster nodes that are eligible to run a cluster service in the event of a node failure. A failover domain can have the following characteristics:
Unrestricted — Allows you to specify that a subset of members are preferred, but that a cluster service assigned to this domain can run on any available member.
Restricted — Allows you to restrict the members that can run a particular cluster service. If none of the members in a restricted failover domain are available, the cluster service cannot be started (either manually or by the cluster software).
Unordered — When a cluster service is assigned to an unordered failover domain, the member on which the cluster service runs is chosen from the available failover domain members with no priority ordering.
Ordered — Allows you to specify a preference order among the members of a failover domain. The member at the top of the list is the most preferred, followed by the second member in the list, and so on.
Failback — Allows you to specify whether a service in the failover domain should fail back to the node that it was originally running on before that node failed. Configuring this characteristic is useful in circumstances where a node repeatedly fails and is part of an ordered failover domain. In that circumstance, if a node is the preferred node in a failover domain, it is possible for a service to fail over and fail back repeatedly between the preferred node and another node, causing severe impact on performance.
The failback characteristic is applicable only if ordered failover is configured.
Changing a failover domain configuration has no effect on currently running services.
Failover domains are not required for operation.
By default, failover domains are unrestricted and unordered.
In a cluster with several members, using a restricted failover domain can minimize the work to set up the cluster to run a cluster service (such as httpd), which requires you to set up the configuration identically on all members that run the cluster service. Instead of setting up the entire cluster to run the cluster service, you can set up only the members in the restricted failover domain that you associate with the cluster service.
To configure a preferred member, you can create an unrestricted failover domain comprising only one cluster member. Doing that causes a cluster service to run on that cluster member primarily (the preferred member), but allows the cluster service to fail over to any of the other members.
To configure a failover domain, perform the following procedure:
To add a failover domain, execute the following command:
ccs -h host --addfailoverdomain name [restricted] [ordered] [nofailback]
The name should be descriptive enough to distinguish its purpose relative to other names used in your cluster.
For example, the following command configures a failover domain named example_pri on node-01.example.com that is unrestricted, ordered, and allows failback:
ccs -h node-01.example.com --addfailoverdomain example_pri ordered
To add a node to a failover domain, execute the following command:
ccs -h host --addfailoverdomainnode failoverdomain node priority
For example, to configure the failover domain example_pri in the configuration file on node-01.example.com so that it contains node-01.example.com with a priority of 1, node-02.example.com with a priority of 2, and node-03.example.com with a priority of 3, execute the following commands:
ccs -h node-01.example.com --addfailoverdomainnode example_pri node-01.example.com 1
ccs -h node-01.example.com --addfailoverdomainnode example_pri node-02.example.com 2
ccs -h node-01.example.com --addfailoverdomainnode example_pri node-03.example.com 3
You can list all of the failover domains and failover domain nodes configured in a cluster with the following command:
ccs -h host --lsfailoverdomain
To remove a failover domain, execute the following command:
ccs -h host --rmfailoverdomain name
To remove a node from a failover domain, execute the following command:
ccs -h host --rmfailoverdomainnode failoverdomain node
5.9. Configuring Global Cluster Resources
You can configure two types of resources:
To see a list of currently configured resources and services in the cluster, execute the following command:
ccs -h host --lsservices
ccs -h host --addresource resourcetype [resource options]
For example, the following command adds a global file system resource to the cluster configuration file on node01.example.com. The name of the resource is web_fs, the file system device is /dev/sdd2, the file system mountpoint is /var/www, and the file system type is ext3.
ccs -h node01.example.com --addresource fs name=web_fs device=/dev/sdd2 mountpoint=/var/www fstype=ext3
To remove a global resource, execute the following command:
ccs -h host --rmresource resourcetype [resource options]
If you need to modify the parameters of an existing global resource, you can remove the resource and configure it again.
5.10. Adding a Cluster Service to the Cluster
To configure a cluster service in a cluster, perform the following steps:
Add a service to the cluster with the following command:
ccs -h host --addservice servicename [service options]
Use a descriptive name that clearly distinguishes the service from other services in the cluster.
When you add a service to the cluster configuration, you configure the following attributes:
autostart — Specifies whether to autostart the service when the cluster starts. Use "1" to enable and "0" to disable; the default is enabled.
domain — Specifies a failover domain (if required).
exclusive — Specifies a policy wherein the service only runs on nodes that have no other services running on them.
recovery — Specifies a recovery policy for the service. The options are to relocate, restart, disable, or restart-disable the service. For information on service recovery policies, refer to
Table B.18, “Service”.
For example, to add a service to the configuration file on the cluster node node-01.example.com named example_apache that uses the failover domain example_pri, and that has recovery policy of relocate, execute the following command:
ccs -h node-01.example.com --addservice example_apache domain=example_pri recovery=relocate
When configuring services for a cluster, you may find it useful to see a listing of available services for your cluster and the options available for each service. For information on using the
ccs command to print a list of available services and their options, refer to
Section 5.11, “Listing Available Cluster Services”.
Add resources to the service with the following command:
ccs -h host --addsubservice servicename subservice [service options]
Depending on the type of resources you want to use, populate the service with global or service-specific resources. To add a global resource, use the --addsubservice option of the ccs to add a resource. For example, to add the global file system resource named web_fs to the service named example_apache on the cluster configuration file on node-01.example.com, execute the following command:
ccs -h node01.example.com --addsubservice example_apache fs ref=web_fs
To add a service-specific resource to the service, you need to specify all of the service options. For example, if you had not previously defined web_fs as a global service, you could add it as a service-specific resource with the following command:
ccs -h node01.example.com --addsubservice example_apache fs name=web_fs device=/dev/sdd2 mountpoint=/var/www fstype=ext3
To add a child service to the service, you also use the --addsubservice option of the ccs command, specifying the service options.
If you need to add services within a tree structure of dependencies, use a colon (":") to separate elements and brackets to identify subservices of the same type. The following example adds a third nfsclient service as a subservice of an nfsclient service which is in itself a subservice of an nfsclient service which is a subservice of a service named service_a:
ccs -h node01.example.com --addsubservice service_a nfsclient[1]:nfsclient[2]:nfsclient
If you are adding a Samba-service resource, add it directly to the service, not as a child of another resource.
To verify the existence of the IP service resource used in a cluster service, you must use the /sbin/ip addr list command on a cluster node. The following output shows the /sbin/ip addr list command executed on a node running a cluster service:
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1356 qdisc pfifo_fast qlen 1000
link/ether 00:05:5d:9a:d8:91 brd ff:ff:ff:ff:ff:ff
inet 10.11.4.31/22 brd 10.11.7.255 scope global eth0
inet6 fe80::205:5dff:fe9a:d891/64 scope link
inet 10.11.4.240/22 scope global secondary eth0
valid_lft forever preferred_lft forever
To remove a service and all of its subservices, execute the following command:
ccs -h host --rmservice servicename
To remove a subservice, execute the following command:
ccs -h host --rmsubservice servicename subservice [service options]
5.11. Listing Available Cluster Services
You can use the ccs command to print a list of services that are available for a cluster. You can also use the ccs command to print a list of the options you can specify for a particular service type.
To print a list of cluster services currently available for your cluster, execute the following command:
ccs -h host --lsserviceopts
For example, the following command lists the cluster services available on the cluster node node1, showing sample output.
[root@ask-03 ~]# ccs -h node1 --lsserviceopts
service - Defines a service (resource group).
ASEHAagent - Sybase ASE Failover Instance
SAPDatabase - SAP database resource agent
SAPInstance - SAP instance resource agent
apache - Defines an Apache web server
clusterfs - Defines a cluster file system mount.
fs - Defines a file system mount.
ip - This is an IP address.
lvm - LVM Failover script
mysql - Defines a MySQL database server
named - Defines an instance of named server
netfs - Defines an NFS/CIFS file system mount.
nfsclient - Defines an NFS client.
nfsexport - This defines an NFS export.
nfsserver - This defines an NFS server resource.
openldap - Defines an Open LDAP server
oracledb - Oracle 10g Failover Instance
orainstance - Oracle 10g Failover Instance
oralistener - Oracle 10g Listener Instance
postgres-8 - Defines a PostgreSQL server
samba - Dynamic smbd/nmbd resource agent
script - LSB-compliant init script as a clustered resource.
tomcat-6 - Defines a Tomcat server
vm - Defines a Virtual Machine
action - Overrides resource action timings for a resource instance.
To print a list of the options you can specify for a particular service type, execute the following command:
ccs -h host --lsserviceopts service_type
For example, the following command lists the service options for the vm service.
[root@ask-03 ~]# ccs -f node1 --lsserviceopts vm
vm - Defines a Virtual Machine
Required Options:
name: Name
Optional Options:
domain: Cluster failover Domain
autostart: Automatic start after quorum formation
exclusive: Exclusive resource group
recovery: Failure recovery policy
migration_mapping: memberhost:targethost,memberhost:targethost ..
use_virsh: If set to 1, vm.sh will use the virsh command to manage virtual machines instead of xm. This is required when using non-Xen virtual machines (e.g. qemu / KVM).
xmlfile: Full path to libvirt XML file describing the domain.
migrate: Migration type (live or pause, default = live).
path: Path to virtual machine configuration files.
snapshot: Path to the snapshot directory where the virtual machine image will be stored.
depend: Top-level service this depends on, in service:name format.
depend_mode: Service dependency mode (soft or hard).
max_restarts: Maximum restarts for this service.
restart_expire_time: Restart expiration time; amount of time before a restart is forgotten.
status_program: Additional status check program
hypervisor: Hypervisor
hypervisor_uri: Hypervisor URI (normally automatic).
migration_uri: Migration URI (normally automatic).
__independent_subtree: Treat this and all children as an independent subtree.
__enforce_timeouts: Consider a timeout for operations as fatal.
__max_failures: Maximum number of failures before returning a failure to a status check.
__failure_expire_time: Amount of time before a failure is forgotten.
__max_restarts: Maximum number restarts for an independent subtree before giving up.
__restart_expire_time: Amount of time before a failure is forgotten for an independent subtree.
5.12. Virtual Machine Resources
Virtual machine resources are configured differently than other cluster resources. In particular, they are not grouped into service definitions. As of the Red Hat Enterprise Linux 6.2 release, when you configure a virtual machine in a cluster with the ccs command you can use the --addvm (rather than the addservice option). This ensures that the vm resource is defined directly under the rm configuration node in the cluster configuration file.
A virtual machine resource requires at least a name and a path attribute. The name attribute should match the name of the libvirt domain and the path attribute should specify the directory where the shared virtual machine definitions are stored.
The path attribute in the cluster configuration file is a path specification or a directory name, not a path to an individual file.
If virtual machine definitions are stored on a shared directory named /mnt/vm_defs, the following command will define a virtual machine named guest1:
# ccs -h node1.example.com --addvm guest1 path=/mnt/vm_defs
Running this command adds the following line to the rm configuration node in the cluster.conf file:
<vm name="guest1" path="/mnt/vm_defs"/>
5.13. Configuring a Quorum Disk
Quorum-disk parameters and heuristics depend on the site environment and the special requirements needed. To understand the use of quorum-disk parameters and heuristics, refer to the qdisk(5) man page. If you require assistance understanding and using quorum disk, contact an authorized Red Hat support representative.
Use the following command to configure your system for using a quorum disk:
ccs -h host --setquorumd [quorumd options]
Table 5.1, “Quorum Disk Options” summarizes the meaning of the quorum disk options you may need to set. For a complete list of quorum disk parameters, refer to the cluster schema at
/usr/share/cluster/cluster.rng, and the annotated schema at
/usr/share/doc/cman-X.Y.ZZ/cluster_conf.html.
Table 5.1. Quorum Disk Options
|
Parameter
|
Description
|
|---|
|
|
The frequency of read/write cycles, in seconds.
|
|
|
The number of votes the quorum daemon advertises to cman when it has a high enough score.
|
|
|
The number of cycles a node must miss to be declared dead.
|
|
|
The minimum score for a node to be considered "alive". If omitted or set to 0, the default function, floor((n+1)/2), is used, where n is the sum of the heuristics scores. The value must never exceed the sum of the heuristic scores; otherwise, the quorum disk cannot be available.
|
|
|
The storage device the quorum daemon uses. The device must be the same on all nodes.
|
|
|
Specifies the quorum disk label created by the mkqdisk utility. If this field contains an entry, the label overrides the field. If this field is used, the quorum daemon reads /proc/partitions and checks for qdisk signatures on every block device found, comparing the label against the specified label. This is useful in configurations where the quorum device name differs among nodes.
|
Use the following command to configure the heuristics for a quorum disk:
ccs -h host --addheuristic [heuristic options]
Table 5.2. Quorum Disk Heuristics
|
Parameter
|
Description
|
|---|
|
|
The path to the program used to determine if this heuristic is available. This can be anything that can be executed by /bin/sh -c. A return value of 0 indicates success; anything else indicates failure. This parameter is required to use a quorum disk.
|
|
|
The frequency (in seconds) at which the heuristic is polled. The default interval for every heuristic is 2 seconds.
|
|
|
The weight of this heuristic. Be careful when determining scores for heuristics. The default score for each heuristic is 1.
|
|
|
The number of consecutive failures required before this heuristic is declared unavailable.
|
To see a list of the quorum disk options and heuristics that are configured on a system, you can execute the following command:
ccs -h host --lsquorum
To remove a heuristic specified by a heuristic option, you can execute the following command:
ccs -h host rmheuristic [heuristic options]
5.14. Miscellaneous Cluster Configuration
This section describes using the ccs command to configure the following:
You can also use the ccs command to set advanced cluster configuration parameters, including totem options, dlm options, rm options and cman options. For information on setting these parameters see the ccs(8) man page and the annotated cluster configuration file schema at /usr/share/doc/cman-X.Y.ZZ/cluster_conf.html.
To view a list of the miscellaneous cluster attributes that have been configured for a cluster, execute the following command:
ccs -h host --lsmisc
5.14.1. Cluster Configuration Version
A cluster configuration file includes a cluster configuration version value. The configuration version value is set to 1 by default when you create a cluster configuration file and it is automatically incremented each time you modify your cluster configuration. However, if you need to set it to another value, you can specify it with the following command:
ccs -h host --setversion n
You can get the current configuration version value on an existing cluster configuration file with the following command:
ccs -h host --getversion
To increment the current configuration version value by 1 in the cluster configuration file on every node in the cluster, execute the following command:
ccs -h host --incversion
5.14.2. Multicast Configuration
If you do not specify a multicast address in the cluster configuration file, the Red Hat High Availability Add-On software creates one based on the cluster ID. It generates the lower 16 bits of the address and appends them to the upper portion of the address according to whether the IP protocol is IPv4 or IPv6:
For IPv4 — The address formed is 239.192. plus the lower 16 bits generated by Red Hat High Availability Add-On software.
For IPv6 — The address formed is FF15:: plus the lower 16 bits generated by Red Hat High Availability Add-On software.
The cluster ID is a unique identifier that cman generates for each cluster. To view the cluster ID, run the cman_tool status command on a cluster node.
You can manually specify a multicast address in the cluster configuration file with the following command:
ccs -h host --setmulticast multicastaddress
If you specify a multicast address, you should use the 239.192.x.x series (or FF15:: for IPv6) that cman uses. Otherwise, using a multicast address outside that range may cause unpredictable results. For example, using 224.0.0.x (which is "All hosts on the network") may not be routed correctly, or even routed at all by some hardware.
If you specify a multicast address, make sure that you check the configuration of routers that cluster packets pass through. Some routers may take a long time to learn addresses, seriously impacting cluster performance.
To remove a multicast address from a configuration file, use the --setmulticast option of the ccs but do not specify a multicast address:
ccs -h host --setmulticast
5.14.3. Configuring a Two-Node Cluster
If you are configuring a two-node cluster, you can execute the following command to allow a single node to maintain quorum (for example, if one node fails):
ccs -h host --setcman two_node=1 expected_votes=1
5.15. Propagating the Configuration File to the Cluster Nodes
After you have created or edited a cluster configuration file on one of the nodes in the cluster, you need to propagate that same file to all of the cluster nodes and activate the configuration.
Use the following command to propagate and activate a cluster configuration file:
ccs -h host --sync --activate
To verify that all of the nodes specified in the hosts cluster configuration file have the identical cluster configuration file, execute the following command:
ccs -h host --checkconf
If you have created or edited a configuration file on a local node, use the following command to send that file to one of the nodes in the cluster:
ccs -f file -h host --setconf
To verify that all of the nodes specified in the local file have the identical cluster configuration file, execute the following command:
ccs -f file --checkconf
Chapter 7. Configuring Red Hat High Availability Add-On With Command Line Tools
This chapter describes how to configure Red Hat High Availability Add-On software by directly editing the cluster configuration file (/etc/cluster/cluster.conf) and using command-line tools. The chapter provides procedures about building a configuration file one section at a time, starting with a sample file provided in the chapter. As an alternative to starting with a sample file provided here, you could copy a skeleton configuration file from the cluster.conf man page. However, doing so would not necessarily align with information provided in subsequent procedures in this chapter. There are other ways to create and configure a cluster configuration file; this chapter provides procedures about building a configuration file one section at a time. Also, keep in mind that this is just a starting point for developing a configuration file to suit your clustering needs.
This chapter consists of the following sections:
Make sure that your deployment of High Availability Add-On meets your needs and can be supported. Consult with an authorized Red Hat representative to verify your configuration prior to deployment. In addition, allow time for a configuration burn-in period to test failure modes.
This chapter references commonly used cluster.conf elements and attributes. For a comprehensive list and description of cluster.conf elements and attributes, refer to the cluster schema at /usr/share/cluster/cluster.rng, and the annotated schema at /usr/share/doc/cman-X.Y.ZZ/cluster_conf.html (for example /usr/share/doc/cman-3.0.12/cluster_conf.html).
Certain procedure in this chapter call for using the
cman_tool version -r command to propagate a cluster configuration throughout a cluster. Using that command requires that
ricci is running. Using
ricci requires a password the first time you interact with
ricci from any specific machine. For information on the
ricci service, refer to
Section 2.14, “Considerations for ricci”.
Procedures in this chapter may include specific commands for some of the command-line tools listed in
Appendix E, Command Line Tools Summary. For more information about all commands and variables, refer to the man page for each command-line tool.
Configuring Red Hat High Availability Add-On software with command-line tools consists of the following steps:
7.2. Creating a Basic Cluster Configuration File
Provided that cluster hardware, Red Hat Enterprise Linux, and High Availability Add-On software are installed, you can create a cluster configuration file (/etc/cluster/cluster.conf) and start running the High Availability Add-On. As a starting point only, this section describes how to create a skeleton cluster configuration file without fencing, failover domains, and HA services. Subsequent sections describe how to configure those parts of the configuration file.
This is just an interim step to create a cluster configuration file; the resultant file does not have any fencing and is not considered to be a supported configuration.
The following steps describe how to create and configure a skeleton cluster configuration file. Ultimately, the configuration file for your cluster will vary according to the number of nodes, the type of fencing, the type and number of HA services, and other site-specific requirements.
(Optional) If you are configuring a two-node cluster, you can add the following line to the configuration file to allow a single node to maintain quorum (for example, if one node fails):
<cman two_node="1" expected_votes="1"/>
In the clusternodes section, specify the node name and the node ID of each node using the clusternode attributes: name and nodeid.
Save /etc/cluster/cluster.conf.
Validate the file with against the cluster schema (cluster.rng) by running the ccs_config_validate command. For example:
[root@example-01 ~]# ccs_config_validate
Configuration validates
Propagate the configuration file to /etc/cluster/ in each cluster node. For example, you could propagate the file to other cluster nodes using the scp command.
Propagating the cluster configuration file this way is necessary the first time a cluster is created. Once a cluster is installed and running, the cluster configuration file can be propagated using the cman_tool version -r. It is possible to use the scp command to propagate an updated configuration file; however, the cluster software must be stopped on all nodes while using the scp command.In addition, you should run the ccs_config_validate if you propagate an updated configuration file via the scp.
While there are other elements and attributes present in the sample configuration file (for example, fence and fencedevices), there is no need to populate them now. Subsequent procedures in this chapter provide information about specifying other elements and attributes.
Start the cluster. At each cluster node run the following command:
service cman start
For example:
[root@example-01 ~]# service cman start
Starting cluster:
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Starting gfs_controld... [ OK ]
Unfencing self... [ OK ]
Joining fence domain... [ OK ]
At any cluster node, run cman_tool nodes to verify that the nodes are functioning as members in the cluster (signified as "M" in the status column, "Sts"). For example:
[root@example-01 ~]# cman_tool nodes
Node Sts Inc Joined Name
1 M 548 2010-09-28 10:52:21 node-01.example.com
2 M 548 2010-09-28 10:52:21 node-02.example.com
3 M 544 2010-09-28 10:52:21 node-03.example.com
Basic Configuration Examples
Example 7.1. cluster.conf Sample: Basic Configuration
<cluster name="mycluster" config_version="2">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
</fencedevices>
<rm>
</rm>
</cluster>
Example 7.2. cluster.conf Sample: Basic Two-Node Configuration
<cluster name="mycluster" config_version="2">
<cman two_node="1" expected_votes="1"/>
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
</fencedevices>
<rm>
</rm>
</cluster>
The consensus Value for totem in a Two-Node Cluster
When you create a two-node cluster and you do not intend to add additional nodes to the cluster at a later time, then you should omit the consensus value in the totem tag in the cluster.conf file so that the consensus value is calculated automatically. When the consensus value is calculated automatically, the following rules are used:
If there are two nodes or fewer, the consensus value will be (token * 0.2), with a ceiling of 2000 msec and a floor of 200 msec.
If there are three or more nodes, the consensus value will be (token + 2000 msec)
If you let the cman utility configure your consensus timeout in this fashion, then moving at a later time from two to three (or more) nodes will require a cluster restart, since the consensus timeout will need to change to the larger value based on the token timeout.
If you are configuring a two-node cluster and intend to upgrade in the future to more than two nodes, you can override the consensus timeout so that a cluster restart is not required when moving from two to three (or more) nodes. This can be done in the cluster.conf as follows:
<totem token="X" consensus="X + 2000" />
Note that the configuration parser does not calculate X + 2000 automatically. An integer value must be used rather than an equation.
The advantage of using the optimized consensus timeout for two-node clusters is that overall failover time is reduced for the two-node case, since consensus is not a function of the token timeout.
Note that for two-node autodetection in cman, the number of physical nodes is what matters and not the presence of the two_node=1 directive in the cluster.conf file.
Configuring fencing consists of (a) specifying one or more fence devices in a cluster and (b) specifying one or more fence methods for each node (using a fence device or fence devices specified).
Based on the type of fence devices and fence methods required for your configuration, configure cluster.conf as follows:
At the
clusternodes section, within the
fence element of each
clusternode section, specify each fence method of the node. Specify the fence method name, using the
method attribute,
name. Specify the fence device for each fence method, using the
device element and its attributes,
name and fence-device-specific parameters.
Example 7.4, “Fence Methods Added to cluster.conf ” shows an example of a fence method with one fence device for each node in the cluster.
For non-power fence methods (that is, SAN/storage fencing), at the clusternodes section, add an unfence section. This ensures that a fenced node is not re-enabled until the node has been rebooted. For more information about unfencing a node, refer to the fence_node(8) man page.
The unfence section does not contain method sections like the fence section does. It contains device references directly, which mirror the corresponding device sections for fence, with the notable addition of the explicit action (action) of "on" or "enable". The same fencedevice is referenced by both fence and unfence device lines, and the same per-node arguments should be repeated.
For more information about unfence refer to the fence_node man page.
Update the config_version attribute by incrementing its value (for example, changing from config_version="2" to config_version="3">).
Save /etc/cluster/cluster.conf.
(Optional) Validate the updated file against the cluster schema (cluster.rng) by running the ccs_config_validate command. For example:
[root@example-01 ~]# ccs_config_validate
Configuration validates
Run the cman_tool version -r command to propagate the configuration to the rest of the cluster nodes. This will also run additional validation. It is necessary that ricci be running in each cluster node to be able to propagate updated cluster configuration information.
Verify that the updated configuration file has been propagated.
If required, you can configure complex configurations with multiple fence methods per node and with multiple fence devices per fence method. When specifying multiple fence methods per node, if fencing fails using the first method, fenced, the fence daemon, tries the next method, and continues to cycle through methods until one succeeds.
Sometimes, fencing a node requires disabling two I/O paths or two power ports. This is done by specifying two or more devices within a fence method. fenced runs the fence agent once for each fence-device line; all must succeed for fencing to be considered successful.
You can find more information about configuring specific fence devices from a fence-device agent man page (for example, the man page for
fence_apc). In addition, you can get more information about fencing parameters from
Appendix A, Fence Device Parameters, the fence agents in
/usr/sbin/, the cluster schema at
/usr/share/cluster/cluster.rng, and the annotated schema at
/usr/share/doc/cman-X.Y.ZZ/cluster_conf.html (for example,
/usr/share/doc/cman-3.0.12/cluster_conf.html).
Fencing Configuration Examples
The following examples show a simple configuration with one fence method per node and one fence device per fence method:
The following examples show more complex configurations:
The examples in this section are not exhaustive; that is, there may be other ways to configure fencing depending on your requirements.
Example 7.3. APC Fence Device Added to cluster.conf
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/>
</fencedevices>
<rm>
</rm>
</cluster>
In this example, a fence device (fencedevice) has been added to the fencedevices element, specifying the fence agent (agent) as fence_apc, the IP address (ipaddr) as apc_ip_example, the login (login) as login_example, the name of the fence device (name) as apc, and the password (passwd) as password_example.
Example 7.4. Fence Methods Added to cluster.conf
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="APC">
<device name="apc" port="1"/>
</method>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="APC">
<device name="apc" port="2"/>
</method>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="APC">
<device name="apc" port="3"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/>
</fencedevices>
<rm>
</rm>
</cluster>
In this example, a fence method (method) has been added to each node. The name of the fence method (name) for each node is APC. The device (device) for the fence method in each node specifies the name (name) as apc and a unique APC switch power port number (port) for each node. For example, the port number for node-01.example.com is 1 (port="1"). The device name for each node (device name="apc") points to the fence device by the name (name) of apc in this line of the fencedevices element: fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example".
Example 7.5. cluster.conf: Multiple Fence Methods per Node
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="APC">
<device name="apc" port="1"/>
</method>
<method name="SAN">
<device name="sanswitch1" port="11"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="11" action="on"/>
</unfence
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="APC">
<device name="apc" port="2"/>
</method>
<method name="SAN">
<device name="sanswitch1" port="12"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="12" action="on"/>
</unfence
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="APC">
<device name="apc" port="3"/>
</method>
<method name="SAN">
<device name="sanswitch1" port="13"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="13" action="on"/>
</unfence
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/>
<fencedevice agent="fence_sanbox2" ipaddr="san_ip_example"
login="login_example" name="sanswitch1" passwd="password_example"/>
</fencedevices>
<rm>
</rm>
</cluster>
Example 7.6. cluster.conf: Fencing, Multipath Multiple Ports
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="SAN-multi">
<device name="sanswitch1" port="11"/>
<device name="sanswitch2" port="11"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="11" action="on"/>
<device name="sanswitch2" port="11" action="on"/>
</unfence
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="SAN-multi">
<device name="sanswitch1" port="12"/>
<device name="sanswitch2" port="12"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="12" action="on"/>
<device name="sanswitch2" port="12" action="on"/>
</unfence
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="SAN-multi">
<device name="sanswitch1" port="13"/>
<device name="sanswitch2" port="13"/>
</method>
</fence>
<unfence>
<device name="sanswitch1" port="13" action="on"/>
<device name="sanswitch2" port="13" action="on"/>
</unfence
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_sanbox2" ipaddr="san_ip_example"
login="login_example" name="sanswitch1" passwd="password_example"/>
<fencedevice agent="fence_sanbox2" ipaddr="san_ip_example"
login="login_example" name="sanswitch2" passwd="password_example"/>
</fencedevices>
<rm>
</rm>
</cluster>
Example 7.7. cluster.conf: Fencing Nodes with Dual Power Supplies
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="APC-dual">
<device name="apc1" port="1"action="off"/>
<device name="apc2" port="1"action="off"/>
<device name="apc1" port="1"action="on"/>
<device name="apc2" port="1"action="on"/>
</method>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="APC-dual">
<device name="apc1" port="2"action="off"/>
<device name="apc2" port="2"action="off"/>
<device name="apc1" port="2"action="on"/>
<device name="apc2" port="2"action="on"/>
</method>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="APC-dual">
<device name="apc1" port="3"action="off"/>
<device name="apc2" port="3"action="off"/>
<device name="apc1" port="3"action="on"/>
<device name="apc2" port="3"action="on"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc1" passwd="password_example"/>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc2" passwd="password_example"/>
</fencedevices>
<rm>
</rm>
</cluster>
When using power switches to fence nodes with dual power supplies, the agents must be told to turn off both power ports before restoring power to either port. The default off-on behavior of the agent could result in the power never being fully disabled to the node.
7.4. Configuring Failover Domains
A failover domain is a named subset of cluster nodes that are eligible to run a cluster service in the event of a node failure. A failover domain can have the following characteristics:
Unrestricted — Allows you to specify that a subset of members are preferred, but that a cluster service assigned to this domain can run on any available member.
Restricted — Allows you to restrict the members that can run a particular cluster service. If none of the members in a restricted failover domain are available, the cluster service cannot be started (either manually or by the cluster software).
Unordered — When a cluster service is assigned to an unordered failover domain, the member on which the cluster service runs is chosen from the available failover domain members with no priority ordering.
Ordered — Allows you to specify a preference order among the members of a failover domain. Ordered failover domains select the node with the lowest priority number first. That is, the node in a failover domain with a priority number of "1" specifies the highest priority, and therefore is the most preferred node in a failover domain. After that node, the next preferred node would be the node with the next highest priority number, and so on.
Failback — Allows you to specify whether a service in the failover domain should fail back to the node that it was originally running on before that node failed. Configuring this characteristic is useful in circumstances where a node repeatedly fails and is part of an ordered failover domain. In that circumstance, if a node is the preferred node in a failover domain, it is possible for a service to fail over and fail back repeatedly between the preferred node and another node, causing severe impact on performance.
The failback characteristic is applicable only if ordered failover is configured.
Changing a failover domain configuration has no effect on currently running services.
Failover domains are not required for operation.
By default, failover domains are unrestricted and unordered.
In a cluster with several members, using a restricted failover domain can minimize the work to set up the cluster to run a cluster service (such as httpd), which requires you to set up the configuration identically on all members that run the cluster service. Instead of setting up the entire cluster to run the cluster service, you can set up only the members in the restricted failover domain that you associate with the cluster service.
To configure a preferred member, you can create an unrestricted failover domain comprising only one cluster member. Doing that causes a cluster service to run on that cluster member primarily (the preferred member), but allows the cluster service to fail over to any of the other members.
To configure a failover domain, use the following procedures:
Open /etc/cluster/cluster.conf at any node in the cluster.
Add the following skeleton section within the rm element for each failover domain to be used:
<failoverdomains>
<failoverdomain name="" nofailback="" ordered="" restricted="">
<failoverdomainnode name="" priority=""/>
<failoverdomainnode name="" priority=""/>
<failoverdomainnode name="" priority=""/>
</failoverdomain>
</failoverdomains>
The number of failoverdomainnode attributes depends on the number of nodes in the failover domain. The skeleton failoverdomain section in preceding text shows three failoverdomainnode elements (with no node names specified), signifying that there are three nodes in the failover domain.
In the
failoverdomain section, provide the values for the elements and attributes. For descriptions of the elements and attributes, refer to the
failoverdomain section of the annotated cluster schema. The annotated cluster schema is available at
/usr/share/doc/cman-X.Y.ZZ/cluster_conf.html (for example
/usr/share/doc/cman-3.0.12/cluster_conf.html) in any of the cluster nodes. For an example of a
failoverdomains section, refer to
Example 7.8, “A Failover Domain Added to cluster.conf ”.
Update the config_version attribute by incrementing its value (for example, changing from config_version="2" to config_version="3">).
Save /etc/cluster/cluster.conf.
(Optional) Validate the file with against the cluster schema (cluster.rng) by running the ccs_config_validate command. For example:
[root@example-01 ~]# ccs_config_validate
Configuration validates
Run the cman_tool version -r command to propagate the configuration to the rest of the cluster nodes.
Example 7.8. A Failover Domain Added to cluster.conf
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="APC">
<device name="apc" port="1"/>
</method>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="APC">
<device name="apc" port="2"/>
</method>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="APC">
<device name="apc" port="3"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="example_pri" nofailback="0" ordered="1" restricted="0">
<failoverdomainnode name="node-01.example.com" priority="1"/>
<failoverdomainnode name="node-02.example.com" priority="2"/>
<failoverdomainnode name="node-03.example.com" priority="3"/>
</failoverdomain>
</failoverdomains>
</rm>
</cluster>
The failoverdomains section contains a failoverdomain section for each failover domain in the cluster. This example has one failover domain. In the failoverdomain line, the name (name) is specified as example_pri. In addition, it specifies no failback (failback="0"), that failover is ordered (ordered="1"), and that the failover domain is unrestricted (restricted="0").
7.5. Configuring HA Services
Configuring HA (High Availability) services consists of configuring resources and assigning them to services.
The following sections describe how to edit /etc/cluster/cluster.conf to add resources and services.
There can be a wide range of configurations possible with High Availability resources and services. For a better understanding about resource parameters and resource behavior, refer to
Appendix B, HA Resource Parameters and
Appendix C, HA Resource Behavior. For optimal performance and to ensure that your configuration can be supported, contact an authorized Red Hat support representative.
7.5.1. Adding Cluster Resources
You can configure two types of resources:
Global — Resources that are available to any service in the cluster. These are configured in the resources section of the configuration file (within the rm element).
Service-specific — Resources that are available to only one service. These are configured in each service section of the configuration file (within the rm element).
To add a global cluster resource, follow the steps in this section.
Open /etc/cluster/cluster.conf at any node in the cluster.
Add a resources section within the rm element. For example:
<rm>
<resources>
</resources>
</rm>
Populate it with resources according to the services you want to create. For example, here are resources that are to be used in an Apache service. They consist of a file system (fs) resource, an IP (ip) resource, and an Apache (apache) resource.
<rm>
<resources>
<fs name="web_fs" device="/dev/sdd2" mountpoint="/var/www" fstype="ext3"/>
<ip address="127.143.131.100" monitor_link="yes" sleeptime="10"/>
<apache config_file="conf/httpd.conf" name="example_server" server_root="/etc/httpd" shutdown_wait="0"/>
</resources>
</rm>
Update the config_version attribute by incrementing its value (for example, changing from config_version="2" to config_version="3").
Save /etc/cluster/cluster.conf.
(Optional) Validate the file with against the cluster schema (cluster.rng) by running the ccs_config_validate command. For example:
[root@example-01 ~]# ccs_config_validate
Configuration validates
Run the cman_tool version -r command to propagate the configuration to the rest of the cluster nodes.
Verify that the updated configuration file has been propagated.
Example 7.9. cluster.conf File with Resources Added
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="APC">
<device name="apc" port="1"/>
</method>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="APC">
<device name="apc" port="2"/>
</method>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="APC">
<device name="apc" port="3"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="example_pri" nofailback="0" ordered="1" restricted="0">
<failoverdomainnode name="node-01.example.com" priority="1"/>
<failoverdomainnode name="node-02.example.com" priority="2"/>
<failoverdomainnode name="node-03.example.com" priority="3"/>
</failoverdomain>
</failoverdomains>
<resources>
<fs name="web_fs" device="/dev/sdd2" mountpoint="/var/www" fstype="ext3"/>
<ip address="127.143.131.100" monitor_link="yes" sleeptime="10"/>
<apache config_file="conf/httpd.conf" name="example_server" server_root="/etc/httpd" shutdown_wait="0"/>
</resources>
</rm>
</cluster>
7.5.2. Adding a Cluster Service to the Cluster
To add a cluster service to the cluster, follow the steps in this section.
Open /etc/cluster/cluster.conf at any node in the cluster.
Add a service section within the rm element for each service. For example:
<rm>
<service autostart="1" domain="" exclusive="0" name="" recovery="restart">
</service>
</rm>
Configure the following parameters (attributes) in the service element:
autostart — Specifies whether to autostart the service when the cluster starts. Use '1' to enable and '0' to disable; the default is enabled.
domain — Specifies a failover domain (if required).
exclusive — Specifies a policy wherein the service only runs on nodes that have no other services running on them.
recovery — Specifies a recovery policy for the service. The options are to relocate, restart, disable, or restart-disable the service.
Depending on the type of resources you want to use, populate the service with global or service-specific resources
For example, here is an Apache service that uses global resources:
<rm>
<resources>
<fs name="web_fs" device="/dev/sdd2" mountpoint="/var/www" fstype="ext3"/>
<ip address="127.143.131.100" monitor_link="yes" sleeptime="10"/>
<apache config_file="conf/httpd.conf" name="example_server" server_root="/etc/httpd" shutdown_wait="0"/>
</resources>
<service autostart="1" domain="example_pri" exclusive="0" name="example_apache" recovery="relocate">
<fs ref="web_fs"/>
<ip ref="127.143.131.100"/>
<apache ref="example_server"/>
</service>
</rm>
For example, here is an Apache service that uses service-specific resources:
<rm>
<service autostart="0" domain="example_pri" exclusive="0" name="example_apache2" recovery="relocate">
<fs name="web_fs2" device="/dev/sdd3" mountpoint="/var/www2" fstype="ext3"/>
<ip address="127.143.131.101" monitor_link="yes" sleeptime="10"/>
<apache config_file="conf/httpd.conf" name="example_server2" server_root="/etc/httpd" shutdown_wait="0"/>
</service>
</rm>
example_apache — This service uses global resources web_fs, 127.143.131.100, and example_server.
example_apache2 — This service uses service-specific resources web_fs2, 127.143.131.101, and example_server2.
Update the config_version attribute by incrementing its value (for example, changing from config_version="2" to config_version="3">).
Save /etc/cluster/cluster.conf.
(Optional) Validate the updated file against the cluster schema (cluster.rng) by running the ccs_config_validate command. For example:
[root@example-01 ~]# ccs_config_validate
Configuration validates
Run the cman_tool version -r command to propagate the configuration to the rest of the cluster nodes.
Verify that the updated configuration file has been propagated.
Example 7.10. cluster.conf with Services Added: One Using Global Resources and One Using Service-Specific Resources
<cluster name="mycluster" config_version="3">
<clusternodes>
<clusternode name="node-01.example.com" nodeid="1">
<fence>
<method name="APC">
<device name="apc" port="1"/>
</method>
</fence>
</clusternode>
<clusternode name="node-02.example.com" nodeid="2">
<fence>
<method name="APC">
<device name="apc" port="2"/>
</method>
</fence>
</clusternode>
<clusternode name="node-03.example.com" nodeid="3">
<fence>
<method name="APC">
<device name="apc" port="3"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="example_pri" nofailback="0" ordered="1" restricted="0">
<failoverdomainnode name="node-01.example.com" priority="1"/>
<failoverdomainnode name="node-02.example.com" priority="2"/>
<failoverdomainnode name="node-03.example.com" priority="3"/>
</failoverdomain>
</failoverdomains>
<resources>
<fs name="web_fs" device="/dev/sdd2" mountpoint="/var/www" fstype="ext3"/>
<ip address="127.143.131.100" monitor_link="yes" sleeptime="10"/>
<apache config_file="conf/httpd.conf" name="example_server" server_root="/etc/httpd" shutdown_wait="0"/>
</resources>
<service autostart="1" domain="example_pri" exclusive="0" name="example_apache" recovery="relocate">
<fs ref="web_fs"/>
<ip ref="127.143.131.100"/>
<apache ref="example_server"/>
</service>
<service autostart="0" domain="example_pri" exclusive="0" name="example_apache2" recovery="relocate">
<fs name="web_fs2" device="/dev/sdd3" mountpoint="/var/www2" fstype="ext3"/>
<ip address="127.143.131.101" monitor_link="yes" sleeptime="10"/>
<apache config_file="conf/httpd.conf" name="example_server2" server_root="/etc/httpd" shutdown_wait="0"/>
</service>
</rm>
</cluster>
7.6. Verifying a Configuration
Once you have created your cluster configuration file, verify that it is running correctly by performing the following steps:
At each node, restart the cluster software. That action ensures that any configuration additions that are checked only at startup time are included in the running configuration. You can restart the cluster software by running service cman restart. For example:
[root@example-01 ~]# service cman restart
Stopping cluster:
Leaving fence domain... [ OK ]
Stopping gfs_controld... [ OK ]
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Waiting for corosync to shutdown: [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
Starting cluster:
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Starting gfs_controld... [ OK ]
Unfencing self... [ OK ]
Joining fence domain... [ OK ]
Run service clvmd start, if CLVM is being used to create clustered volumes. For example:
[root@example-01 ~]# service clvmd start
Activating VGs: [ OK ]
Run service gfs2 start, if you are using Red Hat GFS2. For example:
[root@example-01 ~]# service gfs2 start
Mounting GFS2 filesystem (/mnt/gfsA): [ OK ]
Mounting GFS2 filesystem (/mnt/gfsB): [ OK ]
Run service rgmanager start, if you using high-availability (HA) services. For example:
[root@example-01 ~]# service rgmanager start
Starting Cluster Service Manager: [ OK ]
At any cluster node, run cman_tool nodes to verify that the nodes are functioning as members in the cluster (signified as "M" in the status column, "Sts"). For example:
[root@example-01 ~]# cman_tool nodes
Node Sts Inc Joined Name
1 M 548 2010-09-28 10:52:21 node-01.example.com
2 M 548 2010-09-28 10:52:21 node-02.example.com
3 M 544 2010-09-28 10:52:21 node-03.example.com
At any node, using the clustat utility, verify that the HA services are running as expected. In addition, clustat displays status of the cluster nodes. For example:
[root@example-01 ~]#clustat
Cluster Status for mycluster @ Wed Nov 17 05:40:00 2010
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node-03.example.com 3 Online, rgmanager
node-02.example.com 2 Online, rgmanager
node-01.example.com 1 Online, Local, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:example_apache node-01.example.com started
service:example_apache2 (none) disabled
