8.3. Configuring Fencing
Configuring fencing consists of (a) specifying one or more fence devices in a cluster and (b) specifying one or more fence methods for each node (using a fence device or fence devices specified).
Note
It is recommended that you configure multiple fencing mechanisms for each node. A fencing device can fail due to network split, a power outage, or a problem in the fencing device itself. Configuring multiple fencing mechanisms can reduce the likelihood that the failure of a fencing device will have fatal results.
Based on the type of fence devices and fence methods required for your configuration, configure
cluster.conf
as follows:
- In the
fencedevices
section, specify each fence device, using afencedevice
element and fence-device dependent attributes. Example 8.3, “APC Fence Device Added tocluster.conf
” shows an example of a configuration file with an APC fence device added to it. - At the
clusternodes
section, within thefence
element of eachclusternode
section, specify each fence method of the node. Specify the fence method name, using themethod
attribute,name
. Specify the fence device for each fence method, using thedevice
element and its attributes,name
and fence-device-specific parameters. Example 8.4, “Fence Methods Added tocluster.conf
” shows an example of a fence method with one fence device for each node in the cluster. - For non-power fence methods (that is, SAN/storage fencing), at the
clusternodes
section, add anunfence
section. This ensures that a fenced node is not re-enabled until the node has been rebooted. When you configure a device that requires unfencing, the cluster must first be stopped and the full configuration including devices and unfencing must be added before the cluster is started. For more information about unfencing a node, see thefence_node
(8) man page.Theunfence
section does not containmethod
sections like thefence
section does. It containsdevice
references directly, which mirror the corresponding device sections forfence
, with the notable addition of the explicit action (action
) of "on" or "enable". The samefencedevice
is referenced by bothfence
andunfence
device
lines, and the same per-node arguments should be repeated.Specifying theaction
attribute as "on" or "enable" enables the node when rebooted. Example 8.4, “Fence Methods Added tocluster.conf
” and Example 8.5, “cluster.conf
: Multiple Fence Methods per Node” include examples of theunfence
elements and attributed.For more information aboutunfence
see thefence_node
man page. - Update the
config_version
attribute by incrementing its value (for example, changing fromconfig_version="2"
toconfig_version="3">
). - Save
/etc/cluster/cluster.conf
. - (Optional) Validate the updated file against the cluster schema (
cluster.rng
) by running theccs_config_validate
command. For example:[root@example-01 ~]#
ccs_config_validate
Configuration validates - Run the
cman_tool version -r
command to propagate the configuration to the rest of the cluster nodes. This will also run additional validation. It is necessary thatricci
be running in each cluster node to be able to propagate updated cluster configuration information. - Verify that the updated configuration file has been propagated.
- Proceed to Section 8.4, “Configuring Failover Domains”.
If required, you can configure complex configurations with multiple fence methods per node and with multiple fence devices per fence method. When specifying multiple fence methods per node, if fencing fails using the first method,
fenced
, the fence daemon, tries the next method, and continues to cycle through methods until one succeeds.
Sometimes, fencing a node requires disabling two I/O paths or two power ports. This is done by specifying two or more devices within a fence method.
fenced
runs the fence agent once for each fence-device line; all must succeed for fencing to be considered successful.
More complex configurations are shown in the section called “Fencing Configuration Examples”.
You can find more information about configuring specific fence devices from a fence-device agent man page (for example, the man page for
fence_apc
). In addition, you can get more information about fencing parameters from Appendix A, Fence Device Parameters, the fence agents in /usr/sbin/
, the cluster schema at /usr/share/cluster/cluster.rng
, and the annotated schema at /usr/share/doc/cman-X.Y.ZZ/cluster_conf.html
(for example, /usr/share/doc/cman-3.0.12/cluster_conf.html
).
Note
As of Red Hat Enterprise Linux Release 6.4, you can test the fence configuration for each node in a cluster with the
fence_check
utility. For information on this utility, see the fence_check
(8) man page.
Fencing Configuration Examples
The following examples show a simple configuration with one fence method per node and one fence device per fence method:
The following examples show more complex configurations:
Note
The examples in this section are not exhaustive; that is, there may be other ways to configure fencing depending on your requirements.
Example 8.3. APC Fence Device Added to cluster.conf
<cluster name="mycluster" config_version="3"> <clusternodes> <clusternode name="node-01.example.com" nodeid="1"> <fence> </fence> </clusternode> <clusternode name="node-02.example.com" nodeid="2"> <fence> </fence> </clusternode> <clusternode name="node-03.example.com" nodeid="3"> <fence> </fence> </clusternode> </clusternodes> <fencedevices> <fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/> </fencedevices> <rm> </rm> </cluster>
In this example, a fence device (
fencedevice
) has been added to the fencedevices
element, specifying the fence agent (agent
) as fence_apc
, the IP address (ipaddr
) as apc_ip_example
, the login (login
) as login_example
, the name of the fence device (name
) as apc
, and the password (passwd
) as password_example
.
Example 8.4. Fence Methods Added to cluster.conf
<cluster name="mycluster" config_version="3"> <clusternodes> <clusternode name="node-01.example.com" nodeid="1"> <fence> <method name="APC"> <device name="apc" port="1"/> </method> </fence> </clusternode> <clusternode name="node-02.example.com" nodeid="2"> <fence> <method name="APC"> <device name="apc" port="2"/> </method> </fence> </clusternode> <clusternode name="node-03.example.com" nodeid="3"> <fence> <method name="APC"> <device name="apc" port="3"/> </method> </fence> </clusternode> </clusternodes> <fencedevices> <fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/> </fencedevices> <rm> </rm> </cluster>
In this example, a fence method (
method
) has been added to each node. The name of the fence method (name
) for each node is APC
. The device (device
) for the fence method in each node specifies the name (name
) as apc
and a unique APC switch power port number (port
) for each node. For example, the port number for node-01.example.com is 1
(port="1"
). The device name for each node (device name="apc"
) points to the fence device by the name (name
) of apc
in this line of the fencedevices
element: fencedevice agent="fence_apc"
ipaddr="apc_ip_example" login="login_example"
name="apc" passwd="password_example"
.
Example 8.5. cluster.conf
: Multiple Fence Methods per Node
<cluster name="mycluster" config_version="3"> <clusternodes> <clusternode name="node-01.example.com" nodeid="1"> <fence> <method name="APC"> <device name="apc" port="1"/> </method> <method name="SAN"> <device name="sanswitch1" port="11"/> </method> </fence> <unfence> <device name="sanswitch1" port="11" action="on"/> </unfence> </clusternode> <clusternode name="node-02.example.com" nodeid="2"> <fence> <method name="APC"> <device name="apc" port="2"/> </method> <method name="SAN"> <device name="sanswitch1" port="12"/> </method> </fence> <unfence> <device name="sanswitch1" port="12" action="on"/> </unfence> </clusternode> <clusternode name="node-03.example.com" nodeid="3"> <fence> <method name="APC"> <device name="apc" port="3"/> </method> <method name="SAN"> <device name="sanswitch1" port="13"/> </method> </fence> <unfence> <device name="sanswitch1" port="13" action="on"/> </unfence> </clusternode> </clusternodes> <fencedevices> <fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc" passwd="password_example"/> <fencedevice agent="fence_sanbox2" ipaddr="san_ip_example" login="login_example" name="sanswitch1" passwd="password_example"/> </fencedevices> <rm> </rm> </cluster>
Example 8.6. cluster.conf
: Fencing, Multipath Multiple Ports
<cluster name="mycluster" config_version="3"> <clusternodes> <clusternode name="node-01.example.com" nodeid="1"> <fence> <method name="SAN-multi"> <device name="sanswitch1" port="11"/> <device name="sanswitch2" port="11"/> </method> </fence> <unfence> <device name="sanswitch1" port="11" action="on"/> <device name="sanswitch2" port="11" action="on"/> </unfence> </clusternode> <clusternode name="node-02.example.com" nodeid="2"> <fence> <method name="SAN-multi"> <device name="sanswitch1" port="12"/> <device name="sanswitch2" port="12"/> </method> </fence> <unfence> <device name="sanswitch1" port="12" action="on"/> <device name="sanswitch2" port="12" action="on"/> </unfence> </clusternode> <clusternode name="node-03.example.com" nodeid="3"> <fence> <method name="SAN-multi"> <device name="sanswitch1" port="13"/> <device name="sanswitch2" port="13"/> </method> </fence> <unfence> <device name="sanswitch1" port="13" action="on"/> <device name="sanswitch2" port="13" action="on"/> </unfence> </clusternode> </clusternodes> <fencedevices> <fencedevice agent="fence_sanbox2" ipaddr="san_ip_example" login="login_example" name="sanswitch1" passwd="password_example"/> <fencedevice agent="fence_sanbox2" ipaddr="san_ip_example" login="login_example" name="sanswitch2" passwd="password_example"/> </fencedevices> <rm> </rm> </cluster>
Example 8.7. cluster.conf
: Fencing Nodes with Dual Power Supplies
<cluster name="mycluster" config_version="3"> <clusternodes> <clusternode name="node-01.example.com" nodeid="1"> <fence> <method name="APC-dual"> <device name="apc1" port="1"action="off"/> <device name="apc2" port="1"action="off"/> <device name="apc1" port="1"action="on"/> <device name="apc2" port="1"action="on"/> </method> </fence> </clusternode> <clusternode name="node-02.example.com" nodeid="2"> <fence> <method name="APC-dual"> <device name="apc1" port="2"action="off"/> <device name="apc2" port="2"action="off"/> <device name="apc1" port="2"action="on"/> <device name="apc2" port="2"action="on"/> </method> </fence> </clusternode> <clusternode name="node-03.example.com" nodeid="3"> <fence> <method name="APC-dual"> <device name="apc1" port="3"action="off"/> <device name="apc2" port="3"action="off"/> <device name="apc1" port="3"action="on"/> <device name="apc2" port="3"action="on"/> </method> </fence> </clusternode> </clusternodes> <fencedevices> <fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc1" passwd="password_example"/> <fencedevice agent="fence_apc" ipaddr="apc_ip_example" login="login_example" name="apc2" passwd="password_example"/> </fencedevices> <rm> </rm> </cluster>
When using power switches to fence nodes with dual power supplies, the agents must be told to turn off both power ports before restoring power to either port. The default off-on behavior of the agent could result in the power never being fully disabled to the node.