3.5. Fencing Configuration

Fencing is a technique used to remove a cluster member from an active cluster, as determined by loss of communication with the cluster. There are two fail-safe mechanisms in a typical Oracle HA configuration: the quorum voting disk service, qdisk, and the cman heartbeat mechanism that operates over the private, bonded network. If either node fails to “check-in” within a prescribed time, actions are taken to remove, or fence the node from the rest of the active cluster. Fencing is the most important job that a cluster product must do. Inconsistent, or unreliable fencing can result in corruption of the Oracle database -- it must be bulletproof.

Red Hat Cluster Suite provides more fencing technologies than either Veritas Foundation Suite, or Oracle Clusterware. The fencing methods that we recommend for use with Oracle databases, are all power-based, and have been in the Red Hat Cluster Suite for several releases. Mature, power-based fencing methods are, indeed, the foundation of any robust cluster.

Most Tier 1 server vendors provide built-in baseboard management controllers (BMC), but they are called many things (HP iLO, Dell DRAC, Sun ILOM). All BMCs provide network-signaled access to the server’s power supply. When Red Hat Cluster Suite must fence a node in the cluster, the fencing process on the node that detected the fault will connect to the other nodes BMC and literally power-off the server node. This is the most discrete form of fencing, and it the mechanism we use. In this case, we use HP iLO, which comes standard on all Proliant 300 and 500 series.

Red Hat Cluster Suite also supports levels of fencing for those who find BMC-based fencing insufficient. Among many other available methods (such as FCP switch port disable), Red Hat Cluster Suite also supports signaled power distribution units (PDUs). PDUs are also connected to an Ethernet network, and when engaged for fencing, they cut the power to the server’s power supply, much as the BMC does on-board. The need to use multi-levels can be necessary because most, if not all, BMC interfaces are single Ethernet ports. This could be a single point of failure. Most PDUs also only have 1 network interface, but combined, these two methods provide redundant power signaling.

Our example will show iLO, and how it can be combined with an APC switched PDU infrastructure.

Red Hat Cluster Suite is typically configured to access the fencing network over the private bonded fabric, but any network fabric can be subsequently configured if a dedicated (and likely bonded) network is dedicated just to the fencing network. Our example will access the fencing network over the private, bonded network.