4.3. GFS2 File System Hangs and Requires Reboot of All Nodes
If your GFS2 file system hangs and does not return commands run against it, requiring that you reboot all nodes in the cluster before using it, check for the following issues.
- You may have had a failed fence. GFS2 file systems will freeze to ensure data integrity in the event of a failed fence. Check the messages logs to see if there are any failed fences at the time of the hang. Ensure that fencing is configured correctly.
- The GFS2 file system may have withdrawn. Check through the messages logs for the word
withdrawand check for any messages and call traces from GFS2 indicating that the file system has been withdrawn. A withdraw is indicative of file system corruption, a storage failure, or a bug. At the earliest time when it is convenient to unmount the file system, you should perform the following procedure:- Reboot the node on which the withdraw occurred.
#
/sbin/reboot - Stop the file system resource to unmount the GFS2 file system on all nodes.
#
pcs resource disable --wait=100 mydata_fs - Capture the metadata with the
gfs2_edit savemeta...command. You should ensure that there is sufficient space for the file, which in some cases may be large. In this example, the metadata is saved to a file in the/rootdirectory.#
gfs2_edit savemeta /dev/vg_mydata/mydata /root/gfs2metadata.gz - Update the
gfs2-utilspackage.#
sudo yum update gfs2-utils - On one node, run the
fsck.gfs2command on the file system to ensure file system integrity and repair any damage.#
fsck.gfs2 -y /dev/vg_mydata/mydata > /tmp/fsck.out - After the
fsck.gfs2command has completed, re-enable the file system resource to return it to service:#
pcs resource enable --wait=100 mydata_fs - Open a support ticket with Red Hat Support. Inform them you experienced a GFS2 withdraw and provide logs and the debugging information generated by the
sosreportsandgfs2_edit savemetacommands.
In some instances of a GFS2 withdraw, commands can hang that are trying to access the file system or its block device. In these cases a hard reboot is required to reboot the cluster.For information on the GFS2 withdraw function, see Section 3.11, “The GFS2 Withdraw Function”. - This error may be indicative of a locking problem or bug. Gather data during one of these occurrences and open a support ticket with Red Hat Support, as described in Section 4.2, “GFS2 File System Hangs and Requires Reboot of One Node”.
