Chapter 5. Upgrading to Red Hat Gluster Storage 3.5

Procedure 5.1. Before you upgrade

1. Back up the following configuration directory and files in a location that is not on the operating system partition.
   • /var/lib/glusterd
   • /etc/samba
   • /etc/ctdb
   • /etc/glusterfs
   • /var/lib/samba
   • /var/lib/ctdb
   • /var/run/gluster/shared_storage/nfs-ganesha

   Note

   With the release of 3.5 Batch Update 3, the mount point of shared storage is changed from /var/run/gluster/ to /run/gluster/ .
   If you use NFS-Ganesha, back up the following files from all nodes:
   • /run/gluster/shared_storage/nfs-ganesha/exports/export.*.conf
   • /etc/ganesha/ganesha.conf
   • /etc/ganesha/ganesha-ha.conf
   If upgrading from Red Hat Gluster Storage 3.3 to 3.4 or subsequent releases, back up all xattr by executing the following command individually on the brick root(s) for all nodes:

   # find ./ -type d ! -path "./.*" ! -path "./" | xargs getfattr -d -m. -e hex > /var/log/glusterfs/xattr_dump_brick_name

   Copy to Clipboard
2. Unmount gluster volumes from all clients. On a client, use the following command to unmount a volume from a mount point.

   # umount mount-point

   Copy to Clipboard
3. If you use NFS-Ganesha, run the following on a gluster server to disable the nfs-ganesha service:

   # gluster nfs-ganesha disable

   Copy to Clipboard
4. On a gluster server, disable the shared volume.

   # gluster volume set all cluster.enable-shared-storage disable

   Copy to Clipboard
5. Stop all volumes

   # for vol in `gluster volume list`; do gluster --mode=script volume stop $vol; sleep 2s; done

   Copy to Clipboard
6. Verify that all volumes are stopped.

   # gluster volume info

   Copy to Clipboard
7. Stop the glusterd services on all servers using the following command:

   # service glusterd stop
   # pkill glusterfs
   # pkill glusterfsd

   Copy to Clipboard
8. Stop the pcsd service.

   # systemctl stop pcsd

   Copy to Clipboard

Procedure 5.2. Upgrade using yum

migrate-rhs-classic-to-rhsm --status

# migrate-rhs-classic-to-rhsm --status

migrate-rhs-classic-to-rhsm --rhn-to-rhsm

# migrate-rhs-classic-to-rhsm --rhn-to-rhsm

migrate-rhs-classic-to-rhsm --status

# migrate-rhs-classic-to-rhsm --status

1. If you use Samba:
   1. For Red Hat Enterprise Linux 6.7 or higher, enable the following repository:

      # subscription-manager repos --enable=rh-gluster-3-samba-for-rhel-6-server-rpms

      Copy to Clipboard
      For Red Hat Enterprise Linux 7, enable the following repository:

      # subscription-manager repos --enable=rh-gluster-3-samba-for-rhel-7-server-rpms

      Copy to Clipboard
      For Red Hat Enterprise Linux 8, enable the following repository:

      # subscription-manager repos --enable=rh-gluster-3-samba-for-rhel-8-x86_64-rpms

      Copy to Clipboard
   2. Ensure that Samba is upgraded on all the nodes simultaneously, as running different versions of Samba in the same cluster will lead to data corruption.
      Stop the CTDB and SMB services.
      On Red Hat Enterprise Linux 7 and Red Hat Enterprise Linux 8:

      # systemctl stop ctdb

      Copy to Clipboard
      On Red Hat Enterprise Linux 6:

      # service ctdb stop

      Copy to Clipboard
      To verify that services are stopped, run:

      # ps axf | grep -E '(ctdb|smb|winbind|nmb)[d]'

      Copy to Clipboard
2. Upgrade the server to Red Hat Gluster Storage 3.5.

   # yum update

   Copy to Clipboard
   Wait for the update to complete.

   Important

   Run the following command to install nfs-ganesha-selinux on Red Hat Enterprise Linux 7:

   # yum install nfs-ganesha-selinux

   Copy to Clipboard

   Run the following command to install nfs-ganesha-selinux on Red Hat Enterprise Linux 8:

    # dnf install glusterfs-ganesha

   Copy to Clipboard
3. If you use Samba/CTDB, update the following files to replace META="all" with META="<ctdb_volume_name>", for example, META="ctdb":
   • /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh - This script ensures the file system and its lock volume are mounted on all Red Hat Gluster Storage servers that use Samba, and ensures that CTDB starts at system boot.
   • /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh - This script ensures that the file system and its lock volume are unmounted when the CTDB volume is stopped.

     Note

     For RHEL based Red Hat Gluster Storage upgrading to 3.5 batch update 4 with Samba, the write-behind translator has to manually disabled for all existing samba volumes.

     # gluster volume set <volname> performance.write-behind off

     Copy to Clipboard
4. Reboot the server to ensure that kernel updates are applied.
5. Ensure that glusterd and pcsd services are started.

   # systemctl start glusterd
   # systemctl start pcsd

   Copy to Clipboard

   Note

   During upgrade of servers, the glustershd.log file throws some “Invalid argument” errors during every index crawl (10 mins by default) on the upgraded nodes. It is *EXPECTED* and can be *IGNORED* until the op-version bump up, after which these errors are not triggered. Sample error message:

   [2021-05-25 17:58:38.007134] E [MSGID: 114031] [client-rpc-fops_v2.c:216:client4_0_mkdir_cbk] 0-spvol-client-40: remote operation failed. Path: (null) [Invalid argument]

   Copy to Clipboard

   If you are in op-version '70000' or lower, do not bump up the op-version to '70100' or higher until all the servers and clients are upgraded to the newer version.
6. When all nodes have been upgraded, run the following command to update the op-version of the cluster. This helps to prevent any compatibility issues within the cluster.

   # gluster volume set all cluster.op-version 70200

   Copy to Clipboard

   Note

   70200 is the cluster.op-version value for Red Hat Gluster Storage 3.5. Ater upgrading the cluster-op version, enable the granular-entry-heal for the volume via the given command:

   gluster volume heal $VOLNAME granular-entry-heal enable

   Copy to Clipboard

   The feature is now enabled by default post upgrade to Red Hat Gluster Storage 3.5, but this will come into affect only after bumping up the op-version. Refer to Section 1.5, “Red Hat Gluster Storage Software Components and Versions” for the correct cluster.op-version value for other versions.

   Important

   If the op-version is bumped up to '70100' after upgrading the servers and before upgrading the clients, some internal metadata files under the root of the mount point named '.glusterfs-anonymous-inode-(gfid)' exposed to the older clients. The clients must not do any I/O or remove or touch contents in this directory. The clients must upgrade to 3.5.4 or higher version, then this directory becomes invisible to the clients.
7. If you want to migrate from Gluster NFS to NFS Ganesha as part of this upgrade, install the NFS-Ganesha packages as described in Chapter 4, Deploying NFS-Ganesha on Red Hat Gluster Storage, and configure the NFS Ganesha cluster using the information in the NFS Ganesha section of the Red Hat Gluster Storage 3.5 Administration Guide.
8. Start all volumes.

   # for vol in `gluster volume list`; do gluster --mode=script volume start $vol; sleep 2s; done

   Copy to Clipboard
9. If you are using NFS-Ganesha:
   1. Copy the volume's export information from your backup copy of ganesha.conf to the new /etc/ganesha/ganesha.conf file.
      The export information in the backed up file is similar to the following:

      %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v1.conf"
      %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v2.conf"
      %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v3.conf"

      Copy to Clipboard

      Note

      With the release of 3.5 Batch Update 3, the mount point of shared storage is changed from /var/run/gluster/ to /run/gluster/ .
   2. Copy the backup volume export files from the backup directory to /etc/ganesha/exports by running the following command from the backup directory:

      # cp export.* /etc/ganesha/exports/

      Copy to Clipboard
10. Enable the shared volume.

    # gluster volume set all cluster.enable-shared-storage enable

    Copy to Clipboard
11. Ensure that the shared storage volume is mounted on the server. If the volume is not mounted, run the following command:

    # mount -t glusterfs hostname:gluster_shared_storage /var/run/gluster/shared_storage

    Copy to Clipboard
12. Ensure that the /var/run/gluster/shared_storage/nfs-ganesha directory is created.

    # cd /var/run/gluster/shared_storage/
    # mkdir nfs-ganesha

    Copy to Clipboard
13. Enable firewall settings for new services and ports. See Getting Started in the Red Hat Gluster Storage 3.5 Administration Guide.
14. If you use Samba/CTDB:
    1. Mount /gluster/lock before starting CTDB by executing the following commands:

       # mount <ctdb_volume_name>
       # mount -t glusterfs server:/ctdb_volume_name /gluster/lock/

       Copy to Clipboard
    2. Verify that the lock volume mounted correctly by checking for lock in the output of the mount command on any Samba server.

       # mount | grep 'lock'
       ...
       <hostname>:/<ctdb_volume_name>.tcp on /gluster/lock type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

       Copy to Clipboard
    3. If all servers that host volumes accessed via SMB have been updated, then start the CTDB and Samba services by executing the following commands.
       On Red Hat Enterprise Linux 7 and Red Hat Enterprise Linux 8:

       # systemctl start ctdb

       Copy to Clipboard
       On Red Hat Enterprise Linux 6:

       # service ctdb start

       Copy to Clipboard
    4. To verify that the CTDB and SMB services have started, execute the following command:

       ps axf | grep -E '(ctdb|smb|winbind|nmb)[d]'

       Copy to Clipboard
15. If you use NFS-Ganesha:
    1. Copy the ganesha.conf and ganesha-ha.conf files, and the /etc/ganesha/exports directory to the /var/run/gluster/shared_storage/nfs-ganesha directory.

       # cd /etc/ganesha/
       # cp ganesha.conf  ganesha-ha.conf /var/run/gluster/shared_storage/nfs-ganesha/
       # cp -r exports/ /var/run/gluster/shared_storage/nfs-ganesha/

       Copy to Clipboard
    2. Update the path of any export entries in the ganesha.conf file.

       # sed -i 's/\/etc\/ganesha/\/var\/run\/gluster\/shared_storage\/nfs-ganesha/' /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf

       Copy to Clipboard
    3. Run the following to clean up any existing cluster configuration:

       /usr/libexec/ganesha/ganesha-ha.sh --cleanup /var/run/gluster/shared_storage/nfs-ganesha

       Copy to Clipboard
    4. If you have upgraded to Red Hat Enterprise Linux 7.4 or later, set the following SELinux Boolean:

       # setsebool -P ganesha_use_fusefs on

       Copy to Clipboard
    5. Start the nfs-ganesha service and verify that all nodes are functional.

       # gluster nfs-ganesha enable

       Copy to Clipboard
    6. Enable NFS-Ganesha on all volumes.

       # gluster volume set volname ganesha.enable on

       Copy to Clipboard

Procedure 5.3. Before you upgrade

1. Back up the following configuration directory and files in a location that is not on the operating system partition.
   • /var/lib/glusterd
   • /etc/samba
   • /etc/ctdb
   • /etc/glusterfs
   • /var/lib/samba
   • /var/lib/ctdb
   • /var/run/gluster/shared_storage/nfs-ganesha

   Note

   With the release of 3.5 Batch Update 3, the mount point of shared storage is changed from /var/run/gluster/ to /run/gluster/ .
   If you use NFS-Ganesha, back up the following files from all nodes:
   • /run/gluster/shared_storage/nfs-ganesha/exports/export.*.conf
   • /etc/ganesha/ganesha.conf
   • /etc/ganesha/ganesha-ha.conf
   If upgrading from Red Hat Gluster Storage 3.3 to 3.4 or subsequent releases, back up all xattr by executing the following command individually on the brick root(s) for all nodes:

   # find ./ -type d ! -path "./.*" ! -path "./" | xargs getfattr -d -m. -e hex > /var/log/glusterfs/xattr_dump_brick_name

   Copy to Clipboard
2. Unmount gluster volumes from all clients. On a client, use the following command to unmount a volume from a mount point.

   # umount mount-point

   Copy to Clipboard
3. If you use NFS-Ganesha, run the following on a gluster server to disable the nfs-ganesha service:

   # gluster nfs-ganesha disable

   Copy to Clipboard
4. On a gluster server, disable the shared volume.

   # gluster volume set all cluster.enable-shared-storage disable

   Copy to Clipboard
5. Stop all volumes.

   # for vol in `gluster volume list`; do gluster --mode=script volume stop $vol; sleep 2s; done

   Copy to Clipboard
6. Verify that all volumes are stopped.

   # gluster volume info

   Copy to Clipboard
7. Stop the glusterd services on all servers using the following command:

   # service glusterd stop
   # pkill glusterfs
   # pkill glusterfsd

   Copy to Clipboard
8. Stop the pcsd service.

   # systemctl stop pcsd

   Copy to Clipboard

Procedure 5.4. Upgrade using Satellite

1. Create an Activation Key at the Red Hat Network Satellite Server, and associate it with the following channels. For more information, see Section 2.6, “Installing from Red Hat Satellite Server”
   • For Red Hat Enterprise Linux 6.7 or higher:

     Base Channel: Red Hat Enterprise Linux Server (v.6 for 64-bit x86_64)
     
     Child channels:
     RHEL Server Scalable File System (v. 6 for x86_64)
     Red Hat Gluster Storage Server 3 (RHEL 6 for x86_64)

     Copy to Clipboard
     If you use Samba, add the following channel:

     Red Hat Gluster 3 Samba (RHEL 6 for x86_64)

     Copy to Clipboard
   • For Red Hat Enterprise Linux 7:

     Base Channel: Red Hat Enterprise Linux Server (v.7 for 64-bit x86_64)
     
     Child channels:
     RHEL Server Scalable File System (v. 7 for x86_64)
     Red Hat Gluster Storage Server 3 (RHEL 7 for x86_64)

     Copy to Clipboard
     If you use Samba, add the following channel:

     Red Hat Gluster 3 Samba (RHEL 7 for x86_64)

     Copy to Clipboard
2. Unregister your system from Red Hat Network Satellite by following these steps:
   1. Log in to the Red Hat Network Satellite server.
   2. Click on the Systems tab in the top navigation bar and then the name of the old or duplicated system in the System List.
   3. Click the delete system link in the top-right corner of the page.
   4. To confirm the system profile deletion by clicking the Delete System button.
3. Run the following command on your Red Hat Gluster Storage server, using your credentials and the Activation Key you prepared earlier. This re-registers the system to the Red Hat Gluster Storage 3.5 channels on the Red Hat Network Satellite Server.

   # rhnreg_ks --username username --password password --force --activationkey Activation Key ID

   Copy to Clipboard
4. Verify that the channel subscriptions have been updated.
   On Red Hat Enterprise Linux 6.7 and higher, look for the following channels, as well as the rh-gluster-3-samba-for-rhel-6-server-rpms channel if you use Samba.

   # rhn-channel --list
   rhel-6-server-rpms
   rhel-scalefs-for-rhel-6-server-rpms
   rhs-3-for-rhel-6-server-rpms

   Copy to Clipboard
   On Red Hat Enterprise Linux 7, look for the following channels, as well as the rh-gluster-3-samba-for-rhel-7-server-rpms channel if you use Samba.

   # rhn-channel --list
   rhel-7-server-rpms
   rh-gluster-3-for-rhel-7-server-rpms

   Copy to Clipboard
5. Upgrade to Red Hat Gluster Storage 3.5.

   # yum update

   Copy to Clipboard

   Important

   Run the following command to install nfs-ganesha-selinux on Red Hat Enterprise Linux 7:

   # yum install nfs-ganesha-selinux

   Copy to Clipboard

   Run the following command to install nfs-ganesha-selinux on Red Hat Enterprise Linux 8:

    # dnf install glusterfs-ganesha

   Copy to Clipboard
6. Reboot the server and run volume and data integrity checks.
7. When all nodes have been upgraded, run the following command to update the op-version of the cluster. This helps to prevent any compatibility issues within the cluster.

   # gluster volume set all cluster.op-version 70200

   Copy to Clipboard

   Note

   70200 is the cluster.op-version value for Red Hat Gluster Storage 3.5. Ater upgrading the cluster-op version, enable the granular-entry-heal for the volume via the given command:

   gluster volume heal $VOLNAME granular-entry-heal enable

   Copy to Clipboard

   The feature is now enabled by default post upgrade to Red Hat Gluster Storage 3.5, but this will come into affect only after bumping up the op-version. Refer to Section 1.5, “Red Hat Gluster Storage Software Components and Versions” for the correct cluster.op-version value for other versions.
8. Start all volumes.

   # for vol in `gluster volume list`; do gluster --mode=script volume start $vol; sleep 2s; done

   Copy to Clipboard
9. If you are using NFS-Ganesha:
   1. Copy the volume's export information from your backup copy of ganesha.conf to the new /etc/ganesha/ganesha.conf file.
      The export information in the backed up file is similar to the following:

      %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v1.conf"
      %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v2.conf"
      %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.v3.conf"

      Copy to Clipboard

      Note

      With the release of 3.5 Batch Update 3, the mount point of shared storage is changed from /var/run/gluster/ to /run/gluster/ .
   2. Copy the backup volume export files from the backup directory to /etc/ganesha/exports by running the following command from the backup directory:

      # cp export.* /etc/ganesha/exports/

      Copy to Clipboard
10. Enable the shared volume.

    # gluster volume set all cluster.enable-shared-storage enable

    Copy to Clipboard
11. Ensure that the shared storage volume is mounted on the server. If the volume is not mounted, run the following command:

    # mount -t glusterfs hostname:gluster_shared_storage /var/run/gluster/shared_storage

    Copy to Clipboard
12. Ensure that the /var/run/gluster/shared_storage/nfs-ganesha directory is created.

    # cd /var/run/gluster/shared_storage/
    # mkdir nfs-ganesha

    Copy to Clipboard
13. Enable firewall settings for new services and ports. See Getting Started in the Red Hat Gluster Storage 3.5 Administration Guide.
14. If you use Samba/CTDB:
    1. Mount /gluster/lock before starting CTDB by executing the following commands:

       # mount <ctdb_volume_name>
       # mount -t glusterfs server:/ctdb_volume_name /gluster/lock/

       Copy to Clipboard
    2. Verify that the lock volume mounted correctly by checking for lock in the output of the mount command on any Samba server.

       # mount | grep 'lock'
       ...
       <hostname>:/<ctdb_volume_name>.tcp on /gluster/lock type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

       Copy to Clipboard
    3. If all servers that host volumes accessed via SMB have been updated, then start the CTDB and Samba services by executing the following commands.
       On Red Hat Enterprise Linux 7 and Red Hat Enterprise Linux 8:

       # systemctl start ctdb

       Copy to Clipboard
       On Red Hat Enterprise Linux 6:

       # service ctdb start

       Copy to Clipboard
    4. To verify that the CTDB and SMB services have started, execute the following command:

       ps axf | grep -E '(ctdb|smb|winbind|nmb)[d]'

       Copy to Clipboard
15. If you use NFS-Ganesha:
    1. Copy the ganesha.conf and ganesha-ha.conf files, and the /etc/ganesha/exports directory to the /var/run/gluster/shared_storage/nfs-ganesha directory.

       # cd /etc/ganesha/
       # cp ganesha.conf  ganesha-ha.conf /var/run/gluster/shared_storage/nfs-ganesha/
       # cp -r exports/ /var/run/gluster/shared_storage/nfs-ganesha/

       Copy to Clipboard
    2. Update the path of any export entries in the ganesha.conf file.

       # sed -i 's/\/etc\/ganesha/\/var\/run\/gluster\/shared_storage\/nfs-ganesha/' /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf

       Copy to Clipboard
    3. Run the following to clean up any existing cluster configuration:

       /usr/libexec/ganesha/ganesha-ha.sh --cleanup /var/run/gluster/shared_storage/nfs-ganesha

       Copy to Clipboard
    4. If you have upgraded to Red Hat Enterprise Linux 7.4 or later, set the following SELinux Boolean:

       # setsebool -P ganesha_use_fusefs on

       Copy to Clipboard
16. Start the ctdb service (and nfs-ganesha service, if used) and verify that all nodes are functional.

    # systemctl start ctdb
    # gluster nfs-ganesha enable

    Copy to Clipboard
17. If this deployment uses NFS-Ganesha, enable NFS-Ganesha on all volumes.

    # gluster volume set volname ganesha.enable on

    Copy to Clipboard