Chapter 6. Migrating the Object Storage service to Red Hat OpenStack Services on OpenShift nodes

1. Add new nodes to the Object Storage service rings.
2. Set weights of existing nodes to 0.
3. Rebalance rings by moving one replica.
4. Copy rings to old nodes and restart services.
5. Check replication status and repeat the previous two steps until the old nodes are drained.
6. Remove the old nodes from the rings.

Procedure

1. Add new nodes by scaling up the SwiftStorage resource from 0 to 3:

   $ oc patch openstackcontrolplane openstack --type=merge -p='{"spec":{"swift":{"template":{"swiftStorage":{"replicas": 3}}}}}'

   Copy to Clipboard Toggle word wrap

   This command creates three storage instances on the Red Hat OpenShift Container Platform (RHOCP) cluster that use Persistent Volume Claims.

2. Wait until all three pods are running and the rings include the new devices:

   $ oc wait pods --for condition=Ready -l component=swift-storage
   $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-ring-builder object.builder search --device pv'

   Copy to Clipboard Toggle word wrap

3. From the current rings, get the storage management IP addresses of the nodes to drain:

   $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-ring-builder object.builder search _' | tail -n +2 | awk '{print $4}' | sort -u

   Copy to Clipboard Toggle word wrap

   The output looks similar to the following:

   172.20.0.100
   swift-storage-0.swift-storage.openstack.svc
   swift-storage-1.swift-storage.openstack.svc
   swift-storage-2.swift-storage.openstack.svc

   Copy to Clipboard Toggle word wrap

4. Drain the old nodes. In the following example, the old node 172.20.0.100 is drained:

   $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c '
   swift-ring-tool get
   swift-ring-tool drain 172.20.0.100
   swift-ring-tool rebalance
   swift-ring-tool push'

   Copy to Clipboard Toggle word wrap

   Depending on your deployment, you might have more nodes to include in the command.

5. Copy and apply the updated rings to the original nodes. Run the ssh commands for your existing nodes that store the Object Storage service data:

   $ oc extract --confirm cm/swift-ring-files
   $CONTROLLER1_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz
   $CONTROLLER1_SSH "systemctl restart tripleo_swift_*"

   Copy to Clipboard Toggle word wrap

6. Track the replication progress by using the swift-dispersion-report tool:

   $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c "swift-ring-tool get && swift-dispersion-report"

   Copy to Clipboard Toggle word wrap

   The output shows less than 100% of copies found. Repeat the command until all container and object copies are found:

   Queried 1024 containers for dispersion reporting, 6s, 0 retries
   There were 5 partitions missing 1 copy.
   99.84% of container copies found (3067 of 3072)
   Sample represents 100.00% of the container partition space
   Queried 1024 objects for dispersion reporting, 7s, 0 retries
   There were 739 partitions missing 1 copy.
   There were 285 partitions missing 0 copies.
   75.94% of object copies found (2333 of 3072)
   Sample represents 100.00% of the object partition space

   Copy to Clipboard Toggle word wrap

7. Move the next replica to the new nodes by rebalancing and distributing the rings:

   $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c '
   swift-ring-tool get
   swift-ring-tool rebalance
   swift-ring-tool push'
   
   $ oc extract --confirm cm/swift-ring-files
   $CONTROLLER1_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz
   $CONTROLLER1_SSH "systemctl restart tripleo_swift_*"

   Copy to Clipboard Toggle word wrap

   Monitor the swift-dispersion-report output again, wait until all copies are found, and then repeat this step until all your replicas are moved to the new nodes.

8. Remove the nodes from the rings:

   $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c '
   swift-ring-tool get
   swift-ring-tool remove 172.20.0.100
   swift-ring-tool rebalance
   swift-ring-tool push'

   Copy to Clipboard Toggle word wrap
   Note

   Even if all replicas are on the new nodes and the swift-dispersion-report command reports 100% of the copies found, there might still be data on the old nodes. The replicators remove this data, but it might take more time.

Verification

1. Check the disk usage of all disks in the cluster:

   $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-recon -d'

   Copy to Clipboard Toggle word wrap

2. Confirm that there are no more \*.db or *.data files in the /srv/node directory on the nodes:

   $CONTROLLER1_SSH "find /srv/node/ -type f -name '*.db' -o -name '*.data' | wc -l"

   Copy to Clipboard Toggle word wrap