9.9. Checking Object Storage service cluster health

Procedure

1. Log in to one of the Controller nodes.

2. Run the following command:

   $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-recon -arqlT --md5'

   • Optional: Use the --all option to return additional output.

     This command queries all servers on the ring for the following data:

     • Async pendings: If the cluster load is too high and processes cannot update database files fast enough, some updates occur asynchronously. These numbers decrease over time as the load normalizes.
     • Replication metrics: Review the replication timestamps; full replication passes happen frequently with few errors. An old entry, for example, an entry with a timestamp from six months ago, indicates that replication on the node has not completed in the last six months.
     • Ring md5sums: This ensures that all ring files are consistent on all nodes.
     • swift.conf md5sums: This ensures that all configuration files are consistent on all nodes.
     • Quarantined files: There must be no, or very few, quarantined files for all nodes.
     • Time-sync: All nodes must be synchronized.