9.6. Replacing Object Storage Nodes
To replace nodes on the Object Storage cluster, you need to:
- Update the Overcloud with the new Object Storage nodes and prevent Director from creating the ring files.
- Manually add/remove the nodes to the cluster using
swift-ring-builder.
The following procedure describes how to replace nodes while maintaining the integrity of the cluster. In this example, we have a two node Object Storage cluster. The aim is to add an additional node, then replace the faulty node.
First, create an environment file called
~/templates/swift-ring-prevent.yaml with the following content:
parameter_defaults: SwiftRingBuild: false RingBuild: false ObjectStorageCount: 3
The
SwiftRingBuild and RingBuild parameters define whether the Overcloud automatically builds the ring files for Object Storage and Controller nodes respectively. The ObjectStorageCount defines how many Object Storage nodes in our environment. In this situation, we scale from 2 to 3 nodes.
Include the
swift-ring-prevent.yaml file with the rest of your Overcloud’s environment files as part of the openstack overcloud deploy:
$ openstack overcloud deploy --templates [ENVIRONMENT_FILES] -e swift-ring-prevent.yaml
Note
Add this file to the end of the environment file list so its parameters supercede previous environment file parameters.
After redeployment completes, the Overcloud now contains an additional Object Storage node. However, the node's storage directory has not been created and ring files for the node's object store are unbuilt. This means you must create the storage directory and build the ring files manually.
Note
Use the following procedure to also build ring files on Controller nodes.
Login to the new node and create the storage directory:
$ sudo mkdir -p /srv/node/d1 $ sudo chown -R swift:swift /srv/node/d1
Note
You can also mount an external storage device at this directory.
Copy the existing ring files to the node. Log into a Controller node as the
heat-admin user and then change to the superuser. For example, given a Controller node with an IP address of 192.168.201.24.
$ ssh heat-admin@192.168.201.24 $ sudo -i
Copy the
/etc/swift/*.builder files from the Controller node to the new Object Storage node's /etc/swift/ directory. If necessary, transfer the files to the director host:
[root@overcloud-controller-0 ~]# scp /etc/swift/*.builder stack@192.1.2.1:~/.
Then transfer the files to the new node:
[stack@director ~]$ scp ~/*.builder heat-admin@192.1.2.24:~/.
Log into the new Object Storage node as the
heat-admin user and then change to the superuser. For example, given a Object Storage node with an IP address of 192.168.201.29.
$ ssh heat-admin@192.168.201.29 $ sudo -i
Copy the files to the
/etc/swift directory:
# cp /home/heat-admin/*.builder /etc/swift/.
Add the new Object Storage node to the account, container, and object rings. Run the following commands for the new node:
# swift-ring-builder /etc/swift/account.builder add zX-IP:6002/d1 weight # swift-ring-builder /etc/swift/container.builder add zX-IP:6001/d1 weight # swift-ring-builder /etc/swift/object.builder add zX-IP:6000/d1 weight
Replace the following values in these commands:
- zX
- Replace X with the corresponding integer of a specified zone (for example, z1 for Zone 1).
- IP
- The IP that the account, container, and object services use to listen. This should match the IP address of each storage node; specifically, the value of
bind_ipin theDEFAULTsections of/etc/swift/object-server.conf,/etc/swift/account-server.conf, and/etc/swift/container-server.conf. - weight
- Describes relative weight of the device in comparison to other devices. This is usually 100.
Note
Check the existing values of the current nodes in the ring file using the
swift-ring-builder on the rings files alone:
# swift-ring-builder /etc/swift/account.builder
Remove the node you aim to replace from the account, container, and object rings. Run the following commands for each node:
# swift-ring-builder /etc/swift/account.builder remove IP # swift-ring-builder /etc/swift/container.builder remove IP # swift-ring-builder /etc/swift/object.builder remove IP
Replace IP with the IP address of the node.
Redistribute the partitions across all the nodes:
# swift-ring-builder /etc/swift/account.builder rebalance # swift-ring-builder /etc/swift/container.builder rebalance # swift-ring-builder /etc/swift/object.builder rebalance
Change the ownership of all
/etc/swift/ contents to the root user and swift group:
# chown -R root:swift /etc/swift
Restart the
openstack-swift-proxy service:
# systemctl restart openstack-swift-proxy.service
At this point, the ring files (*.ring.gz and *.builder) should be updated on the new node:
/etc/swift/account.builder /etc/swift/account.ring.gz /etc/swift/container.builder /etc/swift/container.ring.gz /etc/swift/object.builder /etc/swift/object.ring.gz
Copy these files to
/etc/swift/ on the Controller nodes and the existing Object Storage nodes (except for the node to remove). If necessary, transfer the files to the director host:
[root@overcloud-objectstorage-2 swift]# scp *.builder stack@192.1.2.1:~/ [root@overcloud-objectstorage-2 swift]# scp *.ring.gz stack@192.1.2.1:~/
Then copy the files to the
/etc/swift/ on each node.
On each node, change the ownership of all
/etc/swift/ contents to the root user and swift group:
# chown -R root:swift /etc/swift
The new node is added and a part of the ring. Before removing the old node from the ring, check that the new node completes a full data replication pass.
To remove the old node from the ring, reduce the
ObjectStorageCount to the omit the old ring. In this case, we reduce from 3 to 2:
parameter_defaults: SwiftRingBuild: false RingBuild: false ObjectStorageCount: 2
Create a new environment file (
remove-object-node.yaml) to identify and remove the old Object Storage node. In this case, we remove overcloud-objectstorage-1:
parameter_defaults:
ObjectStorageRemovalPolicies:
[{'resource_list': ['1']}]
Include both environment files with the deployment command:
$ openstack overcloud deploy --templates -e swift-ring-prevent.yaml -e remove-object-node.yaml ...
The director deletes the Object Storage node from the Overcloud and updates the rest of the nodes on the Overcloud to accommodate the node removal.
