3.6. Moving etcd to a different disk

Procedure

1. Attach the new disk to the cluster and verify that the disk is detected in the node by running the lsblk command in a debug shell:

   $ oc debug node/<node_name>

   # lsblk

   Note the device name of the new disk reported by the lsblk command.

2. Create the following script and name it etcd-find-secondary-device.sh:

   #!/bin/bash
   set -uo pipefail
   
   for device in <device_type_glob>; do 

   1

   
   /usr/sbin/blkid "${device}" &> /dev/null
    if [ $? == 2  ]; then
       echo "secondary device found ${device}"
       echo "creating filesystem for etcd mount"
       mkfs.xfs -L var-lib-etcd -f "${device}" &> /dev/null
       udevadm settle
       touch /etc/var-lib-etcd-mount
       exit
    fi
   done
   echo "Couldn't find secondary block device!" >&2
   exit 77

   1

   Replace <device_type_glob> with a shell glob for your block device type. For SCSI or SATA drives, use /dev/sd*; for virtual drives, use /dev/vd*; for NVMe drives, use /dev/nvme*[0-9]*n*.

3. Create a base64-encoded string from the etcd-find-secondary-device.sh script and note its contents:

   $ base64 -w0 etcd-find-secondary-device.sh

4. Create a MachineConfig YAML file named etcd-mc.yml with contents such as the following:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 98-var-lib-etcd
   spec:
     config:
       ignition:
         version: 3.5.0
       storage:
         files:
           - path: /etc/find-secondary-device
             mode: 0755
             contents:
               source: data:text/plain;charset=utf-8;base64,<encoded_etcd_find_secondary_device_script> 

   1

   
       systemd:
         units:
           - name: find-secondary-device.service
             enabled: true
             contents: |
               [Unit]
               Description=Find secondary device
               DefaultDependencies=false
               After=systemd-udev-settle.service
               Before=local-fs-pre.target
               ConditionPathExists=!/etc/var-lib-etcd-mount
   
               [Service]
               RemainAfterExit=yes
               ExecStart=/etc/find-secondary-device
   
               RestartForceExitStatus=77
   
               [Install]
               WantedBy=multi-user.target
           - name: var-lib-etcd.mount
             enabled: true
             contents: |
               [Unit]
               Before=local-fs.target
   
               [Mount]
               What=/dev/disk/by-label/var-lib-etcd
               Where=/var/lib/etcd
               Type=xfs
               TimeoutSec=120s
   
               [Install]
               RequiredBy=local-fs.target
           - name: sync-var-lib-etcd-to-etcd.service
             enabled: true
             contents: |
               [Unit]
               Description=Sync etcd data if new mount is empty
               DefaultDependencies=no
               After=var-lib-etcd.mount var.mount
               Before=crio.service
   
               [Service]
               Type=oneshot
               RemainAfterExit=yes
               ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
               ExecStart=/usr/sbin/setsebool -P rsync_full_access 1
               ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
               ExecStart=/usr/sbin/semanage fcontext -a -t container_var_lib_t '/var/lib/etcd(/.*)?'
               ExecStart=/usr/sbin/setsebool -P rsync_full_access 0
               TimeoutSec=0
   
               [Install]
               WantedBy=multi-user.target graphical.target
           - name: restorecon-var-lib-etcd.service
             enabled: true
             contents: |
               [Unit]
               Description=Restore recursive SELinux security contexts
               DefaultDependencies=no
               After=var-lib-etcd.mount
               Before=crio.service
   
               [Service]
               Type=oneshot
               RemainAfterExit=yes
               ExecStart=/sbin/restorecon -R /var/lib/etcd/
               TimeoutSec=0
   
               [Install]
               WantedBy=multi-user.target graphical.target

   1

   Replace <encoded_etcd_find_secondary_device_script> with the encoded script contents that you noted.

5. Apply the created MachineConfig YAML file:

   $ oc create -f etcd-mc.yml