Este contenido no está disponible en el idioma seleccionado.

Chapter 5. Host-level tasks

5.1. Adding a host to the cluster
Copiar enlace

For information on adding master or node hosts to a cluster, see the Adding hosts to an existing cluster section in the Install and configuration guide.

5.2. Master host tasks
Copiar enlace

5.2.1. Deprecating a master host
Copiar enlace

Deprecating a master host removes it from the OpenShift Container Platform environment.

The reasons to deprecate or scale down master hosts include hardware re-sizing or replacing the underlying infrastructure.

Highly available OpenShift Container Platform environments require at least three master hosts and three etcd nodes. Usually, the master hosts are colocated with the etcd services. If you deprecate a master host, you also remove the etcd static pods from that host.

Important

Ensure that the master and etcd services are always deployed in odd numbers due to the voting mechanisms that take place among those services.

5.2.1.1. Creating a master host backup
Copiar enlace

Perform this backup process before any change to the OpenShift Container Platform infrastructure, such as a system update, upgrade, or any other significant modification. Back up data regularly to ensure that recent data is available if a failure occurs.

OpenShift Container Platform files

The master instances run important services, such as the API, controllers. The /etc/origin/master directory stores many important files:

The configuration, the API, controllers, services, and more
Certificates generated by the installation
All cloud provider-related configuration
Keys and other authentication files, such as htpasswd if you use htpasswd
And more

You can customize OpenShift Container Platform services, such as increasing the log level or using proxies. The configuration files are stored in the /etc/sysconfig directory.

Because the masters are also nodes, back up the entire /etc/origin directory.

Procedure

Important

You must perform the following steps on each master node.

Create a backup of the pod definitions, located here.
Create a backup of the master host configuration files:
```
MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
sudo cp -aR /etc/sysconfig/ ${MYBACKUPDIR}/etc/sysconfig/
```
```
$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
$ sudo cp -aR /etc/sysconfig/ ${MYBACKUPDIR}/etc/sysconfig/
```
Copy to Clipboard Toggle word wrap
Note
The master configuration file is/etc/origin/master/master-config.yaml.
Warning
The /etc/origin/master/ca.serial.txt file is generated on only the first master listed in the Ansible host inventory. If you deprecate the first master host, copy the /etc/origin/master/ca.serial.txt file to the rest of master hosts before the process.
Important
In OpenShift Container Platform 3.11 clusters running multiple masters, one of the master nodes includes additional CA certificates in /etc/origin/master, /etc/etcd/ca and /etc/etcd/generated_certs. These are required for application node and etcd node scale-up operations and would need to be restored on another master node should the originating master become permanently unavailable. These directories are included by default within the backup procedures documented here.

Other important files that need to be considered when planning a backup include:

Expand

File	Description
`/etc/cni/*`	Container Network Interface configuration (if used)
`/etc/sysconfig/iptables`	Where the `iptables` rules are stored
`/etc/sysconfig/docker-storage-setup`	The input file for `container-storage-setup` command
`/etc/sysconfig/docker`	The `docker` configuration file
`/etc/sysconfig/docker-network`	`docker` networking configuration (i.e. MTU)
`/etc/sysconfig/docker-storage`	`docker` storage configuration (generated by `container-storage-setup`)
`/etc/dnsmasq.conf`	Main configuration file for `dnsmasq`
`/etc/dnsmasq.d/*`	Different `dnsmasq` configuration files
`/etc/sysconfig/flanneld`	`flannel` configuration file (if used)
`/etc/pki/ca-trust/source/anchors/`	Certificates added to the system (i.e. for external registries)

Create a backup of those files:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    ${MYBACKUPDIR}/etc/sysconfig/
sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
$ sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    ${MYBACKUPDIR}/etc/sysconfig/
$ sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
$ sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

Copy to Clipboard

Toggle word wrap

If a package is accidentally removed or you need to resore a file that is included in an rpm package, having a list of rhel packages installed on the system can be useful.
Note
If you use Red Hat Satellite features, such as content views or the facts store, provide a proper mechanism to reinstall the missing packages and a historical data of packages installed in the systems.
To create a list of the current rhel packages installed in the system:
```
MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo mkdir -p ${MYBACKUPDIR}
rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt
```
```
$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}
$ rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt
```
Copy to Clipboard Toggle word wrap

If you used the previous steps, the following files are present in the backup directory:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'
etc/sysconfig/flanneld
etc/sysconfig/iptables
etc/sysconfig/docker-network
etc/sysconfig/docker-storage
etc/sysconfig/docker-storage-setup
etc/sysconfig/docker-storage-setup.rpmnew
etc/origin/master/ca.crt
etc/origin/master/ca.key
etc/origin/master/ca.serial.txt
etc/origin/master/ca-bundle.crt
etc/origin/master/master.proxy-client.crt
etc/origin/master/master.proxy-client.key
etc/origin/master/service-signer.crt
etc/origin/master/service-signer.key
etc/origin/master/serviceaccounts.private.key
etc/origin/master/serviceaccounts.public.key
etc/origin/master/openshift-master.crt
etc/origin/master/openshift-master.key
etc/origin/master/openshift-master.kubeconfig
etc/origin/master/master.server.crt
etc/origin/master/master.server.key
etc/origin/master/master.kubelet-client.crt
etc/origin/master/master.kubelet-client.key
etc/origin/master/admin.crt
etc/origin/master/admin.key
etc/origin/master/admin.kubeconfig
etc/origin/master/etcd.server.crt
etc/origin/master/etcd.server.key
etc/origin/master/master.etcd-client.key
etc/origin/master/master.etcd-client.csr
etc/origin/master/master.etcd-client.crt
etc/origin/master/master.etcd-ca.crt
etc/origin/master/policy.json
etc/origin/master/scheduler.json
etc/origin/master/htpasswd
etc/origin/master/session-secrets.yaml
etc/origin/master/openshift-router.crt
etc/origin/master/openshift-router.key
etc/origin/master/registry.crt
etc/origin/master/registry.key
etc/origin/master/master-config.yaml
etc/origin/generated-configs/master-master-1.example.com/master.server.crt
...[OUTPUT OMITTED]...
etc/origin/cloudprovider/openstack.conf
etc/origin/node/system:node:master-0.example.com.crt
etc/origin/node/system:node:master-0.example.com.key
etc/origin/node/ca.crt
etc/origin/node/system:node:master-0.example.com.kubeconfig
etc/origin/node/server.crt
etc/origin/node/server.key
etc/origin/node/node-dnsmasq.conf
etc/origin/node/resolv.conf
etc/origin/node/node-config.yaml
etc/origin/node/flannel.etcd-client.key
etc/origin/node/flannel.etcd-client.csr
etc/origin/node/flannel.etcd-client.crt
etc/origin/node/flannel.etcd-ca.crt
etc/pki/ca-trust/source/anchors/openshift-ca.crt
etc/pki/ca-trust/source/anchors/registry-ca.crt
etc/dnsmasq.conf
etc/dnsmasq.d/origin-dns.conf
etc/dnsmasq.d/origin-upstream-dns.conf
etc/dnsmasq.d/node-dnsmasq.conf
packages.txt

Copy to Clipboard

Toggle word wrap

If needed, you can compress the files to save space:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
sudo rm -Rf ${MYBACKUPDIR}

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
$ sudo rm -Rf ${MYBACKUPDIR}

Copy to Clipboard

Toggle word wrap

To create any of these files from scratch, the openshift-ansible-contrib repository contains the backup_master_node.sh script, which performs the previous steps. The script creates a directory on the host where you run the script and copies all the files previously mentioned.

Note

The openshift-ansible-contrib script is not supported by Red Hat, but the reference architecture team performs testing to ensure the code operates as defined and is secure.

You can run the script on every master host with:

mkdir ~/git
cd ~/git
git clone https://github.com/openshift/openshift-ansible-contrib.git
cd openshift-ansible-contrib/reference-architecture/day2ops/scripts
./backup_master_node.sh -h

$ mkdir ~/git
$ cd ~/git
$ git clone https://github.com/openshift/openshift-ansible-contrib.git
$ cd openshift-ansible-contrib/reference-architecture/day2ops/scripts
$ ./backup_master_node.sh -h

Copy to Clipboard

Toggle word wrap

5.2.1.2. Backing up etcd
Copiar enlace

When you back up etcd, you must back up both the etcd configuration files and the etcd data.

5.2.1.2.1. Backing up etcd configuration files
Copiar enlace

The etcd configuration files to be preserved are all stored in the /etc/etcd directory of the instances where etcd is running. This includes the etcd configuration file (/etc/etcd/etcd.conf) and the required certificates for cluster communication. All those files are generated at installation time by the Ansible installer.

Procedure

For each etcd member of the cluster, back up the etcd configuration.

ssh master-0
mkdir -p /backup/etcd-config-$(date +%Y%m%d)/
cp -R /etc/etcd/ /backup/etcd-config-$(date +%Y%m%d)/

$ ssh master-0


# mkdir -p /backup/etcd-config-$(date +%Y%m%d)/
# cp -R /etc/etcd/ /backup/etcd-config-$(date +%Y%m%d)/

Copy to Clipboard

Toggle word wrap

1: Replace master-0 with the name of your etcd member.

Note

The certificates and configuration files on each etcd cluster member are unique.

5.2.1.2.2. Backing up etcd data
Copiar enlace

Prerequisites

Note

The OpenShift Container Platform installer creates aliases to avoid typing all the flags named etcdctl2 for etcd v2 tasks and etcdctl3 for etcd v3 tasks.

However, the etcdctl3 alias does not provide the full endpoint list to the etcdctl command, so you must specify the --endpoints option and list all the endpoints.

Before backing up etcd:

etcdctl binaries must be available or, in containerized installations, the rhel7/etcd container must be available.
Ensure that the OpenShift Container Platform API service is running.
Ensure connectivity with the etcd cluster (port 2379/tcp).
Ensure the proper certificates to connect to the etcd cluster.

Procedure

Note

While the etcdctl backup command is used to perform the backup, etcd v3 has no concept of a backup. Instead, you either take a snapshot from a live member with the etcdctl snapshot save command or copy the member/snap/db file from an etcd data directory.

The etcdctl backup command rewrites some of the metadata contained in the backup, specifically, the node ID and cluster ID, which means that in the backup, the node loses its former identity. To recreate a cluster from the backup, you create a new, single-node cluster, then add the rest of the nodes to the cluster. The metadata is rewritten to prevent the new node from joining an existing cluster.

Back up the etcd data:

Important

Clusters upgraded from previous versions of OpenShift Container Platform might contain v2 data stores. Back up all etcd data stores.

Obtain the etcd endpoint IP address from the static pod manifest:

export ETCD_POD_MANIFEST="/etc/origin/node/pods/etcd.yaml"

$ export ETCD_POD_MANIFEST="/etc/origin/node/pods/etcd.yaml"

Copy to Clipboard

Toggle word wrap

export ETCD_EP=$(grep https ${ETCD_POD_MANIFEST} | cut -d '/' -f3)

$ export ETCD_EP=$(grep https ${ETCD_POD_MANIFEST} | cut -d '/' -f3)

Copy to Clipboard

Toggle word wrap

Log in as an administrator:
```
oc login -u system:admin
```
```
$ oc login -u system:admin
```
Copy to Clipboard Toggle word wrap

Obtain the etcd pod name:

export ETCD_POD=$(oc get pods -n kube-system | grep -o -m 1 '^master-etcd\S*')

$ export ETCD_POD=$(oc get pods -n kube-system | grep -o -m 1 '^master-etcd\S*')

Copy to Clipboard

Toggle word wrap

Change to the kube-system project:
```
oc project kube-system
```
```
$ oc project kube-system
```
Copy to Clipboard Toggle word wrap

Take a snapshot of the etcd data in the pod and store it locally:

oc exec ${ETCD_POD} -c etcd -- /bin/bash -c "ETCDCTL_API=3 etcdctl \
    --cert /etc/etcd/peer.crt \
    --key /etc/etcd/peer.key \
    --cacert /etc/etcd/ca.crt \
    --endpoints $ETCD_EP \
    snapshot save /var/lib/etcd/snapshot.db"

$ oc exec ${ETCD_POD} -c etcd -- /bin/bash -c "ETCDCTL_API=3 etcdctl \
    --cert /etc/etcd/peer.crt \
    --key /etc/etcd/peer.key \
    --cacert /etc/etcd/ca.crt \
    --endpoints $ETCD_EP \
    snapshot save /var/lib/etcd/snapshot.db"

Copy to Clipboard

Toggle word wrap

1: You must write the snapshot to a directory under /var/lib/etcd/.

5.2.1.3. Deprecating a master host
Copiar enlace

Master hosts run important services, such as the OpenShift Container Platform API and controllers services. In order to deprecate a master host, these services must be stopped.

The OpenShift Container Platform API service is an active/active service, so stopping the service does not affect the environment as long as the requests are sent to a separate master server. However, the OpenShift Container Platform controllers service is an active/passive service, where the services use etcd to decide the active master.

Deprecating a master host in a multi-master architecture includes removing the master from the load balancer pool to avoid new connections attempting to use that master. This process depends heavily on the load balancer used. The steps below show the details of removing the master from haproxy. In the event that OpenShift Container Platform is running on a cloud provider, or using a F5 appliance, see the specific product documents to remove the master from rotation.

Procedure

Remove the backend section in the /etc/haproxy/haproxy.cfg configuration file. For example, if deprecating a master named master-0.example.com using haproxy, ensure the host name is removed from the following:

backend mgmt8443
    balance source
    mode tcp
    # MASTERS 8443
    server master-1.example.com 192.168.55.12:8443 check
    server master-2.example.com 192.168.55.13:8443 check

backend mgmt8443
    balance source
    mode tcp
    # MASTERS 8443
    server master-1.example.com 192.168.55.12:8443 check
    server master-2.example.com 192.168.55.13:8443 check

Copy to Clipboard

Toggle word wrap

Then, restart the haproxy service.
```
sudo systemctl restart haproxy
```
```
$ sudo systemctl restart haproxy
```
Copy to Clipboard Toggle word wrap

Once the master is removed from the load balancer, disable the API and controller services by moving definition files out of the static pods dir /etc/origin/node/pods:

mkdir -p /etc/origin/node/pods/disabled
mv /etc/origin/node/pods/controller.yaml /etc/origin/node/pods/disabled/:

# mkdir -p /etc/origin/node/pods/disabled
# mv /etc/origin/node/pods/controller.yaml /etc/origin/node/pods/disabled/:
+

Copy to Clipboard

Toggle word wrap

Because the master host is a schedulable OpenShift Container Platform node, follow the steps in the Deprecating a node host section.
Remove the master host from the [masters] and [nodes] groups in the /etc/ansible/hosts Ansible inventory file to avoid issues if running any Ansible tasks using that inventory file.
Warning
Deprecating the first master host listed in the Ansible inventory file requires extra precautions.
The /etc/origin/master/ca.serial.txt file is generated on only the first master listed in the Ansible host inventory. If you deprecate the first master host, copy the /etc/origin/master/ca.serial.txt file to the rest of master hosts before the process.
Important
In OpenShift Container Platform 3.11 clusters running multiple masters, one of the master nodes includes additional CA certificates in /etc/origin/master, /etc/etcd/ca, and /etc/etcd/generated_certs. These are required for application node and etcd node scale-up operations and must be restored on another master node if the CA host master is being deprecated.

The kubernetes service includes the master host IPs as endpoints. To verify that the master has been properly deprecated, review the kubernetes service output and see if the deprecated master has been removed:

oc describe svc kubernetes -n default

$ oc describe svc kubernetes -n default
Name:			kubernetes
Namespace:		default
Labels:			component=apiserver
			provider=kubernetes
Annotations:		<none>
Selector:		<none>
Type:			ClusterIP
IP:			10.111.0.1
Port:			https	443/TCP
Endpoints:		192.168.55.12:8443,192.168.55.13:8443
Port:			dns	53/UDP
Endpoints:		192.168.55.12:8053,192.168.55.13:8053
Port:			dns-tcp	53/TCP
Endpoints:		192.168.55.12:8053,192.168.55.13:8053
Session Affinity:	ClientIP
Events:			<none>

Copy to Clipboard

Toggle word wrap

After the master has been successfully deprecated, the host where the master was previously running can be safely deleted.

5.2.1.4. Removing an etcd host
Copiar enlace

If an etcd host fails beyond restoration, remove it from the cluster.

Steps to be performed on all masters hosts

Procedure

Remove each other etcd host from the etcd cluster. Run the following command for each etcd node:

etcdctl3 --endpoints=https://<surviving host IP>:2379

# etcdctl3 --endpoints=https://<surviving host IP>:2379
  --cacert=/etc/etcd/ca.crt
  --cert=/etc/etcd/peer.crt
  --key=/etc/etcd/peer.key member remove <failed member ID>

Copy to Clipboard

Toggle word wrap

Restart the master API service on every master:
```
master-restart api restart-master controller
```
```
# master-restart api restart-master controller
```
Copy to Clipboard Toggle word wrap

Steps to be performed in the current etcd cluster

Procedure

Remove the failed host from the cluster:

etcdctl2 cluster-health
etcdctl2 member remove 8372784203e11288
etcdctl2 cluster-health

# etcdctl2 cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
failed to check the health of member 8372784203e11288 on https://192.168.55.21:2379: Get https://192.168.55.21:2379/health: dial tcp 192.168.55.21:2379: getsockopt: connection refused
member 8372784203e11288 is unreachable: [https://192.168.55.21:2379] are all unreachable
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy

# etcdctl2 member remove 8372784203e11288


Removed member 8372784203e11288 from cluster

# etcdctl2 cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy

Copy to Clipboard

Toggle word wrap

1: The remove command requires the etcd ID, not the hostname.

To ensure the etcd configuration does not use the failed host when the etcd service is restarted, modify the /etc/etcd/etcd.conf file on all remaining etcd hosts and remove the failed host in the value for the ETCD_INITIAL_CLUSTER variable:

vi /etc/etcd/etcd.conf

# vi /etc/etcd/etcd.conf

Copy to Clipboard

Toggle word wrap

For example:

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380,master-2.example.com=https://192.168.55.13:2380

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380,master-2.example.com=https://192.168.55.13:2380

Copy to Clipboard

Toggle word wrap

becomes:

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380

Copy to Clipboard

Toggle word wrap

Note

Restarting the etcd services is not required, because the failed host is removed using etcdctl.

Modify the Ansible inventory file to reflect the current status of the cluster and to avoid issues when re-running a playbook:

[OSEv3:children]
masters
nodes
etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com

[OSEv3:children]
masters
nodes
etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com

Copy to Clipboard

Toggle word wrap

If you are using Flannel, modify the flanneld service configuration located at /etc/sysconfig/flanneld on every host and remove the etcd host:

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379

Copy to Clipboard

Toggle word wrap

Restart the flanneld service:
```
systemctl restart flanneld.service
```
```
# systemctl restart flanneld.service
```
Copy to Clipboard Toggle word wrap

5.2.2. Creating a master host backup
Copiar enlace

OpenShift Container Platform files

The master instances run important services, such as the API, controllers. The /etc/origin/master directory stores many important files:

The configuration, the API, controllers, services, and more
Certificates generated by the installation
All cloud provider-related configuration
Keys and other authentication files, such as htpasswd if you use htpasswd
And more

You can customize OpenShift Container Platform services, such as increasing the log level or using proxies. The configuration files are stored in the /etc/sysconfig directory.

Because the masters are also nodes, back up the entire /etc/origin directory.

Procedure

Important

You must perform the following steps on each master node.

Create a backup of the pod definitions, located here.
Create a backup of the master host configuration files:
```
MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
sudo cp -aR /etc/sysconfig/ ${MYBACKUPDIR}/etc/sysconfig/
```
```
$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
$ sudo cp -aR /etc/sysconfig/ ${MYBACKUPDIR}/etc/sysconfig/
```
Copy to Clipboard Toggle word wrap
Note
The master configuration file is/etc/origin/master/master-config.yaml.
Warning
The /etc/origin/master/ca.serial.txt file is generated on only the first master listed in the Ansible host inventory. If you deprecate the first master host, copy the /etc/origin/master/ca.serial.txt file to the rest of master hosts before the process.
Important
In OpenShift Container Platform 3.11 clusters running multiple masters, one of the master nodes includes additional CA certificates in /etc/origin/master, /etc/etcd/ca and /etc/etcd/generated_certs. These are required for application node and etcd node scale-up operations and would need to be restored on another master node should the originating master become permanently unavailable. These directories are included by default within the backup procedures documented here.

Other important files that need to be considered when planning a backup include:

Expand

File	Description
`/etc/cni/*`	Container Network Interface configuration (if used)
`/etc/sysconfig/iptables`	Where the `iptables` rules are stored
`/etc/sysconfig/docker-storage-setup`	The input file for `container-storage-setup` command
`/etc/sysconfig/docker`	The `docker` configuration file
`/etc/sysconfig/docker-network`	`docker` networking configuration (i.e. MTU)
`/etc/sysconfig/docker-storage`	`docker` storage configuration (generated by `container-storage-setup`)
`/etc/dnsmasq.conf`	Main configuration file for `dnsmasq`
`/etc/dnsmasq.d/*`	Different `dnsmasq` configuration files
`/etc/sysconfig/flanneld`	`flannel` configuration file (if used)
`/etc/pki/ca-trust/source/anchors/`	Certificates added to the system (i.e. for external registries)

Create a backup of those files:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    ${MYBACKUPDIR}/etc/sysconfig/
sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
$ sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    ${MYBACKUPDIR}/etc/sysconfig/
$ sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
$ sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

Copy to Clipboard

Toggle word wrap

If a package is accidentally removed or you need to resore a file that is included in an rpm package, having a list of rhel packages installed on the system can be useful.
Note
If you use Red Hat Satellite features, such as content views or the facts store, provide a proper mechanism to reinstall the missing packages and a historical data of packages installed in the systems.
To create a list of the current rhel packages installed in the system:
```
MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo mkdir -p ${MYBACKUPDIR}
rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt
```
```
$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}
$ rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt
```
Copy to Clipboard Toggle word wrap

If you used the previous steps, the following files are present in the backup directory:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'
etc/sysconfig/flanneld
etc/sysconfig/iptables
etc/sysconfig/docker-network
etc/sysconfig/docker-storage
etc/sysconfig/docker-storage-setup
etc/sysconfig/docker-storage-setup.rpmnew
etc/origin/master/ca.crt
etc/origin/master/ca.key
etc/origin/master/ca.serial.txt
etc/origin/master/ca-bundle.crt
etc/origin/master/master.proxy-client.crt
etc/origin/master/master.proxy-client.key
etc/origin/master/service-signer.crt
etc/origin/master/service-signer.key
etc/origin/master/serviceaccounts.private.key
etc/origin/master/serviceaccounts.public.key
etc/origin/master/openshift-master.crt
etc/origin/master/openshift-master.key
etc/origin/master/openshift-master.kubeconfig
etc/origin/master/master.server.crt
etc/origin/master/master.server.key
etc/origin/master/master.kubelet-client.crt
etc/origin/master/master.kubelet-client.key
etc/origin/master/admin.crt
etc/origin/master/admin.key
etc/origin/master/admin.kubeconfig
etc/origin/master/etcd.server.crt
etc/origin/master/etcd.server.key
etc/origin/master/master.etcd-client.key
etc/origin/master/master.etcd-client.csr
etc/origin/master/master.etcd-client.crt
etc/origin/master/master.etcd-ca.crt
etc/origin/master/policy.json
etc/origin/master/scheduler.json
etc/origin/master/htpasswd
etc/origin/master/session-secrets.yaml
etc/origin/master/openshift-router.crt
etc/origin/master/openshift-router.key
etc/origin/master/registry.crt
etc/origin/master/registry.key
etc/origin/master/master-config.yaml
etc/origin/generated-configs/master-master-1.example.com/master.server.crt
...[OUTPUT OMITTED]...
etc/origin/cloudprovider/openstack.conf
etc/origin/node/system:node:master-0.example.com.crt
etc/origin/node/system:node:master-0.example.com.key
etc/origin/node/ca.crt
etc/origin/node/system:node:master-0.example.com.kubeconfig
etc/origin/node/server.crt
etc/origin/node/server.key
etc/origin/node/node-dnsmasq.conf
etc/origin/node/resolv.conf
etc/origin/node/node-config.yaml
etc/origin/node/flannel.etcd-client.key
etc/origin/node/flannel.etcd-client.csr
etc/origin/node/flannel.etcd-client.crt
etc/origin/node/flannel.etcd-ca.crt
etc/pki/ca-trust/source/anchors/openshift-ca.crt
etc/pki/ca-trust/source/anchors/registry-ca.crt
etc/dnsmasq.conf
etc/dnsmasq.d/origin-dns.conf
etc/dnsmasq.d/origin-upstream-dns.conf
etc/dnsmasq.d/node-dnsmasq.conf
packages.txt

Copy to Clipboard

Toggle word wrap

If needed, you can compress the files to save space:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
sudo rm -Rf ${MYBACKUPDIR}

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
$ sudo rm -Rf ${MYBACKUPDIR}

Copy to Clipboard

Toggle word wrap

Note

The openshift-ansible-contrib script is not supported by Red Hat, but the reference architecture team performs testing to ensure the code operates as defined and is secure.

You can run the script on every master host with:

mkdir ~/git
cd ~/git
git clone https://github.com/openshift/openshift-ansible-contrib.git
cd openshift-ansible-contrib/reference-architecture/day2ops/scripts
./backup_master_node.sh -h

$ mkdir ~/git
$ cd ~/git
$ git clone https://github.com/openshift/openshift-ansible-contrib.git
$ cd openshift-ansible-contrib/reference-architecture/day2ops/scripts
$ ./backup_master_node.sh -h

Copy to Clipboard

Toggle word wrap

5.2.3. Restoring a master host backup
Copiar enlace

After creating a backup of important master host files, if they become corrupted or accidentally removed, you can restore the files by copying the files back to master, ensuring they contain the proper content, and restarting the affected services.

Procedure

Restore the /etc/origin/master/master-config.yaml file:

MYBACKUPDIR=*/backup/$(hostname)/$(date +%Y%m%d)*
cp /etc/origin/master/master-config.yaml /etc/origin/master/master-config.yaml.old
cp /backup/$(hostname)/$(date +%Y%m%d)/origin/master/master-config.yaml /etc/origin/master/master-config.yaml
master-restart api
master-restart controllers

# MYBACKUPDIR=*/backup/$(hostname)/$(date +%Y%m%d)*
# cp /etc/origin/master/master-config.yaml /etc/origin/master/master-config.yaml.old
# cp /backup/$(hostname)/$(date +%Y%m%d)/origin/master/master-config.yaml /etc/origin/master/master-config.yaml
# master-restart api
# master-restart controllers

Copy to Clipboard

Toggle word wrap

Warning

Restarting the master services can lead to downtime. However, you can remove the master host from the highly available load balancer pool, then perform the restore operation. Once the service has been properly restored, you can add the master host back to the load balancer pool.

Note

Perform a full reboot of the affected instance to restore the iptables configuration.

If you cannot restart OpenShift Container Platform because packages are missing, reinstall the packages.
1. Get the list of the current installed packages:
  $ rpm -qa | sort > /tmp/current_packages.txt
  Copy to Clipboard Toggle word wrap
2. View the differences between the package lists:
  $ diff /tmp/current_packages.txt ${MYBACKUPDIR}/packages.txt > ansible-2.4.0.0-5.el7.noarch
  Copy to Clipboard Toggle word wrap
3. Reinstall the missing packages:
  # yum reinstall -y <packages>
  1
  Copy to Clipboard Toggle word wrap
  1
  Replace <packages> with the packages that are different between the package lists.
Restore a system certificate by copying the certificate to the /etc/pki/ca-trust/source/anchors/ directory and execute update-ca-trust:
```
MYBACKUPDIR=*/backup/$(hostname)/$(date +%Y%m%d)*
sudo cp ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/<certificate> /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
```
```
$ MYBACKUPDIR=*/backup/$(hostname)/$(date +%Y%m%d)*
$ sudo cp ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/<certificate> /etc/pki/ca-trust/source/anchors/ 
```
1
```
$ sudo update-ca-trust
```
Copy to Clipboard Toggle word wrap
1
Replace <certificate> with the file name of the system certificate to restore.
Note
Always ensure the user ID and group ID are restored when the files are copied back, as well as the SELinux context.

5.3. Node host tasks
Copiar enlace

5.3.1. Deprecating a node host
Copiar enlace

The procedure is the same whether deprecating an infrastructure node or an application node.

Prerequisites

Ensure enough capacity is available to migrate the existing pods from the node set to be removed. Removing an infrastructure node is advised only when at least two more nodes will stay online after the infrastructure node is removed.

Procedure

List all available nodes to find the node to deprecate:

oc get nodes

$ oc get nodes
NAME                  STATUS                     AGE       VERSION
ocp-infra-node-b7pl   Ready                      23h       v1.6.1+5115d708d7
ocp-infra-node-p5zj   Ready                      23h       v1.6.1+5115d708d7
ocp-infra-node-rghb   Ready                      23h       v1.6.1+5115d708d7
ocp-master-dgf8       Ready,SchedulingDisabled   23h       v1.6.1+5115d708d7
ocp-master-q1v2       Ready,SchedulingDisabled   23h       v1.6.1+5115d708d7
ocp-master-vq70       Ready,SchedulingDisabled   23h       v1.6.1+5115d708d7
ocp-node-020m         Ready                      23h       v1.6.1+5115d708d7
ocp-node-7t5p         Ready                      23h       v1.6.1+5115d708d7
ocp-node-n0dd         Ready                      23h       v1.6.1+5115d708d7

Copy to Clipboard

Toggle word wrap

As an example, this topic deprecates the ocp-infra-node-b7pl infrastructure node.

Describe the node and its running services:

oc describe node ocp-infra-node-b7pl

$ oc describe node ocp-infra-node-b7pl
Name:			ocp-infra-node-b7pl
Role:
Labels:			beta.kubernetes.io/arch=amd64
			beta.kubernetes.io/instance-type=n1-standard-2
			beta.kubernetes.io/os=linux
			failure-domain.beta.kubernetes.io/region=europe-west3
			failure-domain.beta.kubernetes.io/zone=europe-west3-c
			kubernetes.io/hostname=ocp-infra-node-b7pl
			role=infra
Annotations:		volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:			<none>
CreationTimestamp:	Wed, 22 Nov 2017 09:36:36 -0500
Phase:
Conditions:
  ...
Addresses:		10.156.0.11,ocp-infra-node-b7pl
Capacity:
 cpu:		2
 memory:	7494480Ki
 pods:		20
Allocatable:
 cpu:		2
 memory:	7392080Ki
 pods:		20
System Info:
 Machine ID:			bc95ccf67d047f2ae42c67862c202e44
 System UUID:			9762CC3D-E23C-AB13-B8C5-FA16F0BCCE4C
 Boot ID:			ca8bf088-905d-4ec0-beec-8f89f4527ce4
 Kernel Version:		3.10.0-693.5.2.el7.x86_64
 OS Image:			Employee SKU
 Operating System:		linux
 Architecture:			amd64
 Container Runtime Version:	docker://1.12.6
 Kubelet Version:		v1.6.1+5115d708d7
 Kube-Proxy Version:		v1.6.1+5115d708d7
ExternalID:			437740049672994824
Non-terminated Pods:		(2 in total)
  Namespace			Name				CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ---------			----				------------	----------	---------------	-------------
  default			docker-registry-1-5szjs		100m (5%)	0 (0%)		256Mi (3%)0 (0%)
  default			router-1-vzlzq			100m (5%)	0 (0%)		256Mi (3%)0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  200m (10%)	0 (0%)		512Mi (7%)	0 (0%)
Events:		<none>

Copy to Clipboard

Toggle word wrap

The output above shows that the node is running two pods: router-1-vzlzq and docker-registry-1-5szjs. Two more infrastructure nodes are available to migrate these two pods.

Note

The cluster described above is a highly available cluster, this means both the router and docker-registry services are running on all infrastructure nodes.

Mark a node as unschedulable and evacuate all of its pods:

oc adm drain ocp-infra-node-b7pl --delete-local-data

$ oc adm drain ocp-infra-node-b7pl --delete-local-data
node "ocp-infra-node-b7pl" cordoned
WARNING: Deleting pods with local storage: docker-registry-1-5szjs
pod "docker-registry-1-5szjs" evicted
pod "router-1-vzlzq" evicted
node "ocp-infra-node-b7pl" drained

Copy to Clipboard

Toggle word wrap

If the pod has attached local storage (for example, EmptyDir), the --delete-local-data option must be provided. Generally, pods running in production should use the local storage only for temporary or cache files, but not for anything important or persistent. For regular storage, applications should use object storage or persistent volumes. In this case, the docker-registry pod’s local storage is empty, because the object storage is used instead to store the container images.

Note

The above operation deletes existing pods running on the node. Then, new pods are created according to the replication controller.

In general, every application should be deployed with a deployment configuration, which creates pods using the replication controller.

oc adm drain will not delete any bare pods (pods that are neither mirror pods nor managed by ReplicationController, ReplicaSet, DaemonSet, StatefulSet, or a job). To do so, the --force option is required. Be aware that the bare pods will not be recreated on other nodes and data may be lost during this operation.

The example below shows the output of the replication controller of the registry:

oc describe rc/docker-registry-1

$ oc describe rc/docker-registry-1
Name:		docker-registry-1
Namespace:	default
Selector:	deployment=docker-registry-1,deploymentconfig=docker-registry,docker-registry=default
Labels:		docker-registry=default
		openshift.io/deployment-config.name=docker-registry
Annotations: ...
Replicas:	3 current / 3 desired
Pods Status:	3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:		deployment=docker-registry-1
			deploymentconfig=docker-registry
			docker-registry=default
  Annotations:		openshift.io/deployment-config.latest-version=1
			openshift.io/deployment-config.name=docker-registry
			openshift.io/deployment.name=docker-registry-1
  Service Account:	registry
  Containers:
   registry:
    Image:	openshift3/ose-docker-registry:v3.6.173.0.49
    Port:	5000/TCP
    Requests:
      cpu:	100m
      memory:	256Mi
    Liveness:	http-get https://:5000/healthz delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:	http-get https://:5000/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
    Environment:
      REGISTRY_HTTP_ADDR:					:5000
      REGISTRY_HTTP_NET:					tcp
      REGISTRY_HTTP_SECRET:					tyGEnDZmc8dQfioP3WkNd5z+Xbdfy/JVXf/NLo3s/zE=
      REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA:	false
      REGISTRY_HTTP_TLS_KEY:					/etc/secrets/registry.key
      OPENSHIFT_DEFAULT_REGISTRY:				docker-registry.default.svc:5000
      REGISTRY_CONFIGURATION_PATH:				/etc/registry/config.yml
      REGISTRY_HTTP_TLS_CERTIFICATE:				/etc/secrets/registry.crt
    Mounts:
      /etc/registry from docker-config (rw)
      /etc/secrets from registry-certificates (rw)
      /registry from registry-storage (rw)
  Volumes:
   registry-storage:
    Type:	EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
   registry-certificates:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	registry-certificates
    Optional:	false
   docker-config:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	registry-config
    Optional:	false
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  49m		49m		1	replication-controller			Normal		SuccessfulCreate	Created pod: docker-registry-1-dprp5

Copy to Clipboard

Toggle word wrap

The event at the bottom of the output displays information about new pod creation. So, when listing all pods:

oc get pods

$ oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-dprp5    1/1       Running   0          52m
docker-registry-1-kr8jq    1/1       Running   0          1d
docker-registry-1-ncpl2    1/1       Running   0          1d
registry-console-1-g4nqg   1/1       Running   0          1d
router-1-2gshr             0/1       Pending   0          52m
router-1-85qm4             1/1       Running   0          1d
router-1-q5sr8             1/1       Running   0          1d

Copy to Clipboard

Toggle word wrap

The docker-registry-1-5szjs and router-1-vzlzq pods that were running on the now deprecated node are no longer available. Instead, two new pods have been created: docker-registry-1-dprp5 and router-1-2gshr. As shown above, the new router pod is router-1-2gshr, but is in the Pending state. This is because every node can be running only on one single router and is bound to the ports 80 and 443 of the host.
When observing the newly created registry pod, the example below shows that the pod has been created on the ocp-infra-node-rghb node, which is different from the deprecating node:
```
oc describe pod docker-registry-1-dprp5
```
```
$ oc describe pod docker-registry-1-dprp5
Name:			docker-registry-1-dprp5
Namespace:		default
Security Policy:	hostnetwork
Node:			ocp-infra-node-rghb/10.156.0.10
...
```
Copy to Clipboard Toggle word wrap
The only difference between deprecating the infrastructure and the application node is that once the infrastructure node is evacuated, and if there is no plan to replace that node, the services running on infrastructure nodes can be scaled down:
```
oc scale dc/router --replicas 2
oc scale dc/docker-registry --replicas 2
```
```
$ oc scale dc/router --replicas 2
deploymentconfig "router" scaled

$ oc scale dc/docker-registry --replicas 2
deploymentconfig "docker-registry" scaled
```
Copy to Clipboard Toggle word wrap

Now, every infrastructure node is running only one kind of each pod:

oc get pods
oc describe po/docker-registry-1-kr8jq | grep Node:
oc describe po/docker-registry-1-ncpl2 | grep Node:

$ oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-kr8jq    1/1       Running   0          1d
docker-registry-1-ncpl2    1/1       Running   0          1d
registry-console-1-g4nqg   1/1       Running   0          1d
router-1-85qm4             1/1       Running   0          1d
router-1-q5sr8             1/1       Running   0          1d

$ oc describe po/docker-registry-1-kr8jq | grep Node:
Node:			ocp-infra-node-p5zj/10.156.0.9

$ oc describe po/docker-registry-1-ncpl2 | grep Node:
Node:			ocp-infra-node-rghb/10.156.0.10

Copy to Clipboard

Toggle word wrap

Note

To provide a full highly available cluster, at least three infrastructure nodes should always be available.

To verify that the scheduling on the node is disabled:

oc get nodes

$ oc get nodes
NAME                  STATUS                     AGE       VERSION
ocp-infra-node-b7pl   Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-infra-node-p5zj   Ready                      1d        v1.6.1+5115d708d7
ocp-infra-node-rghb   Ready                      1d        v1.6.1+5115d708d7
ocp-master-dgf8       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-master-q1v2       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-master-vq70       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-node-020m         Ready                      1d        v1.6.1+5115d708d7
ocp-node-7t5p         Ready                      1d        v1.6.1+5115d708d7
ocp-node-n0dd         Ready                      1d        v1.6.1+5115d708d7

Copy to Clipboard

Toggle word wrap

And that the node does not contain any pods:

oc describe node ocp-infra-node-b7pl

$ oc describe node ocp-infra-node-b7pl
Name:			ocp-infra-node-b7pl
Role:
Labels:			beta.kubernetes.io/arch=amd64
			beta.kubernetes.io/instance-type=n1-standard-2
			beta.kubernetes.io/os=linux
			failure-domain.beta.kubernetes.io/region=europe-west3
			failure-domain.beta.kubernetes.io/zone=europe-west3-c
			kubernetes.io/hostname=ocp-infra-node-b7pl
			role=infra
Annotations:		volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:			<none>
CreationTimestamp:	Wed, 22 Nov 2017 09:36:36 -0500
Phase:
Conditions:
  ...
Addresses:		10.156.0.11,ocp-infra-node-b7pl
Capacity:
 cpu:		2
 memory:	7494480Ki
 pods:		20
Allocatable:
 cpu:		2
 memory:	7392080Ki
 pods:		20
System Info:
 Machine ID:			bc95ccf67d047f2ae42c67862c202e44
 System UUID:			9762CC3D-E23C-AB13-B8C5-FA16F0BCCE4C
 Boot ID:			ca8bf088-905d-4ec0-beec-8f89f4527ce4
 Kernel Version:		3.10.0-693.5.2.el7.x86_64
 OS Image:			Employee SKU
 Operating System:		linux
 Architecture:			amd64
 Container Runtime Version:	docker://1.12.6
 Kubelet Version:		v1.6.1+5115d708d7
 Kube-Proxy Version:		v1.6.1+5115d708d7
ExternalID:			437740049672994824
Non-terminated Pods:		(0 in total)
  Namespace			Name		CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ---------			----		------------	----------	---------------	-------------
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  0 (0%)	0 (0%)		0 (0%)		0 (0%)
Events:		<none>

Copy to Clipboard

Toggle word wrap

Remove the infrastructure instance from the backend section in the /etc/haproxy/haproxy.cfg configuration file:

backend router80
    balance source
    mode tcp
    server infra-1.example.com 192.168.55.12:80 check
    server infra-2.example.com 192.168.55.13:80 check

backend router443
    balance source
    mode tcp
    server infra-1.example.com 192.168.55.12:443 check
    server infra-2.example.com 192.168.55.13:443 check

backend router80
    balance source
    mode tcp
    server infra-1.example.com 192.168.55.12:80 check
    server infra-2.example.com 192.168.55.13:80 check

backend router443
    balance source
    mode tcp
    server infra-1.example.com 192.168.55.12:443 check
    server infra-2.example.com 192.168.55.13:443 check

Copy to Clipboard

Toggle word wrap

Then, restart the haproxy service.
```
sudo systemctl restart haproxy
```
```
$ sudo systemctl restart haproxy
```
Copy to Clipboard Toggle word wrap

Remove the node from the cluster after all pods are evicted with command:

oc delete node ocp-infra-node-b7pl

$ oc delete node ocp-infra-node-b7pl
node "ocp-infra-node-b7pl" deleted

Copy to Clipboard

Toggle word wrap

oc get nodes

$ oc get nodes
NAME                  STATUS                     AGE       VERSION
ocp-infra-node-p5zj   Ready                      1d        v1.6.1+5115d708d7
ocp-infra-node-rghb   Ready                      1d        v1.6.1+5115d708d7
ocp-master-dgf8       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-master-q1v2       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-master-vq70       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-node-020m         Ready                      1d        v1.6.1+5115d708d7
ocp-node-7t5p         Ready                      1d        v1.6.1+5115d708d7
ocp-node-n0dd         Ready                      1d        v1.6.1+5115d708d7

Copy to Clipboard

Toggle word wrap

Note

For more information on evacuating and draining pods or nodes, see Node maintenance section.

5.3.1.1. Replacing a node host
Copiar enlace

In the event that a node would need to be added in place of the deprecated node, follow the Adding hosts to an existing cluster section.

5.3.2. Creating a node host backup
Copiar enlace

Creating a backup of a node host is a different use case from backing up a master host. Because master hosts contain many important files, creating a backup is highly recommended. However, the nature of nodes is that anything special is replicated over the nodes in case of failover, and they typically do not contain data that is necessary to run an environment. If a backup of a node contains something necessary to run an environment, then a creating a backup is recommended.

The backup process is to be performed before any change to the infrastructure, such as a system update, upgrade, or any other significant modification. Backups should be performed on a regular basis to ensure the most recent data is available if a failure occurs.

OpenShift Container Platform files

Node instances run applications in the form of pods, which are based on containers. The /etc/origin/ and /etc/origin/node directories house important files, such as:

The configuration of the node services
Certificates generated by the installation
Cloud provider-related configuration
Keys and other authentication files, such as the dnsmasq configuration

The OpenShift Container Platform services can be customized to increase the log level, use proxies, and more, and the configuration files are stored in the /etc/sysconfig directory.

Procedure

Create a backup of the node configuration files:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
sudo cp -aR /etc/sysconfig/atomic-openshift-node ${MYBACKUPDIR}/etc/sysconfig/

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
$ sudo cp -aR /etc/sysconfig/atomic-openshift-node ${MYBACKUPDIR}/etc/sysconfig/

Copy to Clipboard

Toggle word wrap

OpenShift Container Platform uses specific files that must be taken into account when planning the backup policy, including:

Expand

File	Description
`/etc/cni/*`	Container Network Interface configuration (if used)
`/etc/sysconfig/iptables`	Where the `iptables` rules are stored
`/etc/sysconfig/docker-storage-setup`	The input file for `container-storage-setup` command
`/etc/sysconfig/docker`	The `docker` configuration file
`/etc/sysconfig/docker-network`	`docker` networking configuration (i.e. MTU)
`/etc/sysconfig/docker-storage`	`docker` storage configuration (generated by `container-storage-setup`)
`/etc/dnsmasq.conf`	Main configuration file for `dnsmasq`
`/etc/dnsmasq.d/*`	Different `dnsmasq` configuration files
`/etc/sysconfig/flanneld`	`flannel` configuration file (if used)
`/etc/pki/ca-trust/source/anchors/`	Certificates added to the system (i.e. for external registries)

To create those files:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    ${MYBACKUPDIR}/etc/sysconfig/
sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
$ sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    ${MYBACKUPDIR}/etc/sysconfig/
$ sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
$ sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

Copy to Clipboard

Toggle word wrap

If a package is accidentally removed, or a file included in an rpm package should be restored, having a list of rhel packages installed on the system can be useful.
Note
If using Red Hat Satellite features, such as content views or the facts store, provide a proper mechanism to reinstall the missing packages and a historical data of packages installed in the systems.
To create a list of the current rhel packages installed in the system:
```
MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo mkdir -p ${MYBACKUPDIR}
rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt
```
```
$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}
$ rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt
```
Copy to Clipboard Toggle word wrap

The following files should now be present in the backup directory:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'
etc/sysconfig/atomic-openshift-node
etc/sysconfig/flanneld
etc/sysconfig/iptables
etc/sysconfig/docker-network
etc/sysconfig/docker-storage
etc/sysconfig/docker-storage-setup
etc/sysconfig/docker-storage-setup.rpmnew
etc/origin/node/system:node:app-node-0.example.com.crt
etc/origin/node/system:node:app-node-0.example.com.key
etc/origin/node/ca.crt
etc/origin/node/system:node:app-node-0.example.com.kubeconfig
etc/origin/node/server.crt
etc/origin/node/server.key
etc/origin/node/node-dnsmasq.conf
etc/origin/node/resolv.conf
etc/origin/node/node-config.yaml
etc/origin/node/flannel.etcd-client.key
etc/origin/node/flannel.etcd-client.csr
etc/origin/node/flannel.etcd-client.crt
etc/origin/node/flannel.etcd-ca.crt
etc/origin/cloudprovider/openstack.conf
etc/pki/ca-trust/source/anchors/openshift-ca.crt
etc/pki/ca-trust/source/anchors/registry-ca.crt
etc/dnsmasq.conf
etc/dnsmasq.d/origin-dns.conf
etc/dnsmasq.d/origin-upstream-dns.conf
etc/dnsmasq.d/node-dnsmasq.conf
packages.txt

Copy to Clipboard

Toggle word wrap

If needed, the files can be compressed to save space:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
sudo rm -Rf ${MYBACKUPDIR}

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
$ sudo rm -Rf ${MYBACKUPDIR}

Copy to Clipboard

Toggle word wrap

To create any of these files from scratch, the openshift-ansible-contrib repository contains the backup_master_node.sh script, which performs the previous steps. The script creates a directory on the host running the script and copies all the files previously mentioned.

Note

The openshift-ansible-contrib script is not supported by Red Hat, but the reference architecture team performs testing to ensure the code operates as defined and is secure.

The script can be executed on every master host with:

mkdir ~/git
cd ~/git
git clone https://github.com/openshift/openshift-ansible-contrib.git
cd openshift-ansible-contrib/reference-architecture/day2ops/scripts
./backup_master_node.sh -h

$ mkdir ~/git
$ cd ~/git
$ git clone https://github.com/openshift/openshift-ansible-contrib.git
$ cd openshift-ansible-contrib/reference-architecture/day2ops/scripts
$ ./backup_master_node.sh -h

Copy to Clipboard

Toggle word wrap

5.3.3. Restoring a node host backup
Copiar enlace

After creating a backup of important node host files, if they become corrupted or accidentally removed, you can restore the file by copying back the file, ensuring it contains the proper content and restart the affected services.

Procedure

Restore the /etc/origin/node/node-config.yaml file:

MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
cp /etc/origin/node/node-config.yaml /etc/origin/node/node-config.yaml.old
cp /backup/$(hostname)/$(date +%Y%m%d)/etc/origin/node/node-config.yaml /etc/origin/node/node-config.yaml
reboot

# MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
# cp /etc/origin/node/node-config.yaml /etc/origin/node/node-config.yaml.old
# cp /backup/$(hostname)/$(date +%Y%m%d)/etc/origin/node/node-config.yaml /etc/origin/node/node-config.yaml
# reboot

Copy to Clipboard

Toggle word wrap

Warning

Restarting the services can lead to downtime. See Node maintenance, for tips on how to ease the process.

Note

Perform a full reboot of the affected instance to restore the iptables configuration.

If you cannot restart OpenShift Container Platform because packages are missing, reinstall the packages.
1. Get the list of the current installed packages:
  $ rpm -qa | sort > /tmp/current_packages.txt
  Copy to Clipboard Toggle word wrap
2. View the differences between the package lists:
  $ diff /tmp/current_packages.txt ${MYBACKUPDIR}/packages.txt > ansible-2.4.0.0-5.el7.noarch
  Copy to Clipboard Toggle word wrap
3. Reinstall the missing packages:
  # yum reinstall -y <packages>
  1
  Copy to Clipboard Toggle word wrap
  1
  Replace <packages> with the packages that are different between the package lists.
Restore a system certificate by copying the certificate to the /etc/pki/ca-trust/source/anchors/ directory and execute update-ca-trust:
```
MYBACKUPDIR=*/backup/$(hostname)/$(date +%Y%m%d)*
sudo cp ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/<certificate> /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
```
```
$ MYBACKUPDIR=*/backup/$(hostname)/$(date +%Y%m%d)*
$ sudo cp ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/<certificate> /etc/pki/ca-trust/source/anchors/
$ sudo update-ca-trust
```
Copy to Clipboard Toggle word wrap
Replace <certificate> with the file name of the system certificate to restore.
Note
Always ensure proper user ID and group ID are restored when the files are copied back, as well as the SELinux context.

5.3.4. Node maintenance and next steps
Copiar enlace

See Managing nodes or Managing pods topics for various node management options. These include:

A node can reserve a portion of its resources to be used by specific components. These include the kubelet, kube-proxy, Docker, or other remaining system components such as sshd and NetworkManager. See the Allocating node resources section in the Cluster Administrator guide for more information.

5.4. etcd tasks
Copiar enlace

5.4.1. etcd backup
Copiar enlace

etcd is the key value store for all object definitions, as well as the persistent master state. Other components watch for changes, then bring themselves into the desired state.

OpenShift Container Platform versions prior to 3.5 use etcd version 2 (v2), while 3.5 and later use version 3 (v3). The data model between the two versions of etcd is different. etcd v3 can use both the v2 and v3 data models, whereas etcd v2 can only use the v2 data model. In an etcd v3 server, the v2 and v3 data stores exist in parallel and are independent.

For both v2 and v3 operations, you can use the ETCDCTL_API environment variable to use the correct API:

etcdctl -v
ETCDCTL_API=3 etcdctl version

$ etcdctl -v
etcdctl version: 3.2.28
API version: 2

$ ETCDCTL_API=3 etcdctl version
etcdctl version: 3.2.28
API version: 3.2

Copy to Clipboard

Toggle word wrap

See Migrating etcd Data (v2 to v3) section in the OpenShift Container Platform 3.7 documentation for information about how to migrate to v3.

In OpenShift Container Platform version 3.10 and later, you can either install etcd on separate hosts or run it as a static pod on your master hosts. If you do not specify separate etcd hosts, etcd runs as a static pod on master hosts. Because of this difference, the backup process is different if you use static pods.

The etcd backup process is composed of two different procedures:

Configuration backup: Including the required etcd configuration and certificates
Data backup: Including both v2 and v3 data model.

You can perform the data backup process on any host that has connectivity to the etcd cluster, where the proper certificates are provided, and where the etcdctl tool is installed.

Note

The backup files must be copied to an external system, ideally outside the OpenShift Container Platform environment, and then encrypted.

Note that the etcd backup still has all the references to current storage volumes. When you restore etcd, OpenShift Container Platform starts launching the previous pods on nodes and reattaching the same storage. This process is no different than the process of when you remove a node from the cluster and add a new one back in its place. Anything attached to that node is reattached to the pods on whatever nodes they are rescheduled to.

5.4.1.1. Backing up etcd
Copiar enlace

When you back up etcd, you must back up both the etcd configuration files and the etcd data.

5.4.1.1.1. Backing up etcd configuration files
Copiar enlace

Procedure

For each etcd member of the cluster, back up the etcd configuration.

ssh master-0
mkdir -p /backup/etcd-config-$(date +%Y%m%d)/
cp -R /etc/etcd/ /backup/etcd-config-$(date +%Y%m%d)/

$ ssh master-0


# mkdir -p /backup/etcd-config-$(date +%Y%m%d)/
# cp -R /etc/etcd/ /backup/etcd-config-$(date +%Y%m%d)/

Copy to Clipboard

Toggle word wrap

1: Replace master-0 with the name of your etcd member.

Note

The certificates and configuration files on each etcd cluster member are unique.

5.4.1.1.2. Backing up etcd data
Copiar enlace

Prerequisites

Note

The OpenShift Container Platform installer creates aliases to avoid typing all the flags named etcdctl2 for etcd v2 tasks and etcdctl3 for etcd v3 tasks.

However, the etcdctl3 alias does not provide the full endpoint list to the etcdctl command, so you must specify the --endpoints option and list all the endpoints.

Before backing up etcd:

etcdctl binaries must be available or, in containerized installations, the rhel7/etcd container must be available.
Ensure that the OpenShift Container Platform API service is running.
Ensure connectivity with the etcd cluster (port 2379/tcp).
Ensure the proper certificates to connect to the etcd cluster.

Procedure

Note

Back up the etcd data:

Important

Clusters upgraded from previous versions of OpenShift Container Platform might contain v2 data stores. Back up all etcd data stores.

Obtain the etcd endpoint IP address from the static pod manifest:

export ETCD_POD_MANIFEST="/etc/origin/node/pods/etcd.yaml"

$ export ETCD_POD_MANIFEST="/etc/origin/node/pods/etcd.yaml"

Copy to Clipboard

Toggle word wrap

export ETCD_EP=$(grep https ${ETCD_POD_MANIFEST} | cut -d '/' -f3)

$ export ETCD_EP=$(grep https ${ETCD_POD_MANIFEST} | cut -d '/' -f3)

Copy to Clipboard

Toggle word wrap

Log in as an administrator:
```
oc login -u system:admin
```
```
$ oc login -u system:admin
```
Copy to Clipboard Toggle word wrap

Obtain the etcd pod name:

export ETCD_POD=$(oc get pods -n kube-system | grep -o -m 1 '^master-etcd\S*')

$ export ETCD_POD=$(oc get pods -n kube-system | grep -o -m 1 '^master-etcd\S*')

Copy to Clipboard

Toggle word wrap

Change to the kube-system project:
```
oc project kube-system
```
```
$ oc project kube-system
```
Copy to Clipboard Toggle word wrap

Take a snapshot of the etcd data in the pod and store it locally:

oc exec ${ETCD_POD} -c etcd -- /bin/bash -c "ETCDCTL_API=3 etcdctl \
    --cert /etc/etcd/peer.crt \
    --key /etc/etcd/peer.key \
    --cacert /etc/etcd/ca.crt \
    --endpoints $ETCD_EP \
    snapshot save /var/lib/etcd/snapshot.db"

$ oc exec ${ETCD_POD} -c etcd -- /bin/bash -c "ETCDCTL_API=3 etcdctl \
    --cert /etc/etcd/peer.crt \
    --key /etc/etcd/peer.key \
    --cacert /etc/etcd/ca.crt \
    --endpoints $ETCD_EP \
    snapshot save /var/lib/etcd/snapshot.db"

Copy to Clipboard

Toggle word wrap

1: You must write the snapshot to a directory under /var/lib/etcd/.

5.4.2. Restoring etcd
Copiar enlace

5.4.2.1. Restoring the etcd configuration file
Copiar enlace

If an etcd host has become corrupted and the /etc/etcd/etcd.conf file is lost, restore it using the following procedure:

Access your etcd host:
```
ssh master-0
```
```
$ ssh master-0 
```
1
Copy to Clipboard Toggle word wrap
1
Replace master-0 with the name of your etcd host.

Copy the backup etcd.conf file to /etc/etcd/:

cp /backup/etcd-config-<timestamp>/etcd/etcd.conf /etc/etcd/etcd.conf

# cp /backup/etcd-config-<timestamp>/etcd/etcd.conf /etc/etcd/etcd.conf

Copy to Clipboard

Toggle word wrap

Set the required permissions and selinux context on the file:
```
restorecon -RvF /etc/etcd/etcd.conf
```
```
# restorecon -RvF /etc/etcd/etcd.conf
```
Copy to Clipboard Toggle word wrap

In this example, the backup file is stored in the /backup/etcd-config-<timestamp>/etcd/etcd.conf path where it can be used as an external NFS share, S3 bucket, or other storage solution.

After the etcd configuration file is restored, you must restart the static pod. This is done after you restore the etcd data.

5.4.2.2. Restoring etcd data
Copiar enlace

Before restoring etcd on a static pod:

etcdctl binaries must be available or, in containerized installations, the rhel7/etcd container must be available.
You can install the etcdctl binary with the etcd package by running the following commands:
```
yum install etcd
```
```
# yum install etcd
```
Copy to Clipboard Toggle word wrap
The package also installs the systemd service. Disable and mask the service so that it does not run as a systemd service when etcd runs in static pod. By disabling and masking the service, you ensure that you do not accidentally start it and prevent it from automatically restarting when you reboot the system.
```
systemctl disable etcd.service
```
```
# systemctl disable etcd.service
```
Copy to Clipboard Toggle word wrap
```
systemctl mask etcd.service
```
```
# systemctl mask etcd.service
```
Copy to Clipboard Toggle word wrap

To restore etcd on a static pod:

If the pod is running, stop the etcd pod by moving the pod manifest YAML file to another directory:
```
mkdir -p /etc/origin/node/pods-stopped
```
```
# mkdir -p /etc/origin/node/pods-stopped
```
Copy to Clipboard Toggle word wrap
```
mv /etc/origin/node/pods/etcd.yaml /etc/origin/node/pods-stopped
```
```
# mv /etc/origin/node/pods/etcd.yaml /etc/origin/node/pods-stopped
```
Copy to Clipboard Toggle word wrap
Move all old data:
```
mv /var/lib/etcd /var/lib/etcd.old
```
```
# mv /var/lib/etcd /var/lib/etcd.old
```
Copy to Clipboard Toggle word wrap
You use the etcdctl to recreate the data in the node where you restore the pod.

Restore the etcd snapshot to the mount path for the etcd pod:

export ETCDCTL_API=3

# export ETCDCTL_API=3

Copy to Clipboard

Toggle word wrap

etcdctl snapshot restore /etc/etcd/backup/etcd/snapshot.db \
	 --data-dir /var/lib/etcd/ \
	 --name ip-172-18-3-48.ec2.internal \
	 --initial-cluster "ip-172-18-3-48.ec2.internal=https://172.18.3.48:2380" \
	 --initial-cluster-token "etcd-cluster-1" \
	 --initial-advertise-peer-urls https://172.18.3.48:2380 \
	 --skip-hash-check=true

# etcdctl snapshot restore /etc/etcd/backup/etcd/snapshot.db \
	 --data-dir /var/lib/etcd/ \
	 --name ip-172-18-3-48.ec2.internal \
	 --initial-cluster "ip-172-18-3-48.ec2.internal=https://172.18.3.48:2380" \
	 --initial-cluster-token "etcd-cluster-1" \
	 --initial-advertise-peer-urls https://172.18.3.48:2380 \
	 --skip-hash-check=true

Copy to Clipboard

Toggle word wrap

Obtain the appropriate values for your cluster from your backup etcd.conf file.

Set required permissions and selinux context on the data directory:
```
restorecon -RvF /var/lib/etcd/
```
```
# restorecon -RvF /var/lib/etcd/
```
Copy to Clipboard Toggle word wrap
Restart the etcd pod by moving the pod manifest YAML file to the required directory:
```
mv /etc/origin/node/pods-stopped/etcd.yaml /etc/origin/node/pods/
```
```
# mv /etc/origin/node/pods-stopped/etcd.yaml /etc/origin/node/pods/
```
Copy to Clipboard Toggle word wrap

5.4.3. Replacing an etcd host
Copiar enlace

To replace an etcd host, scale up the etcd cluster and then remove the host. This process ensures that you keep quorum if you lose an etcd host during the replacement procedure.

Warning

The etcd cluster must maintain a quorum during the replacement operation. This means that at least one host must be in operation at all times.

If the host replacement operation occurs while the etcd cluster maintains a quorum, cluster operations are usually not affected. If a large amount of etcd data must replicate, some operations might slow down.

Note

Before you start any procedure involving the etcd cluster, you must have a backup of the etcd data and configuration files so that you can restore the cluster if the procedure fails.

5.4.4. Scaling etcd
Copiar enlace

You can scale the etcd cluster vertically by adding more resources to the etcd hosts or horizontally by adding more etcd hosts.

Note

Due to the voting system etcd uses, the cluster must always contain an odd number of members.

Having a cluster with an odd number of etcd hosts can account for fault tolerance. Having an odd number of etcd hosts does not change the number needed for a quorum but increases the tolerance for failure. For example, with a cluster of three members, quorum is two, which leaves a failure tolerance of one. This ensures the cluster continues to operate if two of the members are healthy.

Having an in-production cluster of three etcd hosts is recommended.

The new host requires a fresh Red Hat Enterprise Linux version 7 dedicated host. The etcd storage should be located on an SSD disk to achieve maximum performance and on a dedicated disk mounted in /var/lib/etcd.

Prerequisites

Before you add a new etcd host, perform a backup of both etcd configuration and data to prevent data loss.

Check the current etcd cluster status to avoid adding new hosts to an unhealthy cluster. Run this command:

ETCDCTL_API=3 etcdctl --cert="/etc/etcd/peer.crt" \
          --key=/etc/etcd/peer.key \
          --cacert="/etc/etcd/ca.crt" \
          --endpoints="https://*master-0.example.com*:2379,\
            https://*master-1.example.com*:2379,\
            https://*master-2.example.com*:2379"

# ETCDCTL_API=3 etcdctl --cert="/etc/etcd/peer.crt" \
          --key=/etc/etcd/peer.key \
          --cacert="/etc/etcd/ca.crt" \
          --endpoints="https://*master-0.example.com*:2379,\
            https://*master-1.example.com*:2379,\
            https://*master-2.example.com*:2379"
            endpoint health
https://master-0.example.com:2379 is healthy: successfully committed proposal: took = 5.011358ms
https://master-1.example.com:2379 is healthy: successfully committed proposal: took = 1.305173ms
https://master-2.example.com:2379 is healthy: successfully committed proposal: took = 1.388772ms

Copy to Clipboard

Toggle word wrap

Before running the scaleup playbook, ensure the new host is registered to the proper Red Hat software channels:

subscription-manager register \
    --username=*<username>* --password=*<password>*
subscription-manager attach --pool=*<poolid>*
subscription-manager repos --disable="*"
subscription-manager repos \
    --enable=rhel-7-server-rpms \
    --enable=rhel-7-server-extras-rpms

# subscription-manager register \
    --username=*<username>* --password=*<password>*
# subscription-manager attach --pool=*<poolid>*
# subscription-manager repos --disable="*"
# subscription-manager repos \
    --enable=rhel-7-server-rpms \
    --enable=rhel-7-server-extras-rpms

Copy to Clipboard

Toggle word wrap

etcd is hosted in the rhel-7-server-extras-rpms software channel.

Make sure all unused etcd members are removed from the etcd cluster. This must be completed before running the scaleup playbook.

List the etcd members:

etcdctl --cert="/etc/etcd/peer.crt" --key="/etc/etcd/peer.key" \
  --cacert="/etc/etcd/ca.crt" --endpoints=ETCD_LISTEN_CLIENT_URLS member list -w table

# etcdctl --cert="/etc/etcd/peer.crt" --key="/etc/etcd/peer.key" \
  --cacert="/etc/etcd/ca.crt" --endpoints=ETCD_LISTEN_CLIENT_URLS member list -w table

Copy to Clipboard

Toggle word wrap

Copy the unused etcd member ID, if applicable.

Remove the unused member by specifying its ID in the following command:

etcdctl --cert="/etc/etcd/peer.crt" --key="/etc/etcd/peer.key" \
  --cacert="/etc/etcd/ca.crt" --endpoints=ETCD_LISTEN_CLIENT_URL member remove UNUSED_ETCD_MEMBER_ID

# etcdctl --cert="/etc/etcd/peer.crt" --key="/etc/etcd/peer.key" \
  --cacert="/etc/etcd/ca.crt" --endpoints=ETCD_LISTEN_CLIENT_URL member remove UNUSED_ETCD_MEMBER_ID

Copy to Clipboard

Toggle word wrap

Upgrade etcd and iptables on the current etcd nodes:
```
yum update etcd iptables-services
```
```
# yum update etcd iptables-services
```
Copy to Clipboard Toggle word wrap
Back up the /etc/etcd configuration for the etcd hosts.
If the new etcd members will also be OpenShift Container Platform nodes, add the desired number of hosts to the cluster.
The rest of this procedure assumes you added one host, but if you add multiple hosts, perform all steps on each host.

5.4.4.1. Adding a new etcd host using Ansible
Copiar enlace

Procedure

In the Ansible inventory file, create a new group named [new_etcd] and add the new host. Then, add the new_etcd group as a child of the [OSEv3] group:
```
[OSEv3:children]
masters
nodes
etcd
new_etcd 

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com
master-2.example.com

[new_etcd] 
etcd0.example.com 
```
```
[OSEv3:children]
masters
nodes
etcd
new_etcd 
```
1
```
... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com
master-2.example.com

[new_etcd] 
```
2
```
etcd0.example.com 
```
3
Copy to Clipboard Toggle word wrap
1 2 3
Add these lines.
Note
Replace the old etcd host entry with the new etcd host entry in the inventory file. While replacing the older etcd host, you must create a copy of /etc/etcd/ca/ directory. Alternatively, you can redeploy etcd ca and certs before scaling up the etcd hosts.
From the host that installed OpenShift Container Platform and hosts the Ansible inventory file, change to the playbook directory and run the etcd scaleup playbook:
```
cd /usr/share/ansible/openshift-ansible
ansible-playbook  playbooks/openshift-etcd/scaleup.yml
```
```
$ cd /usr/share/ansible/openshift-ansible
$ ansible-playbook  playbooks/openshift-etcd/scaleup.yml
```
Copy to Clipboard Toggle word wrap

After the playbook runs, modify the inventory file to reflect the current status by moving the new etcd host from the [new_etcd] group to the [etcd] group:

[OSEv3:children]
masters
nodes
etcd
new_etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com
master-2.example.com
etcd0.example.com

[OSEv3:children]
masters
nodes
etcd
new_etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com
master-2.example.com
etcd0.example.com

Copy to Clipboard

Toggle word wrap

If you use Flannel, modify the flanneld service configuration on every OpenShift Container Platform host, located at /etc/sysconfig/flanneld, to include the new etcd host:

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379

Copy to Clipboard

Toggle word wrap

Restart the flanneld service:
```
systemctl restart flanneld.service
```
```
# systemctl restart flanneld.service
```
Copy to Clipboard Toggle word wrap

5.4.4.2. Manually adding a new etcd host
Copiar enlace

If you do not run etcd as static pods on master nodes, you might need to add another etcd host.

Procedure

Modify the current etcd cluster

To create the etcd certificates, run the openssl command, replacing the values with those from your environment.

Create some environment variables:

export NEW_ETCD_HOSTNAME="*etcd0.example.com*"
export NEW_ETCD_IP="192.168.55.21"

export CN=$NEW_ETCD_HOSTNAME
export SAN="IP:${NEW_ETCD_IP}, DNS:${NEW_ETCD_HOSTNAME}"
export PREFIX="/etc/etcd/generated_certs/etcd-$CN/"
export OPENSSLCFG="/etc/etcd/ca/openssl.cnf"

export NEW_ETCD_HOSTNAME="*etcd0.example.com*"
export NEW_ETCD_IP="192.168.55.21"

export CN=$NEW_ETCD_HOSTNAME
export SAN="IP:${NEW_ETCD_IP}, DNS:${NEW_ETCD_HOSTNAME}"
export PREFIX="/etc/etcd/generated_certs/etcd-$CN/"
export OPENSSLCFG="/etc/etcd/ca/openssl.cnf"

Copy to Clipboard

Toggle word wrap

Note

The custom openssl extensions used as etcd_v3_ca_* include the $SAN environment variable as subjectAltName. See /etc/etcd/ca/openssl.cnf for more information.

Create the directory to store the configuration and certificates:
```
mkdir -p ${PREFIX}
```
```
# mkdir -p ${PREFIX}
```
Copy to Clipboard Toggle word wrap

Create the server certificate request and sign it: (server.csr and server.crt)

openssl req -new -config ${OPENSSLCFG} \
    -keyout ${PREFIX}server.key  \
    -out ${PREFIX}server.csr \
    -reqexts etcd_v3_req -batch -nodes \
    -subj /CN=$CN
openssl ca -name etcd_ca -config ${OPENSSLCFG} \
    -out ${PREFIX}server.crt \
    -in ${PREFIX}server.csr \
    -extensions etcd_v3_ca_server -batch

# openssl req -new -config ${OPENSSLCFG} \
    -keyout ${PREFIX}server.key  \
    -out ${PREFIX}server.csr \
    -reqexts etcd_v3_req -batch -nodes \
    -subj /CN=$CN

# openssl ca -name etcd_ca -config ${OPENSSLCFG} \
    -out ${PREFIX}server.crt \
    -in ${PREFIX}server.csr \
    -extensions etcd_v3_ca_server -batch

Copy to Clipboard

Toggle word wrap

Create the peer certificate request and sign it: (peer.csr and peer.crt)

openssl req -new -config ${OPENSSLCFG} \
    -keyout ${PREFIX}peer.key \
    -out ${PREFIX}peer.csr \
    -reqexts etcd_v3_req -batch -nodes \
    -subj /CN=$CN
openssl ca -name etcd_ca -config ${OPENSSLCFG} \
  -out ${PREFIX}peer.crt \
  -in ${PREFIX}peer.csr \
  -extensions etcd_v3_ca_peer -batch

# openssl req -new -config ${OPENSSLCFG} \
    -keyout ${PREFIX}peer.key \
    -out ${PREFIX}peer.csr \
    -reqexts etcd_v3_req -batch -nodes \
    -subj /CN=$CN

# openssl ca -name etcd_ca -config ${OPENSSLCFG} \
  -out ${PREFIX}peer.crt \
  -in ${PREFIX}peer.csr \
  -extensions etcd_v3_ca_peer -batch

Copy to Clipboard

Toggle word wrap

Copy the current etcd configuration and ca.crt files from the current node as examples to modify later:
```
cp /etc/etcd/etcd.conf ${PREFIX}
cp /etc/etcd/ca.crt ${PREFIX}
```
```
# cp /etc/etcd/etcd.conf ${PREFIX}
# cp /etc/etcd/ca.crt ${PREFIX}
```
Copy to Clipboard Toggle word wrap

While still on the surviving etcd host, add the new host to the cluster. To add additional etcd members to the cluster, you must first adjust the default localhost peer in the peerURLs value for the first member:

Get the member ID for the first member using the member list command:

etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \
    member list

# etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \


    member list

Copy to Clipboard

Toggle word wrap

1: Ensure that you specify the URLs of only active etcd members in the --peers parameter value.

Obtain the IP address where etcd listens for cluster peers:
```
ss -l4n | grep 2380
```
```
$ ss -l4n | grep 2380
```
Copy to Clipboard Toggle word wrap

Update the value of peerURLs using the etcdctl member update command by passing the member ID and IP address obtained from the previous steps:

etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \
    member update 511b7fb6cc0001 https://172.18.1.18:2380

# etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \
    member update 511b7fb6cc0001 https://172.18.1.18:2380

Copy to Clipboard

Toggle word wrap

Re-run the member list command and ensure the peer URLs no longer include localhost.

Add the new host to the etcd cluster. Note that the new host is not yet configured, so the status stays as unstarted until the you configure the new host.

Warning

You must add each member and bring it online one at a time. When you add each additional member to the cluster, you must adjust the peerURLs list for the current peers. The peerURLs list grows by one for each member added. The etcdctl member add command outputs the values that you must set in the etcd.conf file as you add each member, as described in the following instructions.

etcdctl -C https://${CURRENT_ETCD_HOST}:2379 \
  --ca-file=/etc/etcd/ca.crt     \
  --cert-file=/etc/etcd/peer.crt     \
  --key-file=/etc/etcd/peer.key member add ${NEW_ETCD_HOSTNAME} https://${NEW_ETCD_IP}:2380

# etcdctl -C https://${CURRENT_ETCD_HOST}:2379 \
  --ca-file=/etc/etcd/ca.crt     \
  --cert-file=/etc/etcd/peer.crt     \
  --key-file=/etc/etcd/peer.key member add ${NEW_ETCD_HOSTNAME} https://${NEW_ETCD_IP}:2380



Added member named 10.3.9.222 with ID 4e1db163a21d7651 to cluster

ETCD_NAME="<NEW_ETCD_HOSTNAME>"
ETCD_INITIAL_CLUSTER="<NEW_ETCD_HOSTNAME>=https://<NEW_HOST_IP>:2380,<CLUSTERMEMBER1_NAME>=https:/<CLUSTERMEMBER2_IP>:2380,<CLUSTERMEMBER2_NAME>=https:/<CLUSTERMEMBER2_IP>:2380,<CLUSTERMEMBER3_NAME>=https:/<CLUSTERMEMBER3_IP>:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Copy to Clipboard

Toggle word wrap

1: In this line, 10.3.9.222 is a label for the etcd member. You can specify the host name, IP address, or a simple name.

Update the sample ${PREFIX}/etcd.conf file.
1. Replace the following values with the values generated in the previous step:
  - ETCD_NAME
  - ETCD_INITIAL_CLUSTER
  - ETCD_INITIAL_CLUSTER_STATE
2. Modify the following variables with the new host IP from the output of the previous step. You can use ${NEW_ETCD_IP} as the value.
  ETCD_LISTEN_PEER_URLS ETCD_LISTEN_CLIENT_URLS ETCD_INITIAL_ADVERTISE_PEER_URLS ETCD_ADVERTISE_CLIENT_URLS
  Copy to Clipboard Toggle word wrap
3. If you previously used the member system as an etcd node, you must overwrite the current values in the /etc/etcd/etcd.conf file.
4. Check the file for syntax errors or missing IP addresses, otherwise the etcd service might fail:
  # vi ${PREFIX}/etcd.conf
  Copy to Clipboard Toggle word wrap
On the node that hosts the installation files, update the [etcd] hosts group in the /etc/ansible/hosts inventory file. Remove the old etcd hosts and add the new ones.

Create a tgz file that contains the certificates, the sample configuration file, and the ca and copy it to the new host:

tar -czvf /etc/etcd/generated_certs/${CN}.tgz -C ${PREFIX} .
scp /etc/etcd/generated_certs/${CN}.tgz ${CN}:/tmp/

# tar -czvf /etc/etcd/generated_certs/${CN}.tgz -C ${PREFIX} .
# scp /etc/etcd/generated_certs/${CN}.tgz ${CN}:/tmp/

Copy to Clipboard

Toggle word wrap

Modify the new etcd host

Install iptables-services to provide iptables utilities to open the required ports for etcd:
```
yum install -y iptables-services
```
```
# yum install -y iptables-services
```
Copy to Clipboard Toggle word wrap

Create the OS_FIREWALL_ALLOW firewall rules to allow etcd to communicate:

Port 2379/tcp for clients

Port 2380/tcp for peer communication

systemctl enable iptables.service --now
iptables -N OS_FIREWALL_ALLOW
iptables -t filter -I INPUT -j OS_FIREWALL_ALLOW
iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT
iptables-save | tee /etc/sysconfig/iptables

# systemctl enable iptables.service --now
# iptables -N OS_FIREWALL_ALLOW
# iptables -t filter -I INPUT -j OS_FIREWALL_ALLOW
# iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
# iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT
# iptables-save | tee /etc/sysconfig/iptables

Copy to Clipboard

Toggle word wrap

Note

In this example, a new chain OS_FIREWALL_ALLOW is created, which is the standard naming the OpenShift Container Platform installer uses for firewall rules.

Warning

If the environment is hosted in an IaaS environment, modify the security groups for the instance to allow incoming traffic to those ports as well.

Install etcd:
```
yum install -y etcd
```
```
# yum install -y etcd
```
Copy to Clipboard Toggle word wrap
Ensure version etcd-2.3.7-4.el7.x86_64 or greater is installed,

Ensure the etcd service is not running by removing the etcd pod definition:

mkdir -p /etc/origin/node/pods-stopped
mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/

# mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/

Copy to Clipboard

Toggle word wrap

Remove any etcd configuration and data:
```
rm -Rf /etc/etcd/*
rm -Rf /var/lib/etcd/*
```
```
# rm -Rf /etc/etcd/*
# rm -Rf /var/lib/etcd/*
```
Copy to Clipboard Toggle word wrap
Extract the certificates and configuration files:
```
tar xzvf /tmp/etcd0.example.com.tgz -C /etc/etcd/
```
```
# tar xzvf /tmp/etcd0.example.com.tgz -C /etc/etcd/
```
Copy to Clipboard Toggle word wrap
Start etcd on the new host:
```
systemctl enable etcd --now
```
```
# systemctl enable etcd --now
```
Copy to Clipboard Toggle word wrap

Verify that the host is part of the cluster and the current cluster health:

If you use the v2 etcd api, run the following command:

etcdctl --cert-file=/etc/etcd/peer.crt \
          --key-file=/etc/etcd/peer.key \
          --ca-file=/etc/etcd/ca.crt \
          --peers="https://*master-0.example.com*:2379,\
          https://*master-1.example.com*:2379,\
          https://*master-2.example.com*:2379,\
          https://*etcd0.example.com*:2379"\
          cluster-health

# etcdctl --cert-file=/etc/etcd/peer.crt \
          --key-file=/etc/etcd/peer.key \
          --ca-file=/etc/etcd/ca.crt \
          --peers="https://*master-0.example.com*:2379,\
          https://*master-1.example.com*:2379,\
          https://*master-2.example.com*:2379,\
          https://*etcd0.example.com*:2379"\
          cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
member 8b8904727bf526a5 is healthy: got healthy result from https://192.168.55.21:2379
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy

Copy to Clipboard

Toggle word wrap

If you use the v3 etcd api, run the following command:

ETCDCTL_API=3 etcdctl --cert="/etc/etcd/peer.crt" \
          --key=/etc/etcd/peer.key \
          --cacert="/etc/etcd/ca.crt" \
          --endpoints="https://*master-0.example.com*:2379,\
            https://*master-1.example.com*:2379,\
            https://*master-2.example.com*:2379,\
            https://*etcd0.example.com*:2379"\
            endpoint health

# ETCDCTL_API=3 etcdctl --cert="/etc/etcd/peer.crt" \
          --key=/etc/etcd/peer.key \
          --cacert="/etc/etcd/ca.crt" \
          --endpoints="https://*master-0.example.com*:2379,\
            https://*master-1.example.com*:2379,\
            https://*master-2.example.com*:2379,\
            https://*etcd0.example.com*:2379"\
            endpoint health
https://master-0.example.com:2379 is healthy: successfully committed proposal: took = 5.011358ms
https://master-1.example.com:2379 is healthy: successfully committed proposal: took = 1.305173ms
https://master-2.example.com:2379 is healthy: successfully committed proposal: took = 1.388772ms
https://etcd0.example.com:2379 is healthy: successfully committed proposal: took = 1.498829ms

Copy to Clipboard

Toggle word wrap

Modify each OpenShift Container Platform master

Modify the master configuration in the etcClientInfo section of the /etc/origin/master/master-config.yaml file on every master. Add the new etcd host to the list of the etcd servers OpenShift Container Platform uses to store the data, and remove any failed etcd hosts:

etcdClientInfo:
  ca: master.etcd-ca.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
    - https://master-0.example.com:2379
    - https://master-1.example.com:2379
    - https://master-2.example.com:2379
    - https://etcd0.example.com:2379

etcdClientInfo:
  ca: master.etcd-ca.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
    - https://master-0.example.com:2379
    - https://master-1.example.com:2379
    - https://master-2.example.com:2379
    - https://etcd0.example.com:2379

Copy to Clipboard

Toggle word wrap

Restart the master API service:
- On every master:
  # master-restart api # master-restart controllers
  Copy to Clipboard Toggle word wrap
  Warning
  The number of etcd nodes must be odd, so you must add at least two hosts.

If you use Flannel, modify the flanneld service configuration located at /etc/sysconfig/flanneld on every OpenShift Container Platform host to include the new etcd host:

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379

Copy to Clipboard

Toggle word wrap

Restart the flanneld service:
```
systemctl restart flanneld.service
```
```
# systemctl restart flanneld.service
```
Copy to Clipboard Toggle word wrap

5.4.5. Removing an etcd host
Copiar enlace

If an etcd host fails beyond restoration, remove it from the cluster.

Steps to be performed on all masters hosts

Procedure

Remove each other etcd host from the etcd cluster. Run the following command for each etcd node:

etcdctl3 --endpoints=https://<surviving host IP>:2379

# etcdctl3 --endpoints=https://<surviving host IP>:2379
  --cacert=/etc/etcd/ca.crt
  --cert=/etc/etcd/peer.crt
  --key=/etc/etcd/peer.key member remove <failed member ID>

Copy to Clipboard

Toggle word wrap

Restart the master API service on every master:
```
master-restart api restart-master controller
```
```
# master-restart api restart-master controller
```
Copy to Clipboard Toggle word wrap

Steps to be performed in the current etcd cluster

Procedure

Remove the failed host from the cluster:

etcdctl2 cluster-health
etcdctl2 member remove 8372784203e11288
etcdctl2 cluster-health

# etcdctl2 cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
failed to check the health of member 8372784203e11288 on https://192.168.55.21:2379: Get https://192.168.55.21:2379/health: dial tcp 192.168.55.21:2379: getsockopt: connection refused
member 8372784203e11288 is unreachable: [https://192.168.55.21:2379] are all unreachable
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy

# etcdctl2 member remove 8372784203e11288


Removed member 8372784203e11288 from cluster

# etcdctl2 cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy

Copy to Clipboard

Toggle word wrap

1: The remove command requires the etcd ID, not the hostname.

vi /etc/etcd/etcd.conf

# vi /etc/etcd/etcd.conf

Copy to Clipboard

Toggle word wrap

For example:

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380,master-2.example.com=https://192.168.55.13:2380

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380,master-2.example.com=https://192.168.55.13:2380

Copy to Clipboard

Toggle word wrap

becomes:

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380

Copy to Clipboard

Toggle word wrap

Note

Restarting the etcd services is not required, because the failed host is removed using etcdctl.

Modify the Ansible inventory file to reflect the current status of the cluster and to avoid issues when re-running a playbook:

[OSEv3:children]
masters
nodes
etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com

[OSEv3:children]
masters
nodes
etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com

Copy to Clipboard

Toggle word wrap

If you are using Flannel, modify the flanneld service configuration located at /etc/sysconfig/flanneld on every host and remove the etcd host:

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379

Copy to Clipboard

Toggle word wrap

Restart the flanneld service:
```
systemctl restart flanneld.service
```
```
# systemctl restart flanneld.service
```
Copy to Clipboard Toggle word wrap

Este contenido no está disponible en el idioma seleccionado.

Chapter 5. Host-level tasks

5.1. Adding a host to the clusterCopiar enlaceEnlace copiado en el portapapeles!

5.2. Master host tasksCopiar enlaceEnlace copiado en el portapapeles!

5.2.1. Deprecating a master hostCopiar enlaceEnlace copiado en el portapapeles!

5.2.1.1. Creating a master host backupCopiar enlaceEnlace copiado en el portapapeles!

Procedure

5.2.1.2. Backing up etcdCopiar enlaceEnlace copiado en el portapapeles!

5.2.1.2.1. Backing up etcd configuration filesCopiar enlaceEnlace copiado en el portapapeles!

Procedure

5.2.1.2.2. Backing up etcd dataCopiar enlaceEnlace copiado en el portapapeles!

Prerequisites

Procedure

5.2.1.3. Deprecating a master hostCopiar enlaceEnlace copiado en el portapapeles!

Procedure

5.2.1.4. Removing an etcd hostCopiar enlaceEnlace copiado en el portapapeles!

Procedure

Procedure

5.2.2. Creating a master host backupCopiar enlaceEnlace copiado en el portapapeles!

Procedure

5.2.3. Restoring a master host backupCopiar enlaceEnlace copiado en el portapapeles!

Procedure

5.3. Node host tasksCopiar enlaceEnlace copiado en el portapapeles!

5.3.1. Deprecating a node hostCopiar enlaceEnlace copiado en el portapapeles!

Prerequisites

Procedure

5.3.1.1. Replacing a node hostCopiar enlaceEnlace copiado en el portapapeles!

5.3.2. Creating a node host backupCopiar enlaceEnlace copiado en el portapapeles!

Procedure

5.3.3. Restoring a node host backupCopiar enlaceEnlace copiado en el portapapeles!

Procedure

5.3.4. Node maintenance and next stepsCopiar enlaceEnlace copiado en el portapapeles!

5.4. etcd tasksCopiar enlaceEnlace copiado en el portapapeles!

5.4.1. etcd backupCopiar enlaceEnlace copiado en el portapapeles!

5.4.1.1. Backing up etcdCopiar enlaceEnlace copiado en el portapapeles!

5.4.1.1.1. Backing up etcd configuration filesCopiar enlaceEnlace copiado en el portapapeles!

Procedure

5.4.1.1.2. Backing up etcd dataCopiar enlaceEnlace copiado en el portapapeles!

Prerequisites

Procedure

5.4.2. Restoring etcdCopiar enlaceEnlace copiado en el portapapeles!

5.4.2.1. Restoring the etcd configuration fileCopiar enlaceEnlace copiado en el portapapeles!

5.4.2.2. Restoring etcd dataCopiar enlaceEnlace copiado en el portapapeles!

5.4.3. Replacing an etcd hostCopiar enlaceEnlace copiado en el portapapeles!

5.4.4. Scaling etcdCopiar enlaceEnlace copiado en el portapapeles!

Prerequisites

5.4.4.1. Adding a new etcd host using AnsibleCopiar enlaceEnlace copiado en el portapapeles!

Procedure

5.4.4.2. Manually adding a new etcd hostCopiar enlaceEnlace copiado en el portapapeles!

Procedure

Modify the current etcd cluster

Modify the new etcd host

Modify each OpenShift Container Platform master

5.4.5. Removing an etcd hostCopiar enlaceEnlace copiado en el portapapeles!

Procedure

Procedure

Aprender

Pruebe, compre y venda

Comunidades

Acerca de la documentación de Red Hat

Hacer que el código abierto sea más inclusivo

Acerca de Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

5.1. Adding a host to the cluster
Copiar enlace

5.2. Master host tasks
Copiar enlace

5.2.1. Deprecating a master host
Copiar enlace

5.2.1.1. Creating a master host backup
Copiar enlace

5.2.1.2. Backing up etcd
Copiar enlace

5.2.1.2.1. Backing up etcd configuration files
Copiar enlace

5.2.1.2.2. Backing up etcd data
Copiar enlace

5.2.1.3. Deprecating a master host
Copiar enlace

5.2.1.4. Removing an etcd host
Copiar enlace

5.2.2. Creating a master host backup
Copiar enlace

5.2.3. Restoring a master host backup
Copiar enlace

5.3. Node host tasks
Copiar enlace

5.3.1. Deprecating a node host
Copiar enlace

5.3.1.1. Replacing a node host
Copiar enlace

5.3.2. Creating a node host backup
Copiar enlace

5.3.3. Restoring a node host backup
Copiar enlace

5.3.4. Node maintenance and next steps
Copiar enlace

5.4. etcd tasks
Copiar enlace

5.4.1. etcd backup
Copiar enlace

5.4.1.1. Backing up etcd
Copiar enlace

5.4.1.1.1. Backing up etcd configuration files
Copiar enlace

5.4.1.1.2. Backing up etcd data
Copiar enlace

5.4.2. Restoring etcd
Copiar enlace

5.4.2.1. Restoring the etcd configuration file
Copiar enlace

5.4.2.2. Restoring etcd data
Copiar enlace

5.4.3. Replacing an etcd host
Copiar enlace

5.4.4. Scaling etcd
Copiar enlace

5.4.4.1. Adding a new etcd host using Ansible
Copiar enlace

5.4.4.2. Manually adding a new etcd host
Copiar enlace

5.4.5. Removing an etcd host
Copiar enlace