Chapter 7. Migrating etcd Data (v2 to v3)

Stop all master services:

# systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers

Ensure that the etcd cluster is healthy by running the following command:

# etcdctl --ca-file=/etc/etcd/master.etcd-ca.crt  \
--cert-file=/etc/etcd/master.etcd-client.crt \
--key-file=/etc/etcd/master.etcd-client.key \
--endpoints https://etcd-test-1:2379,https://etcd-test-2:2379,https://etcd-test-3:2379 \
cluster-health

Before the migration can proceed, the etcd cluster must be healthy.

The output for a healthy cluster is similar to the following:

member 2a3d833935d9d076 is healthy: got healthy result from https://etcd-test-1:2379
member a83a3258059fee18 is healthy: got healthy result from https://etcd-test-2:2379
member 22a9f2ddf18fee5f is healthy: got healthy result from https://etcd-test-3:2379
cluster is healthy

Ensure the raft index value of the etcd members does not vary by more than one. To check the raft indexes run the following command:

# ETCDCTL_API=3 etcdctl --cacert="/etc/etcd/master.etcd-ca.crt" \
--cert="/etc/etcd/master.etcd-client.crt" \
--key="/etc/etcd/master.etcd-client.key" \
--endpoints <etcd-endpoint>,<etcd-endpoint>,<etcd-endpoint>\ 1
-w table endpoint status

--endpoints https://etcd-test-1:2379,https://etcd-test-2:2379,https://etcd-test-3:2379

The output is similar to the following:

+------------------+------------------+---------+---------+-----------+-----------+------------+
|     ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------+------------------+---------+---------+-----------+-----------+------------+
| etcd-test-1:2379 | 2a3d833935d9d076 | 3.1.9   | 25 kB   | false     |       415 |        995 |
| etcd-test-2:2379 | a83a3258059fee18 | 3.1.9   | 25 kB   | true      |       415 |        994 |
| etcd-test-3:2379 | 22a9f2ddf18fee5f | 3.1.9   | 25 kB   | false     |       415 |        995 |
+------------------+------------------+---------+---------+-----------+-----------+------------+

Here, the indexes of 995, 994, and 995 are considered converged and you can proceed with the migration. If the minimum and maximum of raft indexes over all etcd members differ by more than one unit, wait a minute and try the command again.

Stop etcd on all etcd hosts:

# systemctl stop etcd

Run the following command (with the etcd daemon stopped) on your first etcd host to perform the migration:

# ETCDCTL_API=3 etcdctl migrate --data-dir=/var/lib/etcd

The --data-dir target can in a different location depending on the deployment. For example, embedded etcd operates over the /var/lib/origin/openshift.local.etcd directory, and etcd run as a system container operates over the /var/lib/etcd/etcd.etcd directory.

When complete, the migration responds with the following message if successful:

finished transforming keys

If there is no v2 data, it responds with:

no v2 keys to migrate

On each remaining etcd host, move the existing member directory to a backup location:

$ mv /var/lib/etcd/member /var/lib/etcd/member.old

Create a new cluster on the first host:

# echo "ETCD_FORCE_NEW_CLUSTER=true" >> /etc/etcd/etcd.conf
# systemctl start etcd
# sed -i '/ETCD_FORCE_NEW_CLUSTER=true/d' /etc/etcd/etcd.conf
# systemctl restart etcd

When the etcdctl migrate command is run without the --no-ttl option, TTL keys are migrated as well. Given that the TTL keys in v2 data are replaced with leases in v3 data, you must attach leases to all migrated TTL keys (with the etcd daemon running).

After your etcd cluster is back online with all members, re-introduce the TTL information by running the following on the first master:

$ oc adm migrate etcd-ttl --etcd-address=https://<ip_address>:2379 \
    --cacert=/etc/etcd/master.etcd-ca.crt \
    --cert=/etc/etcd/master.etcd-client.crt \
    --key=/etc/etcd/master.etcd-client.key \
    --ttl-keys-prefix '/kubernetes.io/events' \
    --lease-duration 1h
$ oc adm migrate etcd-ttl --etcd-address=https://<ip_address>:2379 \
    --cacert=/etc/etcd/master.etcd-ca.crt \
    --cert=/etc/etcd/master.etcd-client.crt \
    --key=/etc/etcd/master.etcd-client.key \
    --ttl-keys-prefix '/kubernetes.io/masterleases' \
    --lease-duration 10s
$ oc adm migrate etcd-ttl --etcd-address=https://<ip_address>:2379 \
    --cacert=/etc/etcd/master.etcd-ca.crt \
    --cert=/etc/etcd/master.etcd-client.crt \
    --key=/etc/etcd/master.etcd-client.key \
    --ttl-keys-prefix '/openshift.io/oauth/accesstokens' \
    --lease-duration 86400s
$ oc adm migrate etcd-ttl --etcd-address=https://<ip_address>:2379 \
    --cacert=/etc/etcd/master.etcd-ca.crt \
    --cert=/etc/etcd/master.etcd-client.crt \
    --key=/etc/etcd/master.etcd-client.key \
    --ttl-keys-prefix '/openshift.io/oauth/authorizetokens' \
    --lease-duration 500s
$ oc adm migrate etcd-ttl --etcd-address=https://<ip_address>:2379 \
    --cacert=/etc/etcd/master.etcd-ca.crt \
    --cert=/etc/etcd/master.etcd-client.crt \
    --key=/etc/etcd/master.etcd-client.key \
    --ttl-keys-prefix '/openshift.io/leases/controllers' \
    --lease-duration 10s

After the migration is complete, update the master configuration file, by default /etc/etcd/master-config.yaml, so the master daemons can use the new storage back end:

kubernetesMasterConfig:
  apiServerArguments:
    storage-backend:
    - etcd3
    storage-media-type:
    - application/vnd.kubernetes.protobuf

Restart your services, run:

# systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers