Day Two Operations Guide

Create a new project:

$ oc new-project sdn-test

Deploy an httpd application:

$ oc new-app centos/httpd-24-centos7~https://github.com/sclorg/httpd-ex

Wait until the build is complete:

$ oc get pods
NAME               READY     STATUS      RESTARTS   AGE
httpd-ex-1-205hz   1/1       Running     0          34s
httpd-ex-1-build   0/1       Completed   0          1m

Connect to the running pod:

$ oc rsh po/<pod-name>

For example:

$ oc rsh po/httpd-ex-1-205hz

Check the healthz path of the internal registry service:

$ curl -kv https://docker-registry.default.svc.cluster.local:5000/healthz
* About to connect() to docker-registry.default.svc.cluster.locl port 5000 (#0)
*   Trying 172.30.150.7...
* Connected to docker-registry.default.svc.cluster.local (172.30.150.7) port 5000 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=172.30.150.7
* 	start date: Nov 30 17:21:51 2017 GMT
* 	expire date: Nov 30 17:21:52 2019 GMT
* 	common name: 172.30.150.7
* 	issuer: CN=openshift-signer@1512059618
> GET /healthz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: docker-registry.default.svc.cluster.local:5000
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Date: Mon, 04 Dec 2017 16:26:49 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host docker-registry.default.svc.cluster.local left intact

sh-4.2$ *exit*

The HTTP/1.1 200 OK response means the node is correctly connecting.

Clean up the test project:

$ oc delete project sdn-test
project "sdn-test" deleted

The node host is listening on TCP port 10250. This port needs to be reachable by all master hosts on any node, and if monitoring is deployed in the cluster, the infrastructure nodes must have access to this port on all instances as well. Broken communication on this port can be detected with the following command:

$ oc get nodes
NAME                  STATUS                     AGE       VERSION
ocp-infra-node-1clj   Ready                      4d        v1.6.1+5115d708d7
ocp-infra-node-86qr   Ready                      4d        v1.6.1+5115d708d7
ocp-infra-node-g8qw   Ready                      4d        v1.6.1+5115d708d7
ocp-master-94zd       Ready,SchedulingDisabled   4d        v1.6.1+5115d708d7
ocp-master-gjkm       Ready,SchedulingDisabled   4d        v1.6.1+5115d708d7
ocp-master-wc8w       Ready,SchedulingDisabled   4d        v1.6.1+5115d708d7
ocp-node-c5dg         Ready                      4d        v1.6.1+5115d708d7
ocp-node-ghxn         Ready                      4d        v1.6.1+5115d708d7
ocp-node-w135         NotReady                   4d        v1.6.1+5115d708d7

In the output above, the node service on the ocp-node-w135 node is not reachable by the master services, which is represented by its NotReady status.

The last service is the router, which is responsible for routing connections to the correct services running in the OpenShift Container Platform cluster. Routers listen on TCP ports 80 and 443 on infrastructure nodes for ingress traffic. Before routers can start working, DNS must be configured:

$ dig *.apps.example.com

; <<>> DiG 9.11.1-P3-RedHat-9.11.1-8.P3.fc27 <<>> *.apps.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45790
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;*.apps.example.com.	IN	A

;; ANSWER SECTION:
*.apps.example.com. 3571	IN	CNAME	apps.example.com.
apps.example.com.	3561	IN	A	35.xx.xx.92

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Dec 05 16:03:52 CET 2017
;; MSG SIZE  rcvd: 105

The IP address, in this case 35.xx.xx.92, should be pointing to the load balancer distributing ingress traffic to all infrastructure nodes. To verify the functionality of the routers, check the registry service once more, but this time from outside the cluster:

$ curl -kv https://docker-registry-default.apps.example.com/healthz
*   Trying 35.xx.xx.92...
* TCP_NODELAY set
* Connected to docker-registry-default.apps.example.com (35.xx.xx.92) port 443 (#0)
...
< HTTP/2 200
< cache-control: no-cache
< content-type: text/plain; charset=utf-8
< content-length: 0
< date: Tue, 05 Dec 2017 15:13:27 GMT
<
* Connection #0 to host docker-registry-default.apps.example.com left intact

Create a backup of the master host configuration files:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
$ sudo cp -aR /etc/sysconfig/ ${MYBACKUPDIR}/etc/sysconfig/

Other important files that need to be considered when planning a backup include:

Create a backup of those files:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
$ sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    ${MYBACKUPDIR}/etc/sysconfig/
$ sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
$ sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

If a package is accidentally removed or you need to resore a file that is included in an rpm package, having a list of rhel packages installed on the system can be useful.

To create a list of the current rhel packages installed in the system:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}
$ rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt

If you used the previous steps, the following files are present in the backup directory:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'
etc/sysconfig/flanneld
etc/sysconfig/iptables
etc/sysconfig/docker-network
etc/sysconfig/docker-storage
etc/sysconfig/docker-storage-setup
etc/sysconfig/docker-storage-setup.rpmnew
etc/origin/master/ca.crt
etc/origin/master/ca.key
etc/origin/master/ca.serial.txt
etc/origin/master/ca-bundle.crt
etc/origin/master/master.proxy-client.crt
etc/origin/master/master.proxy-client.key
etc/origin/master/service-signer.crt
etc/origin/master/service-signer.key
etc/origin/master/serviceaccounts.private.key
etc/origin/master/serviceaccounts.public.key
etc/origin/master/openshift-master.crt
etc/origin/master/openshift-master.key
etc/origin/master/openshift-master.kubeconfig
etc/origin/master/master.server.crt
etc/origin/master/master.server.key
etc/origin/master/master.kubelet-client.crt
etc/origin/master/master.kubelet-client.key
etc/origin/master/admin.crt
etc/origin/master/admin.key
etc/origin/master/admin.kubeconfig
etc/origin/master/etcd.server.crt
etc/origin/master/etcd.server.key
etc/origin/master/master.etcd-client.key
etc/origin/master/master.etcd-client.csr
etc/origin/master/master.etcd-client.crt
etc/origin/master/master.etcd-ca.crt
etc/origin/master/policy.json
etc/origin/master/scheduler.json
etc/origin/master/htpasswd
etc/origin/master/session-secrets.yaml
etc/origin/master/openshift-router.crt
etc/origin/master/openshift-router.key
etc/origin/master/registry.crt
etc/origin/master/registry.key
etc/origin/master/master-config.yaml
etc/origin/generated-configs/master-master-1.example.com/master.server.crt
...[OUTPUT OMITTED]...
etc/origin/cloudprovider/openstack.conf
etc/origin/node/system:node:master-0.example.com.crt
etc/origin/node/system:node:master-0.example.com.key
etc/origin/node/ca.crt
etc/origin/node/system:node:master-0.example.com.kubeconfig
etc/origin/node/server.crt
etc/origin/node/server.key
etc/origin/node/node-dnsmasq.conf
etc/origin/node/resolv.conf
etc/origin/node/node-config.yaml
etc/origin/node/flannel.etcd-client.key
etc/origin/node/flannel.etcd-client.csr
etc/origin/node/flannel.etcd-client.crt
etc/origin/node/flannel.etcd-ca.crt
etc/pki/ca-trust/source/anchors/openshift-ca.crt
etc/pki/ca-trust/source/anchors/registry-ca.crt
etc/dnsmasq.conf
etc/dnsmasq.d/origin-dns.conf
etc/dnsmasq.d/origin-upstream-dns.conf
etc/dnsmasq.d/node-dnsmasq.conf
packages.txt

If needed, you can compress the files to save space:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
$ sudo rm -Rf ${MYBACKUPDIR}

Restore the /etc/origin/master/master-config.yaml file:

# MYBACKUPDIR=*/backup/$(hostname)/$(date +%Y%m%d)*
# cp /etc/origin/master/master-config.yaml /etc/origin/master/master-config.yaml.old
# cp /backup/$(hostname)/$(date +%Y%m%d)/origin/master/master-config.yaml /etc/origin/master/master-config.yaml
# master-restart api
# master-restart controllers

If you cannot restart OpenShift Container Platform because packages are missing, reinstall the packages.

Restore a system certificate by copying the certificate to the /etc/pki/ca-trust/source/anchors/ directory and execute the update-ca-trust:

$ MYBACKUPDIR=*/backup/$(hostname)/$(date +%Y%m%d)*
$ sudo cp ${MYBACKUPDIR}/external_certificates/my_company.crt /etc/pki/ca-trust/source/anchors/
$ sudo update-ca-trust

List all available nodes to find the node to deprecate:

$ oc get nodes
NAME                  STATUS                     AGE       VERSION
ocp-infra-node-b7pl   Ready                      23h       v1.6.1+5115d708d7
ocp-infra-node-p5zj   Ready                      23h       v1.6.1+5115d708d7
ocp-infra-node-rghb   Ready                      23h       v1.6.1+5115d708d7
ocp-master-dgf8       Ready,SchedulingDisabled   23h       v1.6.1+5115d708d7
ocp-master-q1v2       Ready,SchedulingDisabled   23h       v1.6.1+5115d708d7
ocp-master-vq70       Ready,SchedulingDisabled   23h       v1.6.1+5115d708d7
ocp-node-020m         Ready                      23h       v1.6.1+5115d708d7
ocp-node-7t5p         Ready                      23h       v1.6.1+5115d708d7
ocp-node-n0dd         Ready                      23h       v1.6.1+5115d708d7

As an example, this topic deprecates the ocp-infra-node-b7pl infrastructure node.

Describe the node and its running services:

$ oc describe node ocp-infra-node-b7pl
Name:			ocp-infra-node-b7pl
Role:
Labels:			beta.kubernetes.io/arch=amd64
			beta.kubernetes.io/instance-type=n1-standard-2
			beta.kubernetes.io/os=linux
			failure-domain.beta.kubernetes.io/region=europe-west3
			failure-domain.beta.kubernetes.io/zone=europe-west3-c
			kubernetes.io/hostname=ocp-infra-node-b7pl
			role=infra
Annotations:		volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:			<none>
CreationTimestamp:	Wed, 22 Nov 2017 09:36:36 -0500
Phase:
Conditions:
  ...
Addresses:		10.156.0.11,ocp-infra-node-b7pl
Capacity:
 cpu:		2
 memory:	7494480Ki
 pods:		20
Allocatable:
 cpu:		2
 memory:	7392080Ki
 pods:		20
System Info:
 Machine ID:			bc95ccf67d047f2ae42c67862c202e44
 System UUID:			9762CC3D-E23C-AB13-B8C5-FA16F0BCCE4C
 Boot ID:			ca8bf088-905d-4ec0-beec-8f89f4527ce4
 Kernel Version:		3.10.0-693.5.2.el7.x86_64
 OS Image:			Employee SKU
 Operating System:		linux
 Architecture:			amd64
 Container Runtime Version:	docker://1.12.6
 Kubelet Version:		v1.6.1+5115d708d7
 Kube-Proxy Version:		v1.6.1+5115d708d7
ExternalID:			437740049672994824
Non-terminated Pods:		(2 in total)
  Namespace			Name				CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ---------			----				------------	----------	---------------	-------------
  default			docker-registry-1-5szjs		100m (5%)	0 (0%)		256Mi (3%)0 (0%)
  default			router-1-vzlzq			100m (5%)	0 (0%)		256Mi (3%)0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  200m (10%)	0 (0%)		512Mi (7%)	0 (0%)
Events:		<none>

The output above shows that the node is running two pods: router-1-vzlzq and docker-registry-1-5szjs. Two more infrastructure nodes are available to migrate these two pods.

Mark a node as unschedulable and evacuate all of its pods:

$ oc adm drain ocp-infra-node-b7pl --delete-local-data
node "ocp-infra-node-b7pl" cordoned
WARNING: Deleting pods with local storage: docker-registry-1-5szjs
pod "docker-registry-1-5szjs" evicted
pod "router-1-vzlzq" evicted
node "ocp-infra-node-b7pl" drained

If the pod has attached local storage (for example, EmptyDir), the --delete-local-data option must be provided. Generally, pods running in production should use the local storage only for temporary or cache files, but not for anything important or persistent. For regular storage, applications should use object storage or persistent volumes. In this case, the docker-registry pod’s local storage is empty, because the object storage is used instead to store the container images.

The example below shows the output of the replication controller of the registry:

$ oc describe rc/docker-registry-1
Name:		docker-registry-1
Namespace:	default
Selector:	deployment=docker-registry-1,deploymentconfig=docker-registry,docker-registry=default
Labels:		docker-registry=default
		openshift.io/deployment-config.name=docker-registry
Annotations: ...
Replicas:	3 current / 3 desired
Pods Status:	3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:		deployment=docker-registry-1
			deploymentconfig=docker-registry
			docker-registry=default
  Annotations:		openshift.io/deployment-config.latest-version=1
			openshift.io/deployment-config.name=docker-registry
			openshift.io/deployment.name=docker-registry-1
  Service Account:	registry
  Containers:
   registry:
    Image:	openshift3/ose-docker-registry:v3.6.173.0.49
    Port:	5000/TCP
    Requests:
      cpu:	100m
      memory:	256Mi
    Liveness:	http-get https://:5000/healthz delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:	http-get https://:5000/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
    Environment:
      REGISTRY_HTTP_ADDR:					:5000
      REGISTRY_HTTP_NET:					tcp
      REGISTRY_HTTP_SECRET:					tyGEnDZmc8dQfioP3WkNd5z+Xbdfy/JVXf/NLo3s/zE=
      REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA:	false
      REGISTRY_HTTP_TLS_KEY:					/etc/secrets/registry.key
      OPENSHIFT_DEFAULT_REGISTRY:				docker-registry.default.svc:5000
      REGISTRY_CONFIGURATION_PATH:				/etc/registry/config.yml
      REGISTRY_HTTP_TLS_CERTIFICATE:				/etc/secrets/registry.crt
    Mounts:
      /etc/registry from docker-config (rw)
      /etc/secrets from registry-certificates (rw)
      /registry from registry-storage (rw)
  Volumes:
   registry-storage:
    Type:	EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
   registry-certificates:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	registry-certificates
    Optional:	false
   docker-config:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	registry-config
    Optional:	false
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  49m		49m		1	replication-controller			Normal		SuccessfulCreate	Created pod: docker-registry-1-dprp5

The event at the bottom of the output displays information about new pod creation. So, when listing all pods:

$ oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-dprp5    1/1       Running   0          52m
docker-registry-1-kr8jq    1/1       Running   0          1d
docker-registry-1-ncpl2    1/1       Running   0          1d
registry-console-1-g4nqg   1/1       Running   0          1d
router-1-2gshr             0/1       Pending   0          52m
router-1-85qm4             1/1       Running   0          1d
router-1-q5sr8             1/1       Running   0          1d

When observing the newly created registry pod, the example below shows that the pod has been created on the ocp-infra-node-rghb node, which is different from the deprecating node:

$ oc describe pod docker-registry-1-dprp5
Name:			docker-registry-1-dprp5
Namespace:		default
Security Policy:	hostnetwork
Node:			ocp-infra-node-rghb/10.156.0.10
...

The only difference between deprecating the infrastructure and the application node is that once the infrastructure node is evacuated, and if there is no plan to replace that node, the services running on infrastructure nodes can be scaled down:

$ oc scale dc/router --replicas 2
deploymentconfig "router" scaled

$ oc scale dc/docker-registry --replicas 2
deploymentconfig "docker-registry" scaled

Now, every infrastructure node is running only one kind of each pod:

$ oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-kr8jq    1/1       Running   0          1d
docker-registry-1-ncpl2    1/1       Running   0          1d
registry-console-1-g4nqg   1/1       Running   0          1d
router-1-85qm4             1/1       Running   0          1d
router-1-q5sr8             1/1       Running   0          1d

$ oc describe po/docker-registry-1-kr8jq | grep Node:
Node:			ocp-infra-node-p5zj/10.156.0.9

$ oc describe po/docker-registry-1-ncpl2 | grep Node:
Node:			ocp-infra-node-rghb/10.156.0.10

To verify that the scheduling on the node is disabled:

$ oc get nodes
NAME                  STATUS                     AGE       VERSION
ocp-infra-node-b7pl   Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-infra-node-p5zj   Ready                      1d        v1.6.1+5115d708d7
ocp-infra-node-rghb   Ready                      1d        v1.6.1+5115d708d7
ocp-master-dgf8       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-master-q1v2       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-master-vq70       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-node-020m         Ready                      1d        v1.6.1+5115d708d7
ocp-node-7t5p         Ready                      1d        v1.6.1+5115d708d7
ocp-node-n0dd         Ready                      1d        v1.6.1+5115d708d7

And that the node does not contain any pods:

$ oc describe node ocp-infra-node-b7pl
Name:			ocp-infra-node-b7pl
Role:
Labels:			beta.kubernetes.io/arch=amd64
			beta.kubernetes.io/instance-type=n1-standard-2
			beta.kubernetes.io/os=linux
			failure-domain.beta.kubernetes.io/region=europe-west3
			failure-domain.beta.kubernetes.io/zone=europe-west3-c
			kubernetes.io/hostname=ocp-infra-node-b7pl
			role=infra
Annotations:		volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:			<none>
CreationTimestamp:	Wed, 22 Nov 2017 09:36:36 -0500
Phase:
Conditions:
  ...
Addresses:		10.156.0.11,ocp-infra-node-b7pl
Capacity:
 cpu:		2
 memory:	7494480Ki
 pods:		20
Allocatable:
 cpu:		2
 memory:	7392080Ki
 pods:		20
System Info:
 Machine ID:			bc95ccf67d047f2ae42c67862c202e44
 System UUID:			9762CC3D-E23C-AB13-B8C5-FA16F0BCCE4C
 Boot ID:			ca8bf088-905d-4ec0-beec-8f89f4527ce4
 Kernel Version:		3.10.0-693.5.2.el7.x86_64
 OS Image:			Employee SKU
 Operating System:		linux
 Architecture:			amd64
 Container Runtime Version:	docker://1.12.6
 Kubelet Version:		v1.6.1+5115d708d7
 Kube-Proxy Version:		v1.6.1+5115d708d7
ExternalID:			437740049672994824
Non-terminated Pods:		(0 in total)
  Namespace			Name		CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ---------			----		------------	----------	---------------	-------------
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  0 (0%)	0 (0%)		0 (0%)		0 (0%)
Events:		<none>

Remove the infrastructure instance from the backend section in the /etc/haproxy/haproxy.cfg configuration file:

backend router80
    balance source
    mode tcp
    server infra-1.example.com 192.168.55.12:80 check
    server infra-2.example.com 192.168.55.13:80 check

backend router443
    balance source
    mode tcp
    server infra-1.example.com 192.168.55.12:443 check
    server infra-2.example.com 192.168.55.13:443 check

Then, restart the haproxy service.

$ sudo systemctl restart haproxy

Remove the node from the cluster after all pods are evicted with command:

$ oc delete node ocp-infra-node-b7pl
node "ocp-infra-node-b7pl" deleted

$ oc get nodes
NAME                  STATUS                     AGE       VERSION
ocp-infra-node-p5zj   Ready                      1d        v1.6.1+5115d708d7
ocp-infra-node-rghb   Ready                      1d        v1.6.1+5115d708d7
ocp-master-dgf8       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-master-q1v2       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-master-vq70       Ready,SchedulingDisabled   1d        v1.6.1+5115d708d7
ocp-node-020m         Ready                      1d        v1.6.1+5115d708d7
ocp-node-7t5p         Ready                      1d        v1.6.1+5115d708d7
ocp-node-n0dd         Ready                      1d        v1.6.1+5115d708d7

Create a backup of the node configuration files:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo cp -aR /etc/origin ${MYBACKUPDIR}/etc
$ sudo cp -aR /etc/sysconfig/atomic-openshift-node ${MYBACKUPDIR}/etc/sysconfig/

OpenShift Container Platform uses specific files that must be taken into account when planning the backup policy, including:

To create those files:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}/etc/sysconfig
$ sudo mkdir -p ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors
$ sudo cp -aR /etc/sysconfig/{iptables,docker-*,flanneld} \
    ${MYBACKUPDIR}/etc/sysconfig/
$ sudo cp -aR /etc/dnsmasq* /etc/cni ${MYBACKUPDIR}/etc/
$ sudo cp -aR /etc/pki/ca-trust/source/anchors/* \
    ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/

If a package is accidentally removed, or a file included in an rpm package should be restored, having a list of rhel packages installed on the system can be useful.

To create a list of the current rhel packages installed in the system:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo mkdir -p ${MYBACKUPDIR}
$ rpm -qa | sort | sudo tee $MYBACKUPDIR/packages.txt

The following files should now be present in the backup directory:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo find ${MYBACKUPDIR} -mindepth 1 -type f -printf '%P\n'
etc/sysconfig/atomic-openshift-node
etc/sysconfig/flanneld
etc/sysconfig/iptables
etc/sysconfig/docker-network
etc/sysconfig/docker-storage
etc/sysconfig/docker-storage-setup
etc/sysconfig/docker-storage-setup.rpmnew
etc/origin/node/system:node:app-node-0.example.com.crt
etc/origin/node/system:node:app-node-0.example.com.key
etc/origin/node/ca.crt
etc/origin/node/system:node:app-node-0.example.com.kubeconfig
etc/origin/node/server.crt
etc/origin/node/server.key
etc/origin/node/node-dnsmasq.conf
etc/origin/node/resolv.conf
etc/origin/node/node-config.yaml
etc/origin/node/flannel.etcd-client.key
etc/origin/node/flannel.etcd-client.csr
etc/origin/node/flannel.etcd-client.crt
etc/origin/node/flannel.etcd-ca.crt
etc/origin/cloudprovider/openstack.conf
etc/pki/ca-trust/source/anchors/openshift-ca.crt
etc/pki/ca-trust/source/anchors/registry-ca.crt
etc/dnsmasq.conf
etc/dnsmasq.d/origin-dns.conf
etc/dnsmasq.d/origin-upstream-dns.conf
etc/dnsmasq.d/node-dnsmasq.conf
packages.txt

If needed, the files can be compressed to save space:

$ MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
$ sudo tar -zcvf /backup/$(hostname)-$(date +%Y%m%d).tar.gz $MYBACKUPDIR
$ sudo rm -Rf ${MYBACKUPDIR}

Restore the /etc/origin/node/node-config.yaml file:

# MYBACKUPDIR=/backup/$(hostname)/$(date +%Y%m%d)
# cp /etc/origin/node/node-config.yaml /etc/origin/node/node-config.yaml.old
# cp /backup/$(hostname)/$(date +%Y%m%d)/etc/origin/node/node-config.yaml /etc/origin/node/node-config.yaml
# reboot

If you cannot restart OpenShift Container Platform because packages are missing, reinstall the packages.

Restore a system certificate by copying the certificate to the /etc/pki/ca-trust/source/anchors/ directory and execute the update-ca-trust:

$ MYBACKUPDIR=*/backup/$(hostname)/$(date +%Y%m%d)*
$ sudo cp ${MYBACKUPDIR}/etc/pki/ca-trust/source/anchors/my_company.crt /etc/pki/ca-trust/source/anchors/
$ sudo update-ca-trust

Check the current etcd cluster status to avoid adding new hosts to an unhealthy cluster.

Before running the scaleup playbook, ensure the new host is registered to the proper Red Hat software channels:

# subscription-manager register \
    --username=*<username>* --password=*<password>*
# subscription-manager attach --pool=*<poolid>*
# subscription-manager repos --disable="*"
# subscription-manager repos \
    --enable=rhel-7-server-rpms \
    --enable=rhel-7-server-extras-rpms

etcd is hosted in the rhel-7-server-extras-rpms software channel.

Upgrade etcd and iptables on the current etcd nodes:

# yum update etcd iptables-services

Remove each other etcd host from the etcd cluster. Run the following command for each etcd node:

# etcdctl -C https://<surviving host IP address>:2379 \
  --ca-file=/etc/etcd/ca.crt     \
  --cert-file=/etc/etcd/peer.crt     \
  --key-file=/etc/etcd/peer.key member remove <failed member ID>

Restart the master API service on every master:

# master-restart api restart-master controller

Remove the failed host from the cluster:

# etcdctl2 cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
failed to check the health of member 8372784203e11288 on https://192.168.55.21:2379: Get https://192.168.55.21:2379/health: dial tcp 192.168.55.21:2379: getsockopt: connection refused
member 8372784203e11288 is unreachable: [https://192.168.55.21:2379] are all unreachable
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy

# etcdctl2 member remove 8372784203e11288 1
Removed member 8372784203e11288 from cluster

# etcdctl2 cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy

To ensure the etcd configuration does not use the failed host when the etcd service is restarted, modify the /etc/etcd/etcd.conf file on all remaining etcd hosts and remove the failed host in the value for the ETCD_INITIAL_CLUSTER variable:

# vi /etc/etcd/etcd.conf

For example:

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380,master-2.example.com=https://192.168.55.13:2380

becomes:

ETCD_INITIAL_CLUSTER=master-0.example.com=https://192.168.55.8:2380,master-1.example.com=https://192.168.55.12:2380

Modify the Ansible inventory file to reflect the current status of the cluster and to avoid issues when re-running a playbook:

[OSEv3:children]
masters
nodes
etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com

If you are using Flannel, modify the flanneld service configuration located at /etc/sysconfig/flanneld on every host and remove the etcd host:

FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379

Restart the flanneld service:

# systemctl restart flanneld.service

View the project and pods:

$ oc get pods
NAME           READY     STATUS      RESTARTS   AGE
demo-1-build   0/1       Completed   0          2h
demo-2-fxx6d   1/1       Running     0          1h

Describe the desired pod to find the volumes that are currently used by a persistent volume:

$ oc describe pod demo-2-fxx6d
Name:			demo-2-fxx6d
Namespace:		test
Security Policy:	restricted
Node:			ip-10-20-6-20.ec2.internal/10.20.6.20
Start Time:		Tue, 05 Dec 2017 12:54:34 -0500
Labels:			app=demo
			deployment=demo-2
			deploymentconfig=demo
Status:			Running
IP:			172.16.12.5
Controllers:		ReplicationController/demo-2
Containers:
  demo:
    Container ID:	docker://201f3e55b373641eb36945d723e1e212ecab847311109b5cee1fd0109424217a
    Image:		docker-registry.default.svc:5000/test/demo@sha256:0a9f2487a0d95d51511e49d20dc9ff6f350436f935968b0c83fcb98a7a8c381a
    Image ID:		docker-pullable://docker-registry.default.svc:5000/test/demo@sha256:0a9f2487a0d95d51511e49d20dc9ff6f350436f935968b0c83fcb98a7a8c381a
    Port:		8080/TCP
    State:		Running
      Started:		Tue, 05 Dec 2017 12:54:52 -0500
    Ready:		True
    Restart Count:	0
    Volume Mounts:
      */opt/app-root/src/uploaded from persistent-volume (rw)*
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8mmrk (ro)
    Environment Variables:	<none>
...omitted...

This output shows that the persistent data is in the /opt/app-root/src/uploaded directory.

Copy the data locally:

$ oc rsync demo-2-fxx6d:/opt/app-root/src/uploaded ./demo-app
receiving incremental file list
uploaded/
uploaded/ocp_sop.txt
uploaded/lost+found/

sent 38 bytes  received 190 bytes  152.00 bytes/sec
total size is 32  speedup is 0.14

The ocp_sop.txt file is downloaded to the local system to be backed up by backup software or another backup mechanism.

From a master instance, or as a cluster administrator, allow the evacuation of any pod from the node and disable scheduling of other pods on that node:

$ NODE=ose-app-node01.example.com
$ oc adm manage-node ${NODE} --schedulable=false
NAME                          STATUS                     AGE       VERSION
ose-app-node01.example.com   Ready,SchedulingDisabled   20m       v1.6.1+5115d708d7

$ oc adm drain ${NODE} --ignore-daemonsets
node "ose-app-node01.example.com" already cordoned
pod "perl-1-build" evicted
pod "perl-1-3lnsh" evicted
pod "perl-1-9jzd8" evicted
node "ose-app-node01.example.com" drained

List the pods on the node to verify that they have been removed:

$ oc adm manage-node ${NODE} --list-pods

Listing matched pods on node: ose-app-node01.example.com

NAME      READY     STATUS    RESTARTS   AGE

Stop the docker:

# systemctl stop docker

Stop the node service by removing the pod definition and rebooting the host:

# mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/

Run the docker-storage-setup command to extend the volume groups and logical volumes associated with container storage:

# docker-storage-setup
INFO: Volume group backing root filesystem could not be determined
INFO: Device /dev/xvdb is already partitioned and is part of volume group docker_vol
INFO: Device node /dev/xvdd1 exists.
  Physical volume "/dev/xvdd1" successfully created.
  Volume group "docker_vol" successfully extended

Start the Docker services:

# systemctl start docker
# vgs
  VG         #PV #LV #SN Attr   VSize  VFree
  docker_vol   2   1   0 wz--n- 64.99g <55.00g

Restart the node service by rebooting the host:

# systemctl restart atomic-openshift-node.service

A benefit in adding a disk compared to creating a new volume group and re-running docker-storage-setup is that the images that were used on the system still exist after the new storage has been added:

# docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
docker-registry.default.svc:5000/tet/perl               latest              8b0b0106fb5e        13 minutes ago      627.4 MB
registry.access.redhat.com/rhscl/perl-524-rhel7         <none>              912b01ac7570        6 days ago          559.5 MB
registry.access.redhat.com/openshift3/ose-deployer      v3.6.173.0.21       89fd398a337d        5 weeks ago         970.2 MB
registry.access.redhat.com/openshift3/ose-pod           v3.6.173.0.21       63accd48a0d7        5 weeks ago         208.6 MB

With the increase in storage capacity, enable the node to be schedulable in order to accept new incoming pods.

As a cluster administrator, run the following from a master instance:

$ oc adm manage-node ${NODE} --schedulable=true

ose-master01.example.com   Ready,SchedulingDisabled   24m       v1.6.1+5115d708d7
ose-master02.example.com   Ready,SchedulingDisabled   24m       v1.6.1+5115d708d7
ose-master03.example.com   Ready,SchedulingDisabled   24m       v1.6.1+5115d708d7
ose-infra-node01.example.com   Ready                      24m       v1.6.1+5115d708d7
ose-infra-node02.example.com   Ready                      24m       v1.6.1+5115d708d7
ose-infra-node03.example.com   Ready                      24m       v1.6.1+5115d708d7
ose-app-node01.example.com   Ready                      24m       v1.6.1+5115d708d7
ose-app-node02.example.com   Ready                      24m       v1.6.1+5115d708d7

Stop the docker:

# systemctl stop docker

Stop the node service by removing the pod definition:

# mkdir -p /etc/origin/node/pods-stopped
# mv /etc/origin/node/pods/* /etc/origin/node/pods-stopped/

Resize the existing disk as desired. This can depend on your environment:

Run docker-storage-setup to reconfigure the new disk:

# docker-storage-setup
INFO: Volume group backing root filesystem could not be determined
INFO: Device /dev/xvdb is already partitioned and is part of volume group docker_vol
INFO: Device node /dev/xvdd1 exists.
  Physical volume "/dev/xvdd1" successfully created.
  Volume group "docker_vol" successfully extended

Start the Docker services:

# systemctl start docker
# vgs
  VG         #PV #LV #SN Attr   VSize  VFree
  docker_vol   2   1   0 wz--n- 64.99g <55.00g

Restart the node service by rebooting the host:

# systemctl restart atomic-openshift-node.service

Copy the CA certificate to /etc/pki/ca-trust/source/anchors/:

$ sudo cp myregistry.example.com.crt /etc/pki/ca-trust/source/anchors/

Extract and add the CA certificate to the list of trusted certificates authorities:

$ sudo update-ca-trust extract

Verify the SSL certificate using the openssl command:

$ openssl verify myregistry.example.com.crt
myregistry.example.com.crt: OK

Once the certificate is in place and the trust is updated, restart the docker service to ensure the new certificates are properly set:

$ sudo systemctl restart docker.service

On every node create a new directory in /etc/docker/certs.d where the name of the directory is the host name of the Docker registry:

$ sudo mkdir -p /etc/docker/certs.d/myregistry.example.com

Accessing the Docker registry via IP address requires the creation of a new directory within /etc/docker/certs.d on every node where the name of the directory is the IP of the Docker registry:

$ sudo mkdir -p /etc/docker/certs.d/10.10.10.10

Copy the CA certificate to the newly created Docker directories from the previous steps:

$ sudo cp myregistry.example.com.crt \
  /etc/docker/certs.d/myregistry.example.com/ca.crt

$ sudo cp myregistry.example.com.crt /etc/docker/certs.d/10.10.10.10/ca.crt

Once the certificates have been copied, restart the docker service to ensure the new certificates are used:

$ sudo systemctl restart docker.service

If using /etc/docker/certs.d, copy all the certificates included in the directory and store the files:

$ sudo tar -czvf docker-registry-certs-$(hostname)-$(date +%Y%m%d).tar.gz /etc/docker/certs.d/

If using a system trust, store the certificates prior to adding them within the system trust. Once the store is complete, extract the certificate for restoration using the trust command. Identify the system trust CAs and note the pkcs11 ID:

$ trust list
...[OUTPUT OMMITED]...
pkcs11:id=%a5%b3%e1%2b%2b%49%b6%d7%73%a1%aa%94%f5%01%e7%73%65%4c%ac%50;type=cert
    type: certificate
    label: MyCA
    trust: anchor
    category: authority
...[OUTPUT OMMITED]...

Extract the certificate in pem format and provide it a name. For example, myca.crt.

$ trust extract --format=pem-bundle \
 --filter="%a5%b3%e1%2b%2b%49%b6%d7%73%a1%aa%94%f5%01%e7%73%65%4c%ac%50;type=cert" myca.crt

Verify the certificate has been properly extracted via openssl:

$ openssl verify myca.crt

To allow users to search for images using docker search with other registries, add those registries to the /etc/containers/registries.conf file under the registries parameter:

registries:
  - registry.access.redhat.com
  - my.registry.example.com

Prior to OpenShift Container Platform version 3.6, this was accomplished using /etc/sysconfig/docker with the following options:

ADD_REGISTRY="--add-registry=registry.access.redhat.com --add-registry=my.registry.example.com"

The first registry added is the first registry searched.

Restart the docker daemon to allow for my.registry.example.com to be used:

$ sudo systemctl restart docker.service

Restarting the docker daemon causes the docker containers to restart.

Using the Ansible installer, this can be configured using the openshift_docker_additional_registries variable in the Ansible hosts file:

openshift_docker_additional_registries=registry.access.redhat.com,my.registry.example.com

Add the allowed registries to the /etc/containers/registries.conf file with the registries flag:

registries:
  - registry.access.redhat.com
  - my.registry.example.com

Prior to 3.6, the /etc/sysconfig/docker file is modified instead:

ADD_REGISTRY="--add-registry=registry.access.redhat.com --add-registry=my.registry.example.com"

Block the rest of the registries:

block_registries:
   - all

Block the rest of the registries in older versions:

BLOCK_REGISTRY='--block-registry=all'

Restart the docker daemon:

$ sudo systemctl restart docker.service

Restarting the docker daemon causes the docker containers to restart.

In this example, the docker.io registry has been blacklisted, so any operation regarding that registry fails:

$ sudo docker pull hello-world
Using default tag: latest
Trying to pull repository registry.access.redhat.com/hello-world ...
Trying to pull repository my.registry.example.com/hello-world ...
Trying to pull repository registry.access.redhat.com/hello-world ...
unknown: Not Found
$ sudo docker pull docker.io/hello-world
Using default tag: latest
Trying to pull repository docker.io/library/hello-world ...
All endpoints blocked.

Add docker.io back to the registries variable by modifying the file again and restarting the service.

registries:
  - registry.access.redhat.com
  - my.registry.example.com
  - docker.io
block_registries:
  - all

or

ADD_REGISTRY="--add-registry=registry.access.redhat.com --add-registry=my.registry.example.com --add-registry=docker.io"
BLOCK_REGISTRY='--block-registry=all'

Restart the Docker service:

$ sudo systemctl restart docker

To verify that the image is now available to be pulled:

$ sudo docker pull docker.io/hello-world
Using default tag: latest
Trying to pull repository docker.io/library/hello-world ...
latest: Pulling from docker.io/library/hello-world

9a0669468bf7: Pull complete
Digest: sha256:0e06ef5e1945a718b02a8c319e15bae44f47039005530bc617a5d071190ed3fc

If using an external registry is required, for example to modify the docker daemon configuration file in all the node hosts that require to use that registry, create a blacklist on those nodes to avoid malicious containers from being executed.

Using the Ansible installer, this can be configured using the openshift_docker_additional_registries and openshift_docker_blocked_registries variables in the Ansible hosts file:

openshift_docker_additional_registries=registry.access.redhat.com,my.registry.example.com
openshift_docker_blocked_registries=all

Modify /etc/containers/registries.conf configuration file to add the localhost insecure registry:

registries:
  - registry.access.redhat.com
  - my.registry.example.com
  - docker.io
insecure_registries:
  - localhost:5000
block_registries:
  - all

Prior to 3.6, modify the /etc/sysconfig/docker configuration file to add the localhost:

ADD_REGISTRY="--add-registry=registry.access.redhat.com --add-registry=my.registry.example.com --add-registry=docker.io --add-registry=localhost:5000"
INSECURE_REGISTRY="--insecure-registry=localhost:5000"
BLOCK_REGISTRY='--block-registry=all'

Restart the docker daemon to use the registry:

$ sudo systemctl restart docker.service

Restarting the docker daemon causes the docker containers to be restarted.

Run a Docker registry pod at localhost:

$ sudo docker run -p 5000:5000 registry:2

Pull an image:

$ sudo docker pull openshift/hello-openshift

Tag the image:

$ sudo docker tag docker.io/openshift/hello-openshift:latest localhost:5000/hello-openshift-local:latest

Push the image to the local registry:

$ sudo docker push localhost:5000/hello-openshift-local:latest

Using the Ansible installer, this can be configured using the openshift_docker_additional_registries, openshift_docker_blocked_registries, and openshift_docker_insecure_registries variables in the Ansible hosts file:

openshift_docker_additional_registries=registry.access.redhat.com,my.registry.example.com,localhost:5000
openshift_docker_insecure_registries=localhost:5000
openshift_docker_blocked_registries=all

Create a dockercfg secret in the project where the user is going to log in to the docker registry:

$ oc project <my_project>
$ oc create secret docker-registry <my_registry> --docker-server=<my.registry.example.com> --docker-username=<username> --docker-password=<my_password> --docker-email=<me@example.com>

If a .dockercfg file exists, create the secret using the oc command:

$ oc create secret generic <my_registry> --from-file=.dockercfg=<path/to/.dockercfg> --type=kubernetes.io/dockercfg

Populate the $HOME/.docker/config.json file:

$ oc create secret generic <my_registry> --from-file=.dockerconfigjson=<path/to/.dockercfg> --type=kubernetes.io/dockerconfigjson

Use the dockercfg secret to pull images from the authenticated registry by linking the secret to the service account performing the pull operations. The default service account to pull images is named default:

$ oc secrets link default <my_registry> --for=pull

For pushing images using the S2I feature, the dockercfg secret is mounted in the S2I pod, so it needs to be linked to the proper service account that performs the build. The default service account used to build images is named builder.

$ oc secrets link builder <my_registry>

In the buildconfig, the secret should be specified for push or pull operations:

"type": "Source",
"sourceStrategy": {
    "from": {
        "kind": "DockerImage",
        "name": "*my.registry.example.com*/myproject/myimage:stable"
    },
    "pullSecret": {
        "name": "*mydockerregistry*"
    },
...[OUTPUT ABBREVIATED]...
"output": {
    "to": {
        "kind": "DockerImage",
        "name": "*my.registry.example.com*/myproject/myimage:latest"
    },
    "pushSecret": {
        "name": "*mydockerregistry*"
    },
...[OUTPUT ABBREVIATED]...

If the external registry delegates authentication to external services, create both dockercfg secrets: the registry one using the registry URL and the external authentication system using its own URL. Both secrets should be added to the service accounts.

$ oc project <my_project>
$ oc create secret docker-registry <my_registry> --docker-server=*<my_registry_example.com> --docker-username=<username> --docker-password=<my_password> --docker-email=<me@example.com>
$ oc create secret docker-registry <my_docker_registry_ext_auth> --docker-server=<my.authsystem.example.com> --docker-username=<username> --docker-password=<my_password> --docker-email=<me@example.com>
$ oc secrets link default <my_registry> --for=pull
$ oc secrets link default <my_docker_registry_ext_auth> --for=pull
$ oc secrets link builder <my_registry>
$ oc secrets link builder <my_docker_registry_ext_auth>

If the ImagePolicy plug-in is enabled, it needs to be modified to allow the external registries to be used by modifying the /etc/origin/master/master-config.yaml file on every master node:

admissionConfig:
  pluginConfig:
    openshift.io/ImagePolicy:
      configuration:
        kind: ImagePolicyConfig
        apiVersion: v1
        executionRules:
        - name: allow-images-from-other-registries
          onResources:
          - resource: pods
          - resource: builds
          matchRegistries:
          - docker.io
          - <my.registry.example.com>
          - registry.access.redhat.com

To enable the admission plug-ins at installation time, the openshift_master_admission_plugin_config variable can be used with a json formatted string including all the pluginConfig configuration:

openshift_master_admission_plugin_config={"openshift.io/ImagePolicy":{"configuration":{"kind":"ImagePolicyConfig","apiVersion":"v1","executionRules":[{"name":"allow-images-from-other-registries","onResources":[{"resource":"pods"},{"resource":"builds"}],"matchRegistries":["docker.io","*my.registry.example.com*","registry.access.redhat.com"]}]}}}

To configure the allowed registries where users can import images, add the following to the /etc/origin/master/master-config.yaml file:

imagePolicyConfig:
  allowedRegistriesForImport:
  - domainName: docker.io
  - domainName: '\*.docker.io'
  - domainName: '*.redhat.com'
  - domainName: 'my.registry.example.com'

Even if not recommended, if the external authenticated registry is insecure or the certificates can not be trusted, the oc import-image command can be used with the --insecure=true option.

If the external authenticated registry is secure, the registry certificate should be trusted in the master hosts as they run the registry import controller as:

Copy the certificate in the /etc/pki/ca-trust/source/anchors/:

$ sudo cp <my.registry.example.com.crt> /etc/pki/ca-trust/source/anchors/<my.registry.example.com.crt>

Run update-ca-trust command:

$ sudo update-ca-trust

Restart the master services on all the master hosts:

$ sudo master-restart api
$ sudo master-restart controllers

The certificate for the external registry should be trusted in the OpenShift Container Platform registry:

$ for i in pem openssl java; do
  oc create configmap ca-trust-extracted-${i} --from-file /etc/pki/ca-trust/extracted/${i}
  oc set volume dc/docker-registry --add -m /etc/pki/ca-trust/extracted/${i} --configmap-name=ca-trust-extracted-${i} --name ca-trust-extracted-${i}
done

Alternatively, modify the registry image in order to trust the proper certificates rebuilding the image using a Dockerfile as:

FROM registry.access.redhat.com/openshift3/ose-docker-registry:v3.6
ADD <my.registry.example.com.crt> /etc/pki/ca-trust/source/anchors/
USER 0
RUN update-ca-trust extract
USER 1001

Rebuild the image, push it to a docker registry, and use that image as spec.template.spec.containers["name":"registry"].image in the registry deploymentconfig:

$ oc patch dc docker-registry -p '{"spec":{"template":{"spec":{"containers":[{"name":"registry","image":"*myregistry.example.com/openshift3/ose-docker-registry:latest*"}]}}}}'