Restore RHOCP on RHOSP authentication after adoption

Update Red Hat OpenShift Container Platform (RHOCP) clusters with installer-provisioned infrastructure to reference the new keystone endpoint in the cloud credentials instead of the director Operator endpoint. This facilitates authentication with keystone and avoids timeouts.

About this task

Important
Run all commands against the guest RHOCP cluster that is running on RHOSP VMs, not the underlying RHOSO host cluster.

Procedure

  1. Verify the stale secret on the RHOCP on RHOSP cluster: 
    $ oc get secret openstack-credentials -n kube-system \
    -o jsonpath='{.data.clouds\.yaml}' | base64 -d | grep auth_url

    The output shows the director Operator endpoint that can no longer be used after adoption. For example, https://overcloud.redhat.local:13000 or http://<tripleo-vip>:5000.

  2. Check if the new RHOSO keystone is reachable from the RHOCP on RHOSP cluster nodes: 
    $ RHOSO_AUTH_URL="http://keystone-public-openstack.apps.<host-cluster-domain>"
$ oc debug node/$(oc get nodes -o jsonpath='{.items[0].metadata.name}') \
    -- chroot /host curl -sk -m 10 "${RHOSO_AUTH_URL}/v3/"
    Important
    If the command returns a JSON response with keystone version information, proceed to step 3. If the command times out, the RHOSO endpoint is not reachable from the RHOCP on RHOSP cluster’s network. You can still proceed with steps 3-5 because the credential update and image-registry fix will work regardless of RHOSP reachability. However, operators such as OCCM and CSI will continue logging timeout errors until the RHOSO endpoint becomes reachable from the guest nodes.
  3. Update the source secret with the RHOSO endpoint:
    Note
    The RHOCP on RHOSP user, password, and project are preserved after adoption. Adoption uses the same keystone database as the one used in Red Hat OpenStack Platform (RHOSP) 17.1. Only the auth_url needs to change. If RHOSO uses HTTPS with a different certificate authority, update the cacert path too. 
    $ oc get secret openstack-credentials -n kube-system \
    -o jsonpath='{.data.clouds\.yaml}' | base64 -d > /tmp/clouds.yaml
# Edit /tmp/clouds.yaml: change auth_url to the RHOSO keystone endpoint
$ B64=$(base64 < /tmp/clouds.yaml | tr -d '\n')
$ oc patch secret openstack-credentials -n kube-system \
    -p "{\"data\":{\"clouds.yaml\":\"${B64}\"}}"
  4. Wait for Cloud Credential Operator (CCO) to propagate the source secret to all namespace-specific secrets, and then verify that the new RHOSO endpoint is reachable: 
    $ oc get secret installer-cloud-credentials -n openshift-image-registry \
    -o jsonpath='{.data.clouds\.yaml}' | base64 -d | grep auth_url
    Note
    The output should show the new RHOSO endpoint in the auth_url field. If the output still shows the previous endpoint after a few minutes, check that the CCO pod is running and that the CCO mode is Passthrough: 
    $ oc get pods -n openshift-cloud-credential-operator
$ oc get cloudcredential cluster -o jsonpath='{.spec.credentialsMode}'

    If the CCO is healthy but not propagating, patch each namespace secret directly by using the following command:

    $ oc patch secret openstack-credentials -n kube-system \
    -p "{\"data\":{\"clouds.yaml\":\"${B64}\"}}"
  5. Switch the image-registry storage from Swift to emptyDir: 
    $ oc patch configs.imageregistry.operator.openshift.io cluster \
    --type json -p '[{"op":"replace","path":"/spec/storage","value":{"emptyDir":{}}}]'
    Warning
    emptyDir is non-persistent. Registry images are lost when the pod restarts. For production, configure PVC-backed storage instead.

    emptyDir does not require working RHOSP credentials. The image-registry operator only calls the RHOSP API when Swift is the configured back end. This step works regardless of whether the new RHOSO keystone is reachable from the RHOCP on RHOSP cluster node.

Results

  • Verify that the credentials are pointing to the new RHOSO endpoint: 
    $ oc get co image-registry cloud-controller-manager storage machine-api

    Wait about 5 minutes.

    The following example shows the output if the new RHOSO keystone was not reachable from the RHOCP on RHOSP cluster nodes in step 2:

    NAME                       VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
image-registry             4.22.0-rc.0   True        False         True       14m     ImagePrunerDegraded: Job has reached the specified backoff limit
cloud-controller-manager   4.22.0-rc.0   True        False         False      3d10h
storage                    4.22.0-rc.0   True        False         False      3d10h
machine-api                4.22.0-rc.0   True        False         False      3d9h
    Note
    • If you completed steps 2-4 with a reachable RHOSO endpoint, the timeout errors should be resolved.
    • The output displays an environment that is running Red Hat OpenShift Container Platform (RHOCP) 4.22. If you are using an earlier version of Red Hat OpenShift Container Platform (RHOCP), you might see different operator states.