Chapter 6. Renewing the AMQ Interconnect certificate

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Verify that some or all dispatch router connections have failed:

   $ oc exec -it $(oc get po -l application=default-interconnect -o jsonpath='{.items[0].metadata.name}') -- qdstat --connections | grep Router | wc
         0       0       0

   Copy to Clipboard Toggle word wrap

4. Check for this error in the Red Hat OpenShift Container Platform-hosted AMQ Interconnect logs:

   $ oc logs -l application=default-interconnect | tail
   [...]
   2022-11-10 20:51:22.863466 +0000 SERVER (info) [C261] Connection from 10.10.10.10:34570 (to 0.0.0.0:5671) failed: amqp:connection:framing-error SSL Failure: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure

   Copy to Clipboard Toggle word wrap
5. Log into your RHOSP undercloud.

6. Check for this error in the RHOSP-hosted AMQ Interconnect logs of a node with a failed connection:

   $ ssh controller-0.ctlplane -- sudo tail /var/log/containers/metrics_qdr/metrics_qdr.log
   [...]
   2022-11-10 20:50:44.311646 +0000 SERVER (info) [C137] Connection to default-interconnect-5671-service-telemetry.apps.mycluster.com:443 failed: amqp:connection:framing-error SSL Failure: error:0A000086:SSL routines::certificate verify failed

   Copy to Clipboard Toggle word wrap

7. Confirm that the CA certificate has expired by examining the file on an RHOSP node:

   $ ssh controller-0.ctlplane -- cat /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem | openssl x509 -text | grep "Not After"
               Not After : Nov 10 20:31:16 2022 GMT
   
   $ date
   Mon Nov 14 11:10:40 EST 2022

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Export the CA certificate to STFCA.pem:

   $ oc get secret/default-interconnect-selfsigned -o jsonpath='{.data.ca\.crt}' | base64 -d > STFCA.pem

   Copy to Clipboard Toggle word wrap
4. Copy STFCA.pem to your RHOSP undercloud.
5. Log into your RHOSP undercloud.
6. Edit the stf-connectors.yaml file to contain the new caCertFileContent. For more information, see Section 4.1.4, “Configuring the STF connection for the overcloud”.

7. Copy the STFCA.pem file to each RHOSP overcloud node:

   [stack@undercloud-0 ~]$ ansible -i overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml allovercloud -b -m copy -a "src=STFCA.pem dest=/var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem"

   Copy to Clipboard Toggle word wrap

8. Restart the metrics_qdr container on each RHOSP overcloud node:

   [stack@undercloud-0 ~]$ ansible -i overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml allovercloud -m shell -a "sudo podman restart metrics_qdr"

   Copy to Clipboard Toggle word wrap
   Note

   You do not need to deploy the overcloud after you copy the STFCA.pem file and restart the metrics_qdr container. You edit the stf-connectors.yaml file so that future deployments do not overwrite the new CA certificate.