Chapter 7. Upgrading Service Telemetry Framework to version 1.5
To upgrade Service Telemetry Framework (STF) 1.4 to STF 1.5, you must complete the following steps:
- Replace AMQ Certificate Manager with Certificate Manager.
-
Remove the
ClusterServiceVersion
andSubscription
objects for Smart Gateway Operator and Service Telemetry Operator in theservice-telemetry
namespace on your Red Hat OpenShift Container Platform environment. - Upgrade Red Hat OpenShift Container Platform from 4.8 to 4.10.
- Re-enable the operators that you removed.
Prerequisites
-
You have backed up your data. There is an outage during the Red Hat OpenShift Container Platform upgrade. You cannot reconfigure the
ServiceTelemetry
andSmartGateway
objects during the Operators replacement. - You have prepared your environment for upgrade from Red Hat OpenShift Container Platform 4.8 to the supported version, 4.10.
- The Red Hat OpenShift Container Platform cluster is fully-connected. STF does not support disconnected or restricted-network clusters.
7.1. Removing the Service Telemetry Framework 1.4 Operators
Remove the Service Telemetry Framework (STF) 1.4 Operators and the AMQ Certificate Manager Operator from the Red Hat OpenShift Container Platform 4.8.
Procedure
- Remove the Service Telemetry Operator.
- Remove the Smart Gateway Operator.
- Remove the AMQ Certificate Manager Operator.
- Remove the Grafana Operator.
Additional resources
- For more information about removing Operators from the Red Hat OpenShift Container Platform, see Deleting Operators from a cluster.
7.1.1. Removing the Service Telemetry Operator
As part of upgrading your Service Telemetry Framework (STF) installation, you must remove the Service Telemetry Operator in the service-telemetry
namespace on your Red Hat OpenShift Container Platform environment.
Procedure
Change to the
service-telemetry
project:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc project service-telemetry
$ oc project service-telemetry
Remove the Service Telemetry Operator Subscription:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry
$ oc delete sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry subscription.operators.coreos.com "service-telemetry-operator" deleted
Remove the Service Telemetry Operator
ClusterServiceVersion
:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry
$ oc delete csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry clusterserviceversion.operators.coreos.com "service-telemetry-operator.v1.4.1669718959" deleted
Verification
Verify that the Service Telemetry Operator deployment is not running:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get deploy --selector=operators.coreos.com/service-telemetry-operator.service-telemetry
$ oc get deploy --selector=operators.coreos.com/service-telemetry-operator.service-telemetry No resources found in service-telemetry namespace.
Verify the Service Telemetry Operator subscription is absent:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry
$ oc get sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry No resources found in service-telemetry namespace.
Verify the Service Telemetry Operator ClusterServiceVersion is absent:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry
$ oc get csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry No resources found in service-telemetry namespace.
7.1.2. Removing the Smart Gateway Operator
As part of upgrading your Service Telemetry Framework (STF) installation, you must remove the Smart Gateway Operator in the service-telemetry
namespace on your Red Hat OpenShift Container Platform environment.
Procedure
Change to the
service-telemetry
project:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc project service-telemetry
$ oc project service-telemetry
Remove the Smart Gateway Operator Subscription:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry
$ oc delete sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry subscription.operators.coreos.com "smart-gateway-operator-stable-1.4-redhat-operators-openshift-marketplace" deleted
Remove the Smart Gateway Operator
ClusterServiceVersion
:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry
$ oc delete csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry clusterserviceversion.operators.coreos.com "smart-gateway-operator.v4.0.1669718962" deleted
Verification
Verify that the Smart Gateway Operator deployment is not running:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get deploy --selector=operators.coreos.com/smart-gateway-operator.service-telemetry
$ oc get deploy --selector=operators.coreos.com/smart-gateway-operator.service-telemetry No resources found in service-telemetry namespace.
Verify the Smart Gateway Operator subscription is absent:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry
$ oc get sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry No resources found in service-telemetry namespace.
Verify the Smart Gateway Operator ClusterServiceVersion is absent:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry
$ oc get csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry No resources found in service-telemetry namespace.
7.1.3. Removing the AMQ Certificate Manager Operator
Procedure
Remove the AMQ Certificate Manager Operator Subscription:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators
$ oc delete sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators subscription.operators.coreos.com "amq7-cert-manager-operator" deleted
Remove the AMQ Certificate Manager Operator
ClusterServiceVersion
:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators
$ oc delete csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators clusterserviceversion.operators.coreos.com "amq7-cert-manager.v1.0.11" deleted
Verification
Verify that the AMQ Certificate Manager Operator deployment is not running:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get deploy --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators
$ oc get deploy --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators No resources found in openshift-operators namespace.
Verify that the AMQ Certificate Manager Operator subscription is absent:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.service-telemetry
$ oc get sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.service-telemetry No resources found in openshift-operators namespace.
Verify that the AMQ Certificate Manager Operator Cluster Service Version is absent:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators
$ oc get csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators No resources found in openshift-operators namespace.
7.1.4. Removing the Grafana Operator
Procedure
Remove the Grafana Operator Subscription:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete sub --selector=operators.coreos.com/grafana-operator.service-telemetry
$ oc delete sub --selector=operators.coreos.com/grafana-operator.service-telemetry subscription.operators.coreos.com "grafana-operator" deleted
Remove the Grafana Operator
ClusterServiceVersion
:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete csv --selector=operators.coreos.com/grafana-operator.service-telemetry
$ oc delete csv --selector=operators.coreos.com/grafana-operator.service-telemetry clusterserviceversion.operators.coreos.com "grafana-operator.v3.10.3" deleted
Verification
Verify the Grafana Operator deployment is not running:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get deploy --selector=operators.coreos.com/grafana-operator.service-telemetry
$ oc get deploy --selector=operators.coreos.com/grafana-operator.service-telemetry No resources found in service-telemetry namespace.
Verify the Grafana Operator subscription is absent:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get sub --selector=operators.coreos.com/grafana-operator.service-telemetry
$ oc get sub --selector=operators.coreos.com/grafana-operator.service-telemetry No resources found in service-telemetry namespace.
Verify the Grafana Operator Cluster Service Version is absent:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get csv --selector=operators.coreos.com/grafana-operator.service-telemetry
$ oc get csv --selector=operators.coreos.com/grafana-operator.service-telemetry No resources found in service-telemetry namespace.
7.2. Upgrading Red Hat OpenShift Container Platform to 4.10
Service Telemetry Framework (STF) 1.5 is only compatible with Red Hat OpenShift Container Platform 4.10. For more information about upgrading your Red Hat OpenShift Container Platform from 4.8 to 4.10, see Updating clusters overview.
Prerequisites
- You removed the STF 1.4 Operators.
- You removed the AMQ Certificate Manager Operator and Grafana Operator. You must remove the Operators before you upgrade Red Hat OpenShift Container Platform because the Operator APIs are incompatible with 4.10. For more information about preparing your Red Hat OpenShift Container Platform for upgrade from 4.8 to 4.10, see Understanding OpenShift Container Platform updates.
Verify the suitability of your Red Hat OpenShift Container Platform upgrade:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc adm upgrade
$ oc adm upgrade
You cannot upgrade the cluster if you encounter the following error message:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: service-telemetry/grafana-operator.v3.10.3 is incompatible with OpenShift minor versions greater than 4.8,openshift-operators/amq7-cert-manager.v1.0.11 is incompatible with OpenShift minor versions greater than 4.8
Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: service-telemetry/grafana-operator.v3.10.3 is incompatible with OpenShift minor versions greater than 4.8,openshift-operators/amq7-cert-manager.v1.0.11 is incompatible with OpenShift minor versions greater than 4.8
7.3. Installing the Service Telemetry Framework 1.5 Operators
Install the Service Telemetry Framework (STF) 1.5 Operators and the Certificate Manager for OpenShift Operator on your Red Hat OpenShift Container Platform environment. See Section 1.1, “Support for Service Telemetry Framework” for more information about STF support status and life cycle.
Prerequisites
- You have upgraded your Red Hat OpenShift Container Platform environment to 4.10. For more information about upgrading Red Hat OpenShift Container Platform, see Section 7.2, “Upgrading Red Hat OpenShift Container Platform to 4.10”.
- Your Red Hat OpenShift Container Platform environment network is fully-connected.
Procedure
Change to the
service-telemetry
project:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc project service-telemetry
$ oc project service-telemetry
Create a
namespace
for thecert-manager
Operator:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc create -f - <<EOF apiVersion: project.openshift.io/v1 kind: Project metadata: name: openshift-cert-manager-operator spec: finalizers: - kubernetes EOF
$ oc create -f - <<EOF apiVersion: project.openshift.io/v1 kind: Project metadata: name: openshift-cert-manager-operator spec: finalizers: - kubernetes EOF
Create an
OperatorGroup
for the cert-manager Operator:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc create -f - <<EOF apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-cert-manager-operator namespace: openshift-cert-manager-operator spec: {} EOF
$ oc create -f - <<EOF apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-cert-manager-operator namespace: openshift-cert-manager-operator spec: {} EOF
Subscribe to the
cert-manager
Operator with theredhat-operators
CatalogSource:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc create -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-cert-manager-operator namespace: openshift-cert-manager-operator spec: channel: tech-preview installPlanApproval: Automatic name: openshift-cert-manager-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
$ oc create -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-cert-manager-operator namespace: openshift-cert-manager-operator spec: channel: tech-preview installPlanApproval: Automatic name: openshift-cert-manager-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
Validate your
ClusterServiceVersion
. Ensure that the phase ofcert-manager
Operator isSucceeded
:Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get csv --namespace openshift-cert-manager-operator --selector=operators.coreos.com/openshift-cert-manager-operator.openshift-cert-manager-operator
$ oc get csv --namespace openshift-cert-manager-operator --selector=operators.coreos.com/openshift-cert-manager-operator.openshift-cert-manager-operator NAME DISPLAY VERSION REPLACES PHASE openshift-cert-manager.v1.7.1 cert-manager Operator for Red Hat OpenShift 1.7.1-1 Succeeded
- Optional: Resubscribe to the Grafana Operator. For more information, see: test Section 5.1.1, “Configuring Grafana to host the dashboard”.
Create the Service Telemetry Operator subscription to manage the STF instances:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc create -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: service-telemetry-operator namespace: service-telemetry spec: channel: stable-1.5 installPlanApproval: Automatic name: service-telemetry-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
$ oc create -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: service-telemetry-operator namespace: service-telemetry spec: channel: stable-1.5 installPlanApproval: Automatic name: service-telemetry-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
Validate the Service Telemetry Operator and the dependent operators:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get csv --namespace service-telemetry
$ oc get csv --namespace service-telemetry NAME DISPLAY VERSION REPLACES PHASE amq7-interconnect-operator.v1.10.13 Red Hat Integration - AMQ Interconnect 1.10.13 amq7-interconnect-operator.v1.10.4 Succeeded elasticsearch-eck-operator-certified.v2.5.0 Elasticsearch (ECK) Operator 2.5.0 elasticsearch-eck-operator-certified.v2.4.0 Succeeded openshift-cert-manager.v1.7.1 cert-manager Operator for Red Hat OpenShift 1.7.1-1 Succeeded prometheusoperator.0.47.0 Prometheus Operator 0.47.0 prometheusoperator.0.37.0 Succeeded service-telemetry-operator.v1.5.1669950702 Service Telemetry Operator 1.5.1669950702 Succeeded smart-gateway-operator.v5.0.1669950681 Smart Gateway Operator 5.0.1669950681 Succeeded
Verification
Verify that the Service Telemetry Operator has successfully reconciled.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc logs -f --selector=name=service-telemetry-operator oc get pods
$ oc logs -f --selector=name=service-telemetry-operator [...] ----- Ansible Task Status Event StdOut (infra.watch/v1beta1, Kind=ServiceTelemetry, default/service-telemetry) ----- PLAY RECAP ********************************************************************* localhost : ok=115 changed=0 unreachable=0 failed=0 skipped=21 rescued=0 ignored=0 $ oc get pods NAME READY STATUS RESTARTS AGE alertmanager-default-0 3/3 Running 0 20h default-cloud1-ceil-event-smartgateway-6d57ffbbdc-5mrj8 2/2 Running 1 (3m42s ago) 4m21s default-cloud1-ceil-meter-smartgateway-67684d88c8-62mp7 3/3 Running 1 (3m43s ago) 4m20s default-cloud1-coll-event-smartgateway-66cddd5567-qb6p2 2/2 Running 1 (3m42s ago) 4m19s default-cloud1-coll-meter-smartgateway-76d5ff6db5-z5ch8 3/3 Running 0 4m20s default-cloud1-sens-meter-smartgateway-7b59669fdd-c42zg 3/3 Running 1 (3m43s ago) 4m20s default-interconnect-845c4b647c-wwfcm 1/1 Running 0 4m10s elastic-operator-57b57964c5-6q88r 1/1 Running 8 (17h ago) 20h elasticsearch-es-default-0 1/1 Running 0 21h grafana-deployment-59c54f7d7c-zjnhm 2/2 Running 0 20h interconnect-operator-848889bf8b-bq2zx 1/1 Running 0 20h prometheus-default-0 3/3 Running 1 (20h ago) 20h prometheus-operator-5d7b69fd46-c2xlv 1/1 Running 0 20h service-telemetry-operator-79fb664dfb-nj8jn 1/1 Running 0 5m11s smart-gateway-operator-79557664f8-ql7xr 1/1 Running 0 5m7s