Chapter 1. Configuring observability

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, on your workstation.

2. Configure the Telemetry service, telemetry, as required for your environment:

     telemetry:
       enabled: true
       template:
         metricStorage:
           enabled: true
           dashboardsEnabled: true
           dataplaneNetwork: ctlplane
           networkAttachments:
             - ctlplane
           monitoringStack:
             alertingEnabled: true
             scrapeInterval: 30s
             storage:
               strategy: persistent
               retention: 24h
               persistent:
                 pvcStorageRequest: 20G
         autoscaling:
           enabled: false
           aodh:
             databaseAccount: aodh
             databaseInstance: openstack
             secret: osp-secret
           heatInstance: heat
         ceilometer:
           enabled: true
           secret: osp-secret
         logging:
           enabled: false
           annotations:
             metallb.universe.tf/address-pool: internalapi
             metallb.universe.tf/allow-shared-ip: internalapi
             metallb.universe.tf/loadBalancerIPs: 172.17.0.80
         cloudkitty:
           enabled: false
           s3StorageConfig:
             schemas:
             - effectiveDate: "2024-11-18"
               version: v13
             secret:
               name: cloudkitty-loki-s3
               type: s3

   Copy to Clipboard Toggle word wrap
   • metricStorage.monitoringStack.scrapeInterval: Specifies the interval at which new metrics are gathered. Changing this interval can affect performance.
   • metricStorage.monitoringStack.storage.retention: Specifies the length of time that telemetry metrics are stored. The duration affects the amount of storage required.
   • storage.persistent.pvcStorageRequest: Specifies the amount of storage to allocate to the Prometheus time series database.
   • autoscaling.enabled: Set to true to enable autoscaling. The autoscaling field must be present even when autoscaling is disabled. For more information about autoscaling, see Autoscaling for Instances.
   • ceilometer.enabled: Set to false to disable the ceilometer service. If you do not disable ceilometer, then a Prometheus metrics exporter is created and exposed from inside the cluster at the following URL: http://ceilometer-internal.openstack.svc:3000/metrics
   • logging.enabled: Set to true to enable observability logging. For more information about configuring observability logging, see Enabling RHOSO observability logging.
   • cloudkitty.enabled: Set to true to enable the Rating service (cloudkitty). For more information about configuring chargeback and rating capabilities, see Enabling the Rating service on the control plane.

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

   Copy to Clipboard Toggle word wrap

4. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack
   NAME 						STATUS 	MESSAGE
   openstack-control-plane 	Unknown 	Setup started

   Copy to Clipboard Toggle word wrap

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

5. Optional: Confirm that the control plane is deployed by reviewing the pods in the openstack namespace for each of your cells:

   $ oc get pods -n openstack

   Copy to Clipboard Toggle word wrap

   The control plane is deployed when all the pods are either completed or running.

Verification

1. Access the remote shell for the OpenStackClient pod from your workstation:

   $ oc rsh -n openstack openstackclient

   Copy to Clipboard Toggle word wrap

2. Confirm that you can query prometheus and that the scrape endpoints are active:

   $ openstack metric query up --disable-rbac -c container -c instance  -c value

   Copy to Clipboard Toggle word wrap

   Example output:

   +-----------------+------------------------+-------+
   | container       | instance               | value |
   +-----------------+------------------------+-------+
   | alertmanager    | 10.217.1.112:9093	   | 1     |
   | prometheus      | 10.217.1.63:9090 	   | 0     |
   | proxy-httpd     | 10.217.1.52:3000       | 1     |
   |                 | 192.168.122.100:9100   | 1     |
   |                 | 192.168.122.101:9100   | 1     |
   +-----------------+------------------------+-------+

   Copy to Clipboard Toggle word wrap
   Note

   Each entry in the value field should be "1" when there are active workloads scheduled on the cluster, except for the prometheus container. The prometheus container reports a value of "0" due to TLS, which is enabled by default.

3. You can find the openstack-telemetry-operator dashboards by clicking Observe and then Dashboards in the RHOCP console. For more information about RHOCP dashboards, see Reviewing monitoring dashboards as a cluster administrator in the RHOCP Monitoring Guide.

Procedure

1. Open the OpenStackDataPlaneNodeSet CR definition file for the node set you want to update, for example, openstack_data_plane.yaml.

2. Add the services field, and include all the required services, including the default services, then add telemetry-power-monitoring after telemetry:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-data-plane
     namespace: openstack
   spec:
     tlsEnabled: true
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"
     services:
       - redhat
       - bootstrap
       - download-cache
       - configure-network
       - validate-network
       - install-os
       - configure-os
       - ssh-known-hosts
       - run-os
       - reboot-os
       - install-certs
       - ovn
       - neutron-metadata
       - libvirt
       - nova
       - telemetry
       - telemetry-power-monitoring

   Copy to Clipboard Toggle word wrap

   For more information about deploying data plane services, see Deploying the data plane in the Deploying Red Hat OpenStack Services on OpenShift guide.

3. Save the OpenStackDataPlaneNodeSet CR definition file.

4. Apply the updated OpenStackDataPlaneNodeSet CR configuration:

   $ oc apply -f openstack_data_plane.yaml

   Copy to Clipboard Toggle word wrap

5. Verify that the data plane resource has been updated by confirming that the status is SetupReady:

   $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

   Copy to Clipboard Toggle word wrap

   When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

   For information about the data plane conditions and states, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

6. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   Copy to Clipboard Toggle word wrap
   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
   Tip

   Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

7. Add the OpenStackDataPlaneNodeSet CR that you modified:

   spec:
     nodeSets:
       - <nodeSet_name>

   Copy to Clipboard Toggle word wrap
8. Save the OpenStackDataPlaneDeployment CR deployment file.

9. Deploy the modified OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   Copy to Clipboard Toggle word wrap

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   Copy to Clipboard Toggle word wrap

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

   Copy to Clipboard Toggle word wrap

10. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

    $ oc get openstackdataplanedeployment -n openstack
    NAME             	STATUS   MESSAGE
    openstack-data-plane   True     Setup Complete
    
    
    $ oc get openstackdataplanenodeset -n openstack
    NAME             	STATUS   MESSAGE
    openstack-data-plane   True     NodeSet Ready

    Copy to Clipboard Toggle word wrap

    For information about the meaning of the returned status, see Data plane conditions and states in the Deploying Red Hat OpenStack Services on OpenShift guide.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in the Deploying Red Hat OpenStack Services on OpenShift guide.

11. Verify that the telemetry-power-monitoring service is deployed by checking for ceilometer_agent_ipmi and kepler containers in the data plane nodes:

    $ podman ps | grep -i -e ceilometer_agent_ipmi -e kepler

    Copy to Clipboard Toggle word wrap