Chapter 1. Configuring observability

Procedure

1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, on your workstation.

2. Configure the Telemetry service, telemetry, as required for your environment:

     telemetry:
       enabled: true
       template:
         metricStorage:
           enabled: true
           dashboardsEnabled: true
           dataplaneNetwork: ctlplane
           networkAttachments:
             - ctlplane
           monitoringStack:
             alertingEnabled: true
             scrapeInterval: 30s
             storage:
               strategy: persistent
               retention: 24h
               persistent:
                 pvcStorageRequest: 20G
         autoscaling:
           enabled: false
           aodh:
             notificationsBus:
               cluster: rabbitmq-notification
             databaseAccount: aodh
             databaseInstance: openstack
             secret: osp-secret
           heatInstance: heat
         ceilometer:
           enabled: true
           notificationsBus:
             cluster: rabbitmq-notification
           secret: osp-secret
         logging:
           enabled: false
           annotations:
             metallb.universe.tf/address-pool: internalapi
             metallb.universe.tf/allow-shared-ip: internalapi
             metallb.universe.tf/loadBalancerIPs: 172.17.0.80
         cloudkitty:
           enabled: false
           messagingBus:
             cluster: rabbitmq
           s3StorageConfig:
             schemas:
             - effectiveDate: "2024-11-18"
               version: v13
             secret:
               name: cloudkitty-loki-s3
               type: s3

   • metricStorage.monitoringStack.scrapeInterval: Specifies the interval at which new metrics are gathered. Changing this interval can affect performance.
   • metricStorage.monitoringStack.storage.retention: Specifies the length of time that telemetry metrics are stored. The duration affects the amount of storage required.
   • storage.persistent.pvcStorageRequest: Specifies the amount of storage to allocate to the Prometheus time series database.
   • autoscaling.enabled: Set to true to enable autoscaling. The autoscaling field must be present even when autoscaling is disabled. For more information about autoscaling, see Autoscaling for Instances.
   • ceilometer.enabled: Set to false to disable the ceilometer service. If you do not disable ceilometer, then a Prometheus metrics exporter is created and exposed from inside the cluster at the following URL: http://ceilometer-internal.openstack.svc:3000/metrics
   • logging.enabled: Set to true to enable observability logging. For more information about configuring observability logging, see Enabling RHOSO observability logging.
   • cloudkitty.enabled: Set to true to enable the Rating service (cloudkitty). For more information about configuring chargeback and rating capabilities, see Enabling the Rating service on the control plane.
   • aodh.notificationsBus.cluster and ceilometer.notificationsBus.cluster: Set to a dedicated RabbitMQ cluster for notifications as in this example or to a combined RabbitMQ cluster for both RPC and notifications as in the default RHOSO deployment that uses the combined rabbitmq cluster.

3. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

4. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack
   NAME 						STATUS 	MESSAGE
   openstack-control-plane 	Unknown 	Setup started

   The OpenStackControlPlane resources are created when the status is "Setup complete".

   Tip

   Append the -w option to the end of the get command to track deployment progress.

5. Optional: Confirm that the control plane is deployed by reviewing the pods in the openstack namespace for each of your cells:

   $ oc get pods -n openstack

   The control plane is deployed when all the pods are either completed or running.

Verification

1. Access the remote shell for the OpenStackClient pod from your workstation:

   $ oc rsh -n openstack openstackclient

2. Confirm that you can query prometheus and that the scrape endpoints are active:

   $ openstack metric query up --disable-rbac -c container -c instance  -c value

   Example output:

   +-----------------+------------------------+-------+
   | container       | instance               | value |
   +-----------------+------------------------+-------+
   | alertmanager    | 10.217.1.112:9093	   | 1     |
   | prometheus      | 10.217.1.63:9090 	   | 0     |
   | proxy-httpd     | 10.217.1.52:3000       | 1     |
   |                 | 192.168.122.100:9100   | 1     |
   |                 | 192.168.122.101:9100   | 1     |
   +-----------------+------------------------+-------+

   Note

   Each entry in the value field should be "1" when there are active workloads scheduled on the cluster, except for the prometheus container. The prometheus container reports a value of "0" due to TLS, which is enabled by default.

3. You can find the openstack-telemetry-operator dashboards by clicking Observe and then Dashboards in the RHOCP console. For more information about RHOCP dashboards, see Reviewing monitoring dashboards as a cluster administrator in the RHOCP Monitoring Guide.

Procedure

1. Create the custom configuration file for the service you want to customize. If the service already has a configuration file, then you must give your custom file the same name so that your custom file replaces the existing configuration file. For example, to customize the configuration of the Ceilometer service, you can make a copy of polling.yaml for editing:

   $ oc rsh -n openstack -c ceilometer-central-agent ceilometer-0 cat /etc/ceilometer/polling.yaml > polling.yaml

2. Add or update the configuration for the service as required. For example, to customize how often samples of volume and image size should be polled, create a file named polling.yaml and add the following configuration:

   sources:
     - name: pollsters
       interval: 300
       meters:
         - volume.size
         - image.size

   For more information on how to configure the Ceilometer service, see Polling properties.

3. Create the Secret CR for the service with the custom configuration file:

   $ oc create secret generic <secret_name> \
    --from-file <custom_config.yaml> -n openstack

   Note

   You can specify the --from-file option as many times as required to pass more than one configuration file to the Secret CR for customizing the service.

4. Verify that the Secret CR is created:

   $ oc describe secret <secret_name>

5. Open the OpenStackControlPlane CR file on your workstation, for example, openstack_control_plane.yaml.

6. Locate the service definition for telemetry and add the customConfigsSecretName field to the Telemetry service that you want to customize. The following example shows where to place the customConfigsSecretName field for each service that you can customize:

   spec:
     telemetry:
       template:
         autoscaling:
           aodh:
             customConfigsSecretName: <secret_name>
         ceilometer:
           customConfigsSecretName: <secret_name>
         cloudkitty:
           cloudKittyAPI:
             customConfigsSecretName: <secret_name>
           cloudKittyProc:
             customConfigsSecretName: <secret_name>

7. Update the control plane:

   $ oc apply -f openstack_control_plane.yaml -n openstack

   Your custom file is copied from the Secret into the /etc/<service>/ folder. If a file with the same name already exists in the folder, it is replaced with the custom configuration file.

8. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status:

   $ oc get openstackcontrolplane -n openstack

9. Verify that the custom configuration file is being used by the service:

   $ oc rsh -c <service_container> <service_pod> \
    cat /etc/<service>/<custom_config>.yaml

   Example:

   $ oc rsh -c ceilometer-central-agent ceilometer-0 cat /etc/ceilometer/polling.yaml

Procedure

1. Open the OpenStackDataPlaneNodeSet CR definition file for the node set you want to update, for example, openstack_data_plane.yaml.

2. Add the services field, and include all the required services, including the default services, then add telemetry-power-monitoring after telemetry:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
   metadata:
     name: openstack-data-plane
     namespace: openstack
   spec:
     tlsEnabled: true
     env:
       - name: ANSIBLE_FORCE_COLOR
         value: "True"
     services:
       - redhat
       - bootstrap
       - download-cache
       - configure-network
       - validate-network
       - install-os
       - configure-os
       - ssh-known-hosts
       - run-os
       - reboot-os
       - install-certs
       - ovn
       - neutron-metadata
       - libvirt
       - nova
       - telemetry
       - telemetry-power-monitoring

   For more information about deploying data plane services, see Deploying the data plane in the Deploying Red Hat OpenStack Services on OpenShift guide.

3. Save the OpenStackDataPlaneNodeSet CR definition file.

4. Apply the updated OpenStackDataPlaneNodeSet CR configuration:

   $ oc apply -f openstack_data_plane.yaml

5. Verify that the data plane resource has been updated by confirming that the status is SetupReady:

   $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

   When the status is SetupReady the command returns a condition met message, otherwise it returns a timeout error.

   For information about the data plane conditions and states, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

6. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
   Tip

   Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

7. Add the OpenStackDataPlaneNodeSet CR that you modified:

   spec:
     nodeSets:
       - <nodeSet_name>

8. Save the OpenStackDataPlaneDeployment CR deployment file.

9. Deploy the modified OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

10. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

    $ oc get openstackdataplanedeployment -n openstack
    NAME             	STATUS   MESSAGE
    openstack-data-plane   True     Setup Complete
    
    
    $ oc get openstackdataplanenodeset -n openstack
    NAME             	STATUS   MESSAGE
    openstack-data-plane   True     NodeSet Ready

    For information about the meaning of the returned status, see Data plane conditions and states in the Deploying Red Hat OpenStack Services on OpenShift guide.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in the Deploying Red Hat OpenStack Services on OpenShift guide.

11. Verify that the telemetry-power-monitoring service is deployed by checking for ceilometer_agent_ipmi and kepler containers in the data plane nodes:

    $ podman ps | grep -i -e ceilometer_agent_ipmi -e kepler