Prepare the optimization framework

Review software requirements, supported features, and optimization strategies before deploying the Optimize service (watcher) in your Red Hat OpenStack Services on OpenShift (RHOSO) environment. Understanding these requirements helps you configure the service correctly and choose the right strategies for your infrastructure goals.

Optimization feature support
Copy link

To improve service reliability, use only the documented features and strategies when you implement the Optimize service (watcher) in your production environment.

Expand
Table 1. Optimize service (watcher) feature support matrix
Feature Supported in RHOSO 18.0

OneShot audits

Yes

Continuous audits

Yes

Event based audits

No

Host maintenance strategy

Yes

Node resource consolidation strategy

Yes

VM workload consolidation strategy

Yes

Workload balance migration strategy

Yes

Workload stabilization strategy

Yes

Zone migration strategy

Yes

Basic offline server consolidation strategy

No

Noisy neighbor strategy

No

Outlet temperature based strategy

No

Saving energy strategy

No

Storage capacity balance strategy

No

Uniform airflow migration strategy

No

Actuator strategy

No

Known limitations
Copy link

Workflow Engine does not revert actions for failed Action Plans

In RHOSO 18, the Optimize service (watcher) Engine does not automatically revert failed actions when an Action Plan fails, even when configured to do so by enabling the watcher_applier.rollback_when_actionplan_failed configuration option.

Workaround: Manually revert each failed action in the Action Plan. To avoid the rollback, you can diagnose and fix the root cause of the failure and then run the Audit again to propose a new solution.

Instance migration destination discovery limitations because of missing metadata

Migration constraints can cause Watcher live migration actions to fail.

Additionally, limited data integration between Watcher and other service elements, such as Compute service instances, flavors, and host aggregates, restrict compatibility checking during migration destination discovery. This can result in Action Plans that assign migration actions to unsuitable nodes. This limitation affects environments by using specific features:

  • Affinity and anti-affinity rules such as server groups, same or different host targeting, tenant affinity, network affinity, and so on.
  • Pinned to an availability zone instances.
  • NUMA and CPU specific requirements, such as PCI NUMA affinity, CPU pinning, or huge pages.
  • PCI passthrough, SR-IOV devices, or vGPUs.
  • Specific compute node rules through require or forbid traits, custom resource classes or standard resource overrides, host aggregate matching, compute capabilities matching, and so on.
  • CPU architecture or hypervisor version filtering.
  • Instances that have shares.
  • vTPM.
  • Instances with locks.
  • Mixed RHEL deployments, where Compute nodes can run different minor version of Red Hat Enterprise Linux.

    In these scenarios watcher migration actions fail because the Compute service (nova) checks and enforces all constraints before migration.

    This limitation applies to audits based on the following strategies:

  • Node Resource Consolidation: When the user-supplied parameter host_choice is not set to auto.
  • VM Workload Consolidation
  • Workload Stabilization
  • Workload Balance
Action failures after a user specifies a destination hosts
When you create an audit with a strategy such as zone_migration or hardware_maintenance wherein you specify the intended destination or backup Compute node, the Optimize service does not validate your selection against Compute service constraints. This can lead to a failed migration action.
Action failures after topology changes
Because the action plan execution is an asynchronous operation, it is possible that the status of the cluster can change between the creation and execution of a plan. For example, the plan might include the migration of an instance that has been deleted or manually stopped. When this occurs, a Watcher action fails.
Multiple Availability Zones deployments
When the cross-az feature is disabled in the Compute service, an instance cannot run in a different availability zone than the one used by its attached Block Storage service (cinder) volumes. Watcher does not validate this constraint during its audit process and might plan a migration to a host located in a different availability zone, which results in failure.
Horizontal scalability on Watcher decision-engine and applier services
Due to architectural limitations of Watcher in RHOSO 18, it is not possible to run more than one instance of the Watcher decision-engine and applier services. Running more than one instance might lead to unfinished actions and audits being orphaned in scale-down scenarios.

Optimization software requirements
Copy link

Verify that your Red Hat OpenStack Services on OpenShift (RHOSO) environment meets the necessary prerequisites before you deploy the Optimize service (watcher). Ensuring you have the required services and the required version helps prevent deployment issues and ensures the Optimize service functions correctly.

The Red Hat OpenStack Services on OpenShift (RHOSO) Optimize service (watcher) requires that you have an existing deployment of RHOSO 18.0.6 or later that includes, at least, the following components:

  • Compute service (nova)
  • Identity service (keystone)
  • Image service (glance)
  • MariaDB
  • RabbitMQ
  • Telemetry service, including the Prometheus metrics store

The Dashboard service (horizon) is not required. However, if it is enabled in the OpenStack cloud, the Optimize service dashboard is available in the Dashboard service post deployment.

Synchronize Optimize service data with real-time notifications
Copy link

Enable service notifications to keep the Optimize service (watcher) internal model synchronized with your infrastructure in real time. Configuring notifications prevents the service from working with outdated cluster information when creating action plans for large environments.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges.
  • You have the oc command line tool installed on your workstation.

The Optimize service (watcher) has an internal model of your cluster that it uses to create action plans. By default, the Optimize service queries for updates every 15 minutes. In large clusters, the internal model might be out of date when an action plan is created.

You can ensure that the internal model used by the Optimize (watcher) service is always up to date by enabling notifications to be sent from the Compute (nova) and Storage (cinder) services, and then enabling Optimize to receive those notifications.

Procedure
Copy link

  1. On your workstation, open your OpenStackControlPlane custom resource (CR) file called openstack_control_plane.yaml.
  2. Add a new messaging bus instance for your cluster: 
    spec:
    rabbitmq:
        enabled: true
        templates:
        ...
            <rabbitmq_notification_server>:
                delayStartSeconds: 30
                override:
                    service:
                        metadata:
                            annotations:
                                metallb.universe.tf/address-pool: internalapi
                                metallb.universe.tf/loadBalancerIPs: <ip_address>
                        spec:
                            type: LoadBalancer
    • Replace <rabbitmq_notification_server> with the name of your notification server, for example, rabbitmq-notifications.
    • Replace <ip_address> with the appropriate ip address based on your networking plan and configuration.
  3. Enable the notifications in your cluster. This change enables all services supporting notifications to send or consume notifications: 
    spec:
  notificationsBusInstance: <rabbitmq_bus_name>
    • Replace <rabbitmq_bus_name> with the name of your notification bus, for example, rabbitmq-notifications.
  4. Save your changes to the openstack_control_plane.yaml CR file.
  5. Update the control plane: 
    $ oc apply -f openstack_control_plane.yaml -n openstack