Chapter 3. Policy controller advanced configuration

Configure the concurrency of the governance framework for each managed cluster. To change the default value of 2, set the policy-evaluation-concurrency annotation with a nonzero integer within quotation marks. Then set the value on the ManagedClusterAddOn object name to governance-policy-framework in the managed cluster namespace of the hub cluster.

See the following YAML example where the concurrency is set to 2 on the managed cluster named cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: governance-policy-framework
  namespace: cluster1
  annotations:
    policy-evaluation-concurrency: "2"
spec:
  installNamespace: open-cluster-management-agent-addon

To set the client-qps and client-burst annotations, update the ManagedClusterAddOn resource and define the parameters.

See the following YAML example where the queries for each second is set to 30 and the burst is set to 45 on the managed cluster called cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: governance-policy-framework
  namespace: cluster1
  annotations:
    client-qps: "30"
    client-burst: "45"
spec:
  installNamespace: open-cluster-management-agent-addon

You can configure the concurrency of the configuration policy controller for each managed cluster to change how many configuration policies it can evaluate at the same time. To change the default value of 2, set the policy-evaluation-concurrency annotation with a nonzero integer within quotation marks. Then set the value on the ManagedClusterAddOn object name to config-policy-controller in the managed cluster namespace of the hub cluster.

Note: Increased concurrency values increase CPU and memory utilization on the config-policy-controller pod, Kubernetes API server, and OpenShift API server.

See the following YAML example where the concurrency is set to 5 on the managed cluster named cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: cluster1
  annotations:
    policy-evaluation-concurrency: "5"
spec:
  installNamespace: open-cluster-management-agent-addon

Configure the rate of requests to the API server that the configuration policy controller makes on each managed cluster. An increased rate improves the responsiveness of the configuration policy controller, which also increases the CPU and memory utilization of the Kubernetes API server and OpenShift API server. By default, the rate of requests scales with the policy-evaluation-concurrency setting and is set to 30 queries for each second (QPS), with a 45 burst value, representing a higher number of requests over short periods of time.

You can configure the rate and burst by setting the client-qps and client-burst annotations with nonzero integers within quotation marks. You can set the value on the ManagedClusterAddOn object name to config-policy-controller in the managed cluster namespace of the hub cluster.

See the following YAML example where the queries for each second is set to 20 and the burst is set to 100 on the managed cluster called cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: cluster1
  annotations:
    client-qps: "20"
    client-burst: "100"
spec:
  installNamespace: open-cluster-management-agent-addon

When you configure and collect debug logs for each policy controller, you can adjust the log level.

Note: Reducing the volume of debug logs means there is less information displayed from the logs.

You can reduce the debug logs emitted by the policy controllers to be display error-only bugs in the logs. To reduce the debug logs, set the debug log value to -1 in the annotation. See what each value represents:

To receive the second level of debugging information for the Kubernetes configuration controller, add the log-level annotation with the value of 2 to the ManagedClusterAddOn custom resource. By default, the log-level is set to 0, which means you receive informative messages. View the following example:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: cluster1
  annotations:
    log-level: "2"
spec:
  installNamespace: open-cluster-management-agent-addon

Additionally, for each spec.object-template[] in a ConfigurationPolicy resource, you can set the parameter recordDiff to Log. The difference between the objectDefinition and the object on the managed cluster is logged in the config-policy-controller pod on the managed cluster. View the following example:

This ConfigurationPolicy resource with recordDiff: Log:

apiVersion: policy.open-cluster-management.io/v1
kind: ConfigurationPolicy
metadata:
  name: my-config-policy
spec:
  object-templates:
  - complianceType: musthave
    recordDiff: Log
    objectDefinition:
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: my-configmap
      data:
        fieldToUpdate: "2"

If the ConfigMap resource on the cluster lists fieldToUpdate: "1", then the diff appears in the config-policy-controller pod logs with the following information:

Logging the diff:
--- default/my-configmap : existing
+++ default/my-configmap : updated
@@ -2,3 +2,3 @@
 data:
-  fieldToUpdate: "1"
+  fieldToUpdate: "2"
 kind: ConfigMap

Important: Avoid logging the difference for a secure object. The difference is logged in plain text.

The policy framework exposes metrics that show policy distribution and compliance. Use the policy_governance_info metric on the hub cluster to view trends and analyze any policy failures. See the following topics for an overview of metrics:

When you apply the new configuration with the controller, the ManifestApplied parameter is updated in the ManagedClusterAddOn. That condition timestamp helps verify the configuration correctly. For example, this command can verify when the cert-policy-controller on the local-cluster was updated:

oc get -n local-cluster managedclusteraddon cert-policy-controller | grep -B4 'type: ManifestApplied'

You might receive the following output:

 - lastTransitionTime: "2023-01-26T15:42:22Z"
    message: manifests of addon are applied successfully
    reason: AddonManifestApplied
    status: "True"
    type: ManifestApplied