Chapter 3. Policy controller advanced configuration


You can customize policy controller configurations on your managed clusters by using the ManagedClusterAddOn custom resources. The following ManagedClusterAddOns configure the policy framework, Kubernetes configuration policy controller, and the Certificate policy controller. Required access: Cluster administrator

3.1. Configure the concurrency of the governance framework

Configure the concurrency of the governance framework for each managed cluster. To change the default value of 2, set the policy-evaluation-concurrency annotation with a nonzero integer within quotation marks. Then set the value on the ManagedClusterAddOn object name to governance-policy-framework in the managed cluster namespace of the hub cluster.

See the following YAML example where the concurrency is set to 2 on the managed cluster named cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: governance-policy-framework
  namespace: cluster1
  annotations:
    policy-evaluation-concurrency: "2"
spec:
  installNamespace: open-cluster-management-agent-addon

To set the client-qps and client-burst annotations, update the ManagedClusterAddOn resource and define the parameters.

See the following YAML example where the queries for each second is set to 30 and the burst is set to 45 on the managed cluster called cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: governance-policy-framework
  namespace: cluster1
  annotations:
    client-qps: "30"
    client-burst: "45"
spec:
  installNamespace: open-cluster-management-agent-addon

3.2. Configure the concurrency of the configuration policy controller

You can configure the concurrency of the configuration policy controller for each managed cluster to change how many configuration policies it can evaluate at the same time. To change the default value of 2, set the policy-evaluation-concurrency annotation with a nonzero integer within quotation marks. Then set the value on the ManagedClusterAddOn object name to config-policy-controller in the managed cluster namespace of the hub cluster.

Note: Increased concurrency values increase CPU and memory utilization on the config-policy-controller pod, Kubernetes API server, and OpenShift API server.

See the following YAML example where the concurrency is set to 5 on the managed cluster named cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: cluster1
  annotations:
    policy-evaluation-concurrency: "5"
spec:
  installNamespace: open-cluster-management-agent-addon

3.3. Configure the rate of requests to the API server

Configure the rate of requests to the API server that the configuration policy controller makes on each managed cluster. An increased rate improves the responsiveness of the configuration policy controller, which also increases the CPU and memory utilization of the Kubernetes API server and OpenShift API server. By default, the rate of requests scales with the policy-evaluation-concurrency setting and is set to 30 queries for each second (QPS), with a 45 burst value, representing a higher number of requests over short periods of time.

You can configure the rate and burst by setting the client-qps and client-burst annotations with nonzero integers within quotation marks. You can set the value on the ManagedClusterAddOn object name to config-policy-controller in the managed cluster namespace of the hub cluster.

See the following YAML example where the queries for each second is set to 20 and the burst is set to 100 on the managed cluster called cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: cluster1
  annotations:
    client-qps: "20"
    client-burst: "100"
spec:
  installNamespace: open-cluster-management-agent-addon

3.4. Configure debug log

When you configure and collect debug logs for each policy controller, you can adjust the log level.

Note: Reducing the volume of debug logs means there is less information displayed from the logs.

You can reduce the debug logs emitted by the policy controllers to be display error-only bugs in the logs. To reduce the debug logs, set the debug log value to -1 in the annotation. See what each value represents:

  • -1: error logs only
  • 0: informative logs
  • 1: debug logs
  • 2: verbose debugging logs

To receive the second level of debugging information for the Kubernetes configuration controller, add the log-level annotation with the value of 2 to the ManagedClusterAddOn custom resource. By default, the log-level is set to 0, which means you receive informative messages. View the following example:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: cluster1
  annotations:
    log-level: "2"
spec:
  installNamespace: open-cluster-management-agent-addon

Additionally, for each spec.object-template[] in a ConfigurationPolicy resource, you can set the parameter recordDiff to Log. The difference between the objectDefinition and the object on the managed cluster is logged in the config-policy-controller pod on the managed cluster. View the following example:

This ConfigurationPolicy resource with recordDiff: Log:

apiVersion: policy.open-cluster-management.io/v1
kind: ConfigurationPolicy
metadata:
  name: my-config-policy
spec:
  object-templates:
  - complianceType: musthave
    recordDiff: Log
    objectDefinition:
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: my-configmap
      data:
        fieldToUpdate: "2"

If the ConfigMap resource on the cluster lists fieldToUpdate: "1", then the diff appears in the config-policy-controller pod logs with the following information:

Logging the diff:
--- default/my-configmap : existing
+++ default/my-configmap : updated
@@ -2,3 +2,3 @@
 data:
-  fieldToUpdate: "1"
+  fieldToUpdate: "2"
 kind: ConfigMap

Important: Avoid logging the difference for a secure object. The difference is logged in plain text.

3.5. Governance metric

The policy framework exposes metrics that show policy distribution and compliance. Use the policy_governance_info metric on the hub cluster to view trends and analyze any policy failures. See the following topics for an overview of metrics:

3.5.1. Metric: policy_governance_info

The OpenShift Container Platform monitoring component collects the policy_governance_info metric. If you enable observability, the component collects some aggregate data.

Note: If you enable observability, enter a query for the metric from the Grafana Explore page. When you create a policy, you are creating a root policy. The framework watches for root policies, Placement resources, and PlacementBindings resources to for information about where to create propagated policies, to distribute the policy to managed clusters.

For both root and propagated policies, a metric of 0 is recorded if the policy is compliant, 1 if it is non-compliant, and -1 if it is in an unknown or pending state.

The policy_governance_info metric uses the following labels:

  • type: The label values are root or propagated.
  • policy: The name of the associated root policy.
  • policy_namespace: The namespace on the hub cluster where the root policy is defined.
  • cluster_namespace: The namespace for the cluster where the policy is distributed.

These labels and values enable queries that can show us many things happening in the cluster that might be difficult to track.

Note: If you do not need the metrics, and you have any concerns about performance or security, you can disable the metric collection. Set the DISABLE_REPORT_METRICS environment variable to true in the propagator deployment. You can also add policy_governance_info metric to the observability allowlist as a custom metric. See Adding custom metrics for more details.

3.5.2. Metric: config_policies_evaluation_duration_seconds

The config_policies_evaluation_duration_seconds histogram tracks the number of seconds it takes to process all configuration policies that are ready to be evaluated on the cluster. Use the following metrics to query the histogram:

  • config_policies_evaluation_duration_seconds_bucket: The buckets are cumulative and represent seconds with the following possible entries: 1, 3, 9, 10.5, 15, 30, 60, 90, 120, 180, 300, 450, 600, and greater.
  • config_policies_evaluation_duration_seconds_count: The count of all events.
  • config_policies_evaluation_duration_seconds_sum: The sum of all values.

Use the config_policies_evaluation_duration_seconds metric to determine if the ConfigurationPolicy evaluationInterval setting needs to be changed for resource intensive policies that do not need frequent evaluation. You can also increase the concurrency at the cost of higher resource utilization on the Kubernetes API server. See Configure the concurrency section for more details.

To receive information about the time used to evaluate configuration policies, perform a Prometheus query that resembles the following expression:

rate(config_policies_evaluation_duration_seconds_sum[10m])/rate (config_policies_evaluation_duration_seconds_count[10m]

The config-policy-controller pod running on managed clusters in the open-cluster-management-agent-addon namespace calculates the metric. The config-policy-controller does not send the metric to observability by default.

3.6. Verify configuration changes

When you apply the new configuration with the controller, the ManifestApplied parameter is updated in the ManagedClusterAddOn. That condition timestamp helps verify the configuration correctly. For example, this command can verify when the cert-policy-controller on the local-cluster was updated:

oc get -n local-cluster managedclusteraddon cert-policy-controller | grep -B4 'type: ManifestApplied'

You might receive the following output:

 - lastTransitionTime: "2023-01-26T15:42:22Z"
    message: manifests of addon are applied successfully
    reason: AddonManifestApplied
    status: "True"
    type: ManifestApplied

3.7. Additional resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.