Chapter 3. Policy controller advanced configuration
You can customize policy controller configurations on your managed clusters by using the ManagedClusterAddOn
custom resources. The following ManagedClusterAddOns
configure the policy framework, Kubernetes configuration policy controller, and the Certificate policy controller. Required access: Cluster administrator
3.1. Configure the concurrency of the governance framework
Configure the concurrency of the governance framework for each managed cluster. To change the default value of 2
, set the policy-evaluation-concurrency
annotation with a nonzero integer within quotation marks. Then set the value on the ManagedClusterAddOn
object name to governance-policy-framework
in the managed cluster namespace of the hub cluster.
See the following YAML example where the concurrency is set to 2
on the managed cluster named cluster1
:
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: governance-policy-framework namespace: cluster1 annotations: policy-evaluation-concurrency: "2" spec: installNamespace: open-cluster-management-agent-addon
To set the client-qps
and client-burst
annotations, update the ManagedClusterAddOn
resource and define the parameters.
See the following YAML example where the queries for each second is set to 30
and the burst is set to 45
on the managed cluster called cluster1
:
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: governance-policy-framework namespace: cluster1 annotations: client-qps: "30" client-burst: "45" spec: installNamespace: open-cluster-management-agent-addon
3.2. Configure the concurrency of the configuration policy controller
You can configure the concurrency of the configuration policy controller for each managed cluster to change how many configuration policies it can evaluate at the same time. To change the default value of 2
, set the policy-evaluation-concurrency
annotation with a nonzero integer within quotation marks. Then set the value on the ManagedClusterAddOn
object name to config-policy-controller
in the managed cluster namespace of the hub cluster.
Note: Increased concurrency values increase CPU and memory utilization on the config-policy-controller
pod, Kubernetes API server, and OpenShift API server.
See the following YAML example where the concurrency is set to 5
on the managed cluster named cluster1
:
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: config-policy-controller namespace: cluster1 annotations: policy-evaluation-concurrency: "5" spec: installNamespace: open-cluster-management-agent-addon
3.3. Configure the rate of requests to the API server
Configure the rate of requests to the API server that the configuration policy controller makes on each managed cluster. An increased rate improves the responsiveness of the configuration policy controller, which also increases the CPU and memory utilization of the Kubernetes API server and OpenShift API server. By default, the rate of requests scales with the policy-evaluation-concurrency
setting and is set to 30
queries for each second (QPS), with a 45
burst value, representing a higher number of requests over short periods of time.
You can configure the rate and burst by setting the client-qps
and client-burst
annotations with nonzero integers within quotation marks. You can set the value on the ManagedClusterAddOn
object name to config-policy-controller
in the managed cluster namespace of the hub cluster.
See the following YAML example where the queries for each second is set to 20
and the burst is set to 100
on the managed cluster called cluster1
:
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: config-policy-controller namespace: cluster1 annotations: client-qps: "20" client-burst: "100" spec: installNamespace: open-cluster-management-agent-addon
3.4. Configure debug log
When you configure and collect debug logs for each policy controller, you can adjust the log level.
Note: Reducing the volume of debug logs means there is less information displayed from the logs.
You can reduce the debug logs emitted by the policy controllers to be display error-only bugs in the logs. To reduce the debug logs, set the debug log value to -1
in the annotation. See what each value represents:
-
-1
: error logs only -
0
: informative logs -
1
: debug logs -
2
: verbose debugging logs
To receive the second level of debugging information for the Kubernetes configuration controller, add the log-level
annotation with the value of 2
to the ManagedClusterAddOn
custom resource. By default, the log-level
is set to 0
, which means you receive informative messages. View the following example:
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: config-policy-controller namespace: cluster1 annotations: log-level: "2" spec: installNamespace: open-cluster-management-agent-addon
Additionally, for each spec.object-template[]
in a ConfigurationPolicy
resource, you can set the parameter recordDiff
to Log
. The difference between the objectDefinition
and the object on the managed cluster is logged in the config-policy-controller
pod on the managed cluster. View the following example:
This ConfigurationPolicy
resource with recordDiff: Log
:
apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: my-config-policy spec: object-templates: - complianceType: musthave recordDiff: Log objectDefinition: apiVersion: v1 kind: ConfigMap metadata: name: my-configmap data: fieldToUpdate: "2"
If the ConfigMap
resource on the cluster lists fieldToUpdate: "1"
, then the diff appears in the config-policy-controller
pod logs with the following information:
Logging the diff: --- default/my-configmap : existing +++ default/my-configmap : updated @@ -2,3 +2,3 @@ data: - fieldToUpdate: "1" + fieldToUpdate: "2" kind: ConfigMap
Important: Avoid logging the difference for a secure object. The difference is logged in plain text.
3.5. Governance metric
The policy framework exposes metrics that show policy distribution and compliance. Use the policy_governance_info
metric on the hub cluster to view trends and analyze any policy failures. See the following topics for an overview of metrics:
3.5.1. Metric: policy_governance_info
The OpenShift Container Platform monitoring component collects the policy_governance_info
metric. If you enable observability, the component collects some aggregate data.
Note: If you enable observability, enter a query for the metric from the Grafana Explore page. When you create a policy, you are creating a root policy. The framework watches for root policies, Placement
resources, and PlacementBindings
resources to for information about where to create propagated policies, to distribute the policy to managed clusters.
For both root and propagated policies, a metric of 0
is recorded if the policy is compliant, 1
if it is non-compliant, and -1
if it is in an unknown or pending state.
The policy_governance_info
metric uses the following labels:
-
type
: The label values areroot
orpropagated
. -
policy
: The name of the associated root policy. -
policy_namespace
: The namespace on the hub cluster where the root policy is defined. -
cluster_namespace
: The namespace for the cluster where the policy is distributed.
These labels and values enable queries that can show us many things happening in the cluster that might be difficult to track.
Note: If you do not need the metrics, and you have any concerns about performance or security, you can disable the metric collection. Set the DISABLE_REPORT_METRICS
environment variable to true
in the propagator deployment. You can also add policy_governance_info
metric to the observability allowlist as a custom metric. See Adding custom metrics for more details.
3.5.2. Metric: config_policies_evaluation_duration_seconds
The config_policies_evaluation_duration_seconds
histogram tracks the number of seconds it takes to process all configuration policies that are ready to be evaluated on the cluster. Use the following metrics to query the histogram:
-
config_policies_evaluation_duration_seconds_bucket
: The buckets are cumulative and represent seconds with the following possible entries: 1, 3, 9, 10.5, 15, 30, 60, 90, 120, 180, 300, 450, 600, and greater. -
config_policies_evaluation_duration_seconds_count
: The count of all events. -
config_policies_evaluation_duration_seconds_sum
: The sum of all values.
Use the config_policies_evaluation_duration_seconds
metric to determine if the ConfigurationPolicy
evaluationInterval
setting needs to be changed for resource intensive policies that do not need frequent evaluation. You can also increase the concurrency at the cost of higher resource utilization on the Kubernetes API server. See Configure the concurrency section for more details.
To receive information about the time used to evaluate configuration policies, perform a Prometheus query that resembles the following expression:
rate(config_policies_evaluation_duration_seconds_sum[10m])/rate (config_policies_evaluation_duration_seconds_count[10m]
The config-policy-controller
pod running on managed clusters in the open-cluster-management-agent-addon
namespace calculates the metric. The config-policy-controller
does not send the metric to observability by default.
3.6. Verify configuration changes
When you apply the new configuration with the controller, the ManifestApplied
parameter is updated in the ManagedClusterAddOn
. That condition timestamp helps verify the configuration correctly. For example, this command can verify when the cert-policy-controller
on the local-cluster
was updated:
oc get -n local-cluster managedclusteraddon cert-policy-controller | grep -B4 'type: ManifestApplied'
You might receive the following output:
- lastTransitionTime: "2023-01-26T15:42:22Z" message: manifests of addon are applied successfully reason: AddonManifestApplied status: "True" type: ManifestApplied
3.7. Additional resources
- See Kubernetes configuration policy controller
- Return to the Governance topic for more topics.
- Return to the beginning of this topic, Policy controller advanced configuration.