Chapter 1. Risk and compliance
Manage your security of Red Hat Advanced Cluster Management for Kubernetes components. Govern your cluster with defined policies and processes to identify and minimize risks. Use policies to define rules and set controls.
Prerequisite: You must configure authentication service requirements for Red Hat Advanced Cluster Management for Kubernetes. See Access control for more information.
Review the following topics to learn more about securing your cluster:
1.1. Certificates
Various certificates are created and used throughout Red Hat Advanced Cluster Management for Kubernetes.
You can bring your own certificates. You must create a Kubernetes TLS Secret for your certificate. After you create your certificates, you can replace certain certificates that are created by the Red Hat Advanced Cluster Management installer.
Required access: Cluster administrator or team administrator.
Note: Replacing certificates is supported only on native Red Hat Advanced Cluster Management installations.
All certificates required by services that run on Red Hat Advanced Cluster Management are created during the installation of Red Hat Advanced Cluster Management. Certificates are created and managed by the OpenShift Service Serving Certificates service.
You can also rotate the OpenShift Service Serving certificates. For more information, follow the OpenShift documentation to Manually rotate the generated service certificate and Manually rotate the service CA certificate. After the rotation is complete, apply the new certificates to all of the services with the following command:
oc -n open-cluster-management delete pod -l chart=management-ingress
The related pods in your cluster restart automatically.
Continue reading to learn more about certificate management:
Red Hat Advanced Cluster Management hub cluster certificates
Red Hat Advanced Cluster Management component certificates
Red Hat Advanced Cluster Management managed certificates
Note: Users are responsible for certificate rotations and updates.
1.1.1. Red Hat Advanced Cluster Management hub cluster certificates
1.1.1.1. Observability certificates
After Red Hat Advanced Cluster Management is installed, observability certificates are created and used by the observability components, to provide mutual TLS on the traffic between the hub cluster and managed cluster. The Kubernetes secrets that are associated with the observability certificates.
The open-cluster-management-observability
namespace contain the following certificates:
-
observability-server-ca-certs
: Has the CA certificate to sign server-side certificates -
observability-client-ca-certs
: Has the CA certificate to sign client-side certificates -
observability-server-certs
: Has the server certificate used by theobservability-observatorium-api
deployment -
observability-grafana-certs
: Has the client certificate used by theobservability-rbac-query-proxy
deployment
The open-cluster-management-addon-observability
namespace contain the following certificates on managed clusters:
-
observability-managed-cluster-certs
: Has the same server CA certificate asobservability-server-ca-certs
in the hub server -
observability-controller-open-cluster-management.io-observability-signer-client-cert
: Has the client certificate used by themetrics-collector-deployment
The CA certificates are valid for five years and other certificates are valid for one year. All observability certificates are automatically refreshed upon expiration.
View the following list to understand the effects when certificates are automatically renewed:
- Non-CA certificates are renewed automatically when the remaining valid time is no more than 73 days. After the certificate is renewed, the pods in the related deployments restart automatically to use the renewed certificates.
- CA certificates are renewed automatically when the remaining valid time is no more than one year. After the certificate is renewed, the old CA is not deleted but co-exist with the renewed ones. Both old and renewed certificates are used by related deployments, and continue to work. The old CA certificates are deleted when they expire.
- When a certificate is renewed, the traffic between the hub cluster and managed cluster is not interrupted.
1.1.1.2. Bring Your Own (BYO) observability certificate authority (CA) certificates
If you do not want to use the default observability CA certificates generated by Red Hat Advanced Cluster Management, you can choose to use the BYO observability CA certificates before you enable observability.
1.1.1.2.1. OpenSSL commands to generate CA certificate
Observability requires two CA certificates; one is for the server-side and the other is for the client-side.
Generate your CA RSA private keys with the following commands:
openssl genrsa -out serverCAKey.pem 2048 openssl genrsa -out clientCAKey.pem 2048
Generate the self-signed CA certificates using the private keys. Run the following commands:
openssl req -x509 -sha256 -new -nodes -key serverCAKey.pem -days 1825 -out serverCACert.pem openssl req -x509 -sha256 -new -nodes -key clientCAKey.pem -days 1825 -out clientCACert.pem
1.1.1.2.2. Create the secrets associated with the BYO observability CA certificates
Complete the following steps to create the secrets:
Create the
observability-server-ca-certs
secret by using your certificate and private key. Run the following command:oc -n open-cluster-management-observability create secret tls observability-server-ca-certs --cert ./serverCACert.pem --key ./serverCAKey.pem
Create the
observability-client-ca-certs
secret by using your certificate and private key. Run the following command:oc -n open-cluster-management-observability create secret tls observability-client-ca-certs --cert ./clientCACert.pem --key ./clientCAKey.pem
1.1.1.2.3. Replacing certificates for alertmanager route
You can replace alertmanager certificates by updating the alertmanager route, if you do not want to use the OpenShift default ingress certificate. Complete the following steps:
Examine the observability certificate with the following command:
openssl x509 -noout -text -in ./observability.crt
-
Change the common name (
CN
) on the certificate toalertmanager
. -
Change the SAN in the
csr.cnf
configuration file with the hostname for your alertmanager route. Create the two following secrets in the
open-cluster-management-observability
namespace. Run the following command:oc -n open-cluster-management-observability create secret tls alertmanager-byo-ca --cert ./ca.crt --key ./ca.key oc -n open-cluster-management-observability create secret tls alertmanager-byo-cert --cert ./ingress.crt --key ./ingress.key
For more information, see OpenSSL commands for generating a certificate. If you want to restore the default self-signed certificate for alertmanager route, see Restore the default self-signed certificate for management ingress to delete the two secrets in the open-cluster-management-observability
namespace.
1.1.2. Red Hat Advanced Cluster Management component certificates
1.1.2.1. List hub cluster managed certificates
You can view a list of hub cluster managed certificates that use OpenShift Service Serving Certificates service internally. Run the following command to list the certificates:
oc get secret -n open-cluster-management -o custom-columns=Name:.metadata.name,Expiration:.metadata.annotations.service\\.beta\\.openshift\\.io/expiry | grep -v '<none>'
Note: If observability is enabled, there are additional namespaces where certificates are created.
1.1.2.2. Refresh hub cluster managed certificates
You can refresh a hub cluster managed certificate by running the command in the List hub cluster managed certificates section. When you identify the certificate that you need to refresh, delete the secret that is associated with the certificate. For example, you can delete a secret by running the following command:
oc delete secret grc-0c925-grc-secrets -n open-cluster-management
Note: After you delete the secret, a new one is created. However, you must restart pods that use the secret manually so they can begin to use the new certificate.
1.1.2.3. Refresh a OpenShift Container Platform managed certificate
You can refresh OpenShift Container Platform managed certificates, which are certificates that are used by Red Hat Advanced Cluster Management webhooks and the proxy server.
Complete the following steps to refresh OpenShift Container Platform managed certificates:
Delete the secret that is associated with the OpenShift Container Platform managed certificate by running the following command:
oc delete secret -n open-cluster-management ocm-webhook-secret
Note: Some services might not have a secret that needs to be deleted.
Restart the services that are associated with the OpenShift Container Platform managed certificate(s) by running the following command:
oc delete po -n open-cluster-management ocm-webhook-679444669c-5cg76
Important: There are replicas of many services; each service must be restarted.
View the following table for a summarized list of the pods that contain certificates and whether a secret needs to be deleted prior to restarting the pod:
Service name | Namespace | Sample pod name | Secret name (if applicable) |
---|---|---|---|
channels-apps-open-cluster-management-webhook-svc | open-cluster-management | multicluster-operators-application-8c446664c-5lbfk | - |
multicluster-operators-application-svc | open-cluster-management | multicluster-operators-application-8c446664c-5lbfk | - |
multiclusterhub-operator-webhook | open-cluster-management | multiclusterhub-operator-bfd948595-mnhjc | - |
ocm-webhook | open-cluster-management | ocm-webhook-679444669c-5cg76 | ocm-webhook-secret |
cluster-manager-registration-webhook | open-cluster-management-hub | cluster-manager-registration-webhook-fb7b99c-d8wfc | registration-webhook-serving-cert |
cluster-manager-work-webhook | open-cluster-management-hub | cluster-manager-work-webhook-89b8d7fc-f4pv8 | work-webhook-serving-cert |
1.1.3. Red Hat Advanced Cluster Management managed certificates
1.1.3.1. Channel certificates
CA certificates can be associated with Git channel that are a part of the Red Hat Advanced Cluster Management application management. See Using custom CA certificates for a secure HTTPS connection for more details.
Helm channels allow you to disable certificate validation. Helm channels where certificate validation is disabled, must be configured in development environments. Disabling certificate validation introduces security risks.
1.1.3.2. Managed cluster certificates
Certificates are used to authenticate managed clusters with the hub. Therefore, it is important to be aware of troubleshooting scenarios associated with these certificates. View Troubleshooting imported clusters offline after certificate change for more details.
The managed cluster certificates are refreshed automatically.
1.1.4. Third-party certificates
1.1.4.1. Rotating the gatekeeper webhook certificate
Complete the following steps to rotate the gatekeeper webhook certificate:
Edit the secret that contains the certificate with the following command:
oc edit secret -n openshift-gatekeeper-system gatekeeper-webhook-server-cert
-
Delete the following content in the
data
section:ca.crt
,ca.key
, tls.crt`, andtls.key
. Restart the gatekeeper webhook service by deleting the
gatekeeper-controller-manager
pods with the following command:oc delete po -n openshift-gatekeeper-system -l control-plane=controller-manager
The gatekeeper webhook certificate is rotated.
1.1.4.2. Rotating the integrity shield webhook certificate (Technology preview)
Complete the following steps to rotate the integrity shield webhook certificate:
Edit the IntegrityShield custom resource and add the
integrity-shield-operator-system
namespace to the excluded list of namespaces in theinScopeNamespaceSelector
setting. Run the following command to edit the resource:oc edit integrityshield integrity-shield-server -n integrity-shield-operator-system
Delete the secret that contains the integrity shield certificate by running the following command:
oc delete secret -n integrity-shield-operator-system ishield-server-tls
Delete the operator so that the secret is recreated. Be sure that the operator pod name matches the pod name on your system. Run the following command:
oc delete po -n integrity-shield-operator-system integrity-shield-operator-controller-manager-64549569f8-v4pz6
Delete the integrity shield server pod to begin using the new certificate with the following command:
oc delete po -n integrity-shield-operator-system integrity-shield-server-5fbdfbbbd4-bbfbz
Use the certificate policy controller to create and manage certificate policies on managed clusters. See Policy controllers to learn more about controllers. Return to the Risk and compliance page for more information.
1.2. Replacing the management ingress certificates
You can replace management ingress certificates by updating the Red Hat Advanced Cluster Management for Kubernetes route if you do not want to use the OpenShift default ingress certificate.
1.2.1. Prerequisites to replace management ingress certificate
Prepare and have your management-ingress
certificates and private keys ready. If needed, you can generate a TLS certificate by using OpenSSL. Set the common name parameter (CN
) on the certificate to manangement-ingress
. If you are generating the certificate, include the following settings:
Include the route name for Red Hat Advanced Cluster Management for Kubernetes as the domain name in your certificate Subject Alternative Name (SAN) list.
Receive the route name by running the following command:
oc get route -n open-cluster-management
You might receive the following response:
multicloud-console.apps.grchub2.dev08.red-chesterfield.com
1.2.1.1. Example configuration file for generating a certificate
The following example configuration file and OpenSSL commands provide an example for how to generate a TLS certificate by using OpenSSL. View the following csr.cnf
configuration file, which defines the configuration settings for generating certificates with OpenSSL.
[ req ] # Main settings default_bits = 2048 # Default key size in bits. prompt = no # Disables prompting for certificate values so the configuration file values are used. default_md = sha256 # Specifies the digest algorithm. req_extensions = req_ext # Specifies the configuration file section that includes any extensions. distinguished_name = dn # Specifies the section that includes the distinguished name information. [ dn ] # Distinguished name settings C = US # Country ST = North Carolina # State or province L = Raleigh # Locality O = Red Hat Open Shift # Organization OU = Red Hat Advanced Container Management # Organizational unit CN = management-ingress # Common name. [ req_ext ] # Extensions subjectAltName = @alt_names # Subject alternative names [ alt_names ] # Subject alternative names DNS.1 = multicloud-console.apps.grchub2.dev08.red-chesterfield.com [ v3_ext ] # x509v3 extensions authorityKeyIdentifier=keyid,issuer:always # Specifies the public key that corresponds to the private key that is used to sign a certificate. basicConstraints=CA:FALSE # Indicates whether the certificate is a CA certificate during the certificate chain verification process. #keyUsage=keyEncipherment,dataEncipherment # Defines the purpose of the key that is contained in the certificate. extendedKeyUsage=serverAuth # Defines the purposes for which the public key can be used. subjectAltName=@alt_names # Identifies the subject alternative names for the identify that is bound to the public key by the CA.
Note: Be sure to update the SAN labeled, DNS.1
with the correct hostname for your management ingress.
1.2.1.2. OpenSSL commands for generating a certificate
The following OpenSSL commands are used with the preceding configuration file to generate the required TLS certificate.
Generate your certificate authority (CA) RSA private key:
openssl genrsa -out ca.key 4096
Generate a self-signed CA certificate by using your CA key:
openssl req -x509 -new -nodes -key ca.key -subj "/C=US/ST=North Carolina/L=Raleigh/O=Red Hat OpenShift" -days 400 -out ca.crt
Generate the RSA private key for your certificate:
openssl genrsa -out ingress.key 4096
Generate the Certificate Signing request (CSR) by using the private key:
openssl req -new -key ingress.key -out ingress.csr -config csr.cnf
Generate a signed certificate by using your CA certificate and key and CSR:
openssl x509 -req -in ingress.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ingress.crt -sha256 -days 300 -extensions v3_ext -extfile csr.cnf
Examine the certificate contents:
openssl x509 -noout -text -in ./ingress.crt
1.2.2. Replace the Bring Your Own (BYO) ingress certificate
Complete the following steps to replace your BYO ingress certificate:
Create the
byo-ingress-tls
secret by using your certificate and private key. Run the following command:oc -n open-cluster-management create secret tls byo-ingress-tls-secret --cert ./ingress.crt --key ./ingress.key
Verify that the secret is created in the correct namespace with the following command:
oc get secret -n open-cluster-management | grep -e byo-ingress-tls-secret -e byo-ca-cert
Optional: Create a secret containing the CA certificate by running the following command:
oc -n open-cluster-management create secret tls byo-ca-cert --cert ./ca.crt --key ./ca.key
Delete the
management-ingress
subscription in order to redeploy the subscription. The secrets created in the previous steps are used automatically. Run the following command:oc delete subscription management-ingress-sub -n open-cluster-management
- Verify that the current certificate is your certificate, and that all console access and login functionality remain the same.
1.2.3. Restore the default self-signed certificate for management ingress
Delete the bring your own certificate secrets with the following command:
oc delete secret byo-ca-cert byo-ingress-tls-secret -n open-cluster-management
Delete the
management-ingress
subscription in order to redeploy the subscription. The secrets created in the previous steps are used automatically. Run the following command:oc delete subscription management-ingress-sub -n open-cluster-management
- Verify that the current certificate is your certificate, and that all console access and login functionality remain the same.
See Certificates for more information about certificates that are created and managed by Red Hat Advanced Cluster Management. Return to the Risk and compliance page for more information on securing your cluster.