Chapter 7. Working with certificates
Certificates are used by various components in OpenShift to validate access to the cluster. For clusters that rely on self-signed certificates, you can add those self-signed certificates to a cluster-wide Certificate Authority (CA) bundle and use the CA bundle in Red Hat OpenShift AI. You can also use self-signed certificates in a custom CA bundle that is separate from the cluster-wide bundle. Administrators can add a CA bundle, remove a CA bundle from all namespaces, remove a CA bundle from individual namespaces, or manually manage certificate changes instead of the system.
7.1. Understanding certificates in OpenShift AI
For OpenShift clusters that rely on self-signed certificates, you can add those self-signed certificates to a cluster-wide Certificate Authority (CA) bundle (ca-bundle.crt
) and use the CA bundle in Red Hat OpenShift AI. You can also use self-signed certificates in a custom CA bundle (odh-ca-bundle.crt
) that is separate from the cluster-wide bundle.
7.1.1. How CA bundles are injected
After installing OpenShift AI, the Red Hat OpenShift AI Operator automatically creates an empty odh-trusted-ca-bundle
configuration file (ConfigMap), and the Cluster Network Operator (CNO) injects the cluster-wide CA bundle into the odh-trusted-ca-bundle
configMap with the label "config.openshift.io/inject-trusted-cabundle". The components deployed in the affected namespaces are responsible for mounting this configMap as a volume in the deployment pods.
apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/part-of: opendatahub-operator config.openshift.io/inject-trusted-cabundle: 'true' name: odh-trusted-ca-bundle
After the CNO operator injects the bundle, it updates the ConfigMap with the ca-bundle.crt
file containing the certificates.
apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/part-of: opendatahub-operator config.openshift.io/inject-trusted-cabundle: 'true' name: odh-trusted-ca-bundle data: ca-bundle.crt: | <BUNDLE OF CLUSTER-WIDE CERTIFICATES>
7.1.2. How the ConfigMap is managed
By default, the Red Hat OpenShift AI Operator manages the odh-trusted-ca-bundle
ConfigMap. If you want to manage or remove the odh-trusted-ca-bundle
ConfigMap, or add a custom CA bundle (odh-ca-bundle.crt
) separate from the cluster-wide CA bundle (ca-bundle.crt
), you can use the trustedCABundle
property in the Operator’s DSC Initialization (DSCI) object.
spec: trustedCABundle: managementState: Managed customCABundle: ""
In the Operator’s DSCI object, you can set the spec.trustedCABundle.managementState
field to the following values:
-
Managed
: The Red Hat OpenShift AI Operator manages theodh-trusted-ca-bundle
ConfigMap and adds it to all non-reserved existing and new namespaces (the ConfigMap is not added to any reserved or system namespaces, such asdefault
,openshift-\*
orkube-*
). The ConfigMap is automatically updated to reflect any changes made to thecustomCABundle
field. This is the default value after installing Red Hat OpenShift AI. -
Removed
: The Red Hat OpenShift AI Operator removes theodh-trusted-ca-bundle
ConfigMap (if present) and disables the creation of the ConfigMap in new namespaces. If you change this field fromManaged
toRemoved
, theodh-trusted-ca-bundle
ConfigMap is also deleted from namespaces. This is the default value after upgrading Red Hat OpenShift AI from 2.7 or earlier versions to 2-latest. -
Unmanaged
: The Red Hat OpenShift AI Operator does not manage theodh-trusted-ca-bundle
ConfigMap, allowing for an administrator to manage it instead. Changing themanagementState
fromManaged
toUnmanaged
does not remove theodh-trusted-ca-bundle
ConfigMap, but the ConfigMap is not updated if you make changes to thecustomCABundle
field.
In the Operator’s DSCI object, you can add a custom certificate to the spec.trustedCABundle.customCABundle
field. This adds the odh-ca-bundle.crt
file containing the certificates to the odh-trusted-ca-bundle
ConfigMap, as shown in the following example:
apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/part-of: opendatahub-operator config.openshift.io/inject-trusted-cabundle: 'true' name: odh-trusted-ca-bundle data: ca-bundle.crt: | <BUNDLE OF CLUSTER-WIDE CERTIFICATES> odh-ca-bundle.crt: | <BUNDLE OF CUSTOM CERTIFICATES>
7.2. Adding a CA bundle
There are two ways to add a Certificate Authority (CA) bundle to OpenShift AI. You can use one or both of these methods:
-
For OpenShift clusters that rely on self-signed certificates, you can add those self-signed certificates to a cluster-wide Certificate Authority (CA) bundle (
ca-bundle.crt
) and use the CA bundle in Red Hat OpenShift AI. To use this method, log in to the OpenShift as a cluster administrator and follow the steps as described in Configuring the cluster-wide proxy during installation. -
You can use self-signed certificates in a custom CA bundle (
odh-ca-bundle.crt
) that is separate from the cluster-wide bundle. To use this method, follow the steps in this section.
Prerequisites
-
You have admin access to the
DSCInitialization
resources in the OpenShift cluster. -
You installed the OpenShift command line interface (
oc
) as described in Get Started with the CLI. - You are working in a new installation of Red Hat OpenShift AI. If you upgraded Red Hat OpenShift AI, see Adding a CA bundle after upgrading.
Procedure
- Log in to the OpenShift.
-
Click Operators
Installed Operators and then click the Red Hat OpenShift AI Operator. - Click the DSC Initialization tab.
- Click the default-dsci object.
- Click the YAML tab.
In the
spec
section, add the custom certificate to thecustomCABundle
field fortrustedCABundle
, as shown in the following example:spec: trustedCABundle: managementState: Managed customCABundle: | -----BEGIN CERTIFICATE----- examplebundle123 -----END CERTIFICATE-----
- Click Save.
Verification
If you are using a cluster-wide CA bundle, run the following command to verify that all non-reserved namespaces contain the
odh-trusted-ca-bundle
ConfigMap:$ oc get configmaps --all-namespaces -l app.kubernetes.io/part-of=opendatahub-operator | grep odh-trusted-ca-bundle
If you are using a custom CA bundle, run the following command to verify that a non-reserved namespace contains the
odh-trusted-ca-bundle
ConfigMap and that the ConfigMap contains yourcustomCABundle
value. In the following command, example-namespace is the non-reserved namespace and examplebundle123 is the customCABundle value.$ oc get configmap odh-trusted-ca-bundle -n example-namespace -o yaml | grep examplebundle123
7.3. Removing a CA bundle
You can remove a Certificate Authority (CA) bundle from all non-reserved namespaces in OpenShift AI. This process changes the default configuration and disables the creation of the odh-trusted-ca-bundle
configuration file (ConfigMap), as described in Understanding certificates in OpenShift AI.
The odh-trusted-ca-bundle
ConfigMaps are only deleted from namespaces when you set the managementState
of trustedCABundle
to Removed
; deleting the DSC Initialization does not delete the ConfigMaps.
To remove a CA bundle from a single namespace only, see Removing a CA bundle from a namespace.
Prerequisites
- You have cluster administrator privileges for your OpenShift cluster.
-
You installed the OpenShift command line interface (
oc
) as described in Get Started with the CLI.
Procedure
-
In the OpenShift web console, click Operators
Installed Operators and then click the Red Hat OpenShift AI Operator. - Click the DSC Initialization tab.
- Click the default-dsci object.
- Click the YAML tab.
In the
spec
section, change the value of themanagementState
field fortrustedCABundle
toRemoved
:spec: trustedCABundle: managementState: Removed
- Click Save.
Verification
Run the following command to verify that the
odh-trusted-ca-bundle
ConfigMap has been removed from all namespaces:$ oc get configmaps --all-namespaces | grep odh-trusted-ca-bundle
The command should not return any ConfigMaps.
7.4. Removing a CA bundle from a namespace
You can remove a custom Certificate Authority (CA) bundle from individual namespaces in OpenShift AI. This process disables the creation of the odh-trusted-ca-bundle
configuration file (ConfigMap) for the specified namespace only.
To remove a certificate bundle from all namespaces, see Removing a CA bundle.
Prerequisites
- You have cluster administrator privileges for your OpenShift cluster.
-
You installed the OpenShift command line interface (
oc
) as described in Get Started with the CLI.
Procedure
Run the following command to remove a CA bundle from a namespace. In the following command, example-namespace is the non-reserved namespace.
$ oc annotate ns example-namespace security.opendatahub.io/inject-trusted-ca-bundle=false
Verification
Run the following command to verify that the CA bundle has been removed from the namespace. In the following command, example-namespace is the non-reserved namespace.
$ oc get configmap odh-trusted-ca-bundle -n example-namespace
The command should return
configmaps "odh-trusted-ca-bundle" not found
.
7.5. Managing certificates
After installing OpenShift AI, the Red Hat OpenShift AI Operator creates the odh-trusted-ca-bundle
configuration file (ConfigMap) that contains the trusted CA bundle and adds it to all new and existing non-reserved namespaces in the cluster. By default, the Red Hat OpenShift AI Operator manages the odh-trusted-ca-bundle
ConfigMap and automatically updates it if any changes are made to the CA bundle. You can choose to manage the odh-trusted-ca-bundle
ConfigMap instead of allowing the Red Hat OpenShift AI Operator to manage it.
Prerequisites
- You have cluster administrator privileges for your OpenShift cluster.
Procedure
-
In the OpenShift web console, click Operators
Installed Operators and then click the Red Hat OpenShift AI Operator. - Click the DSC Initialization tab.
- Click the default-dsci object.
- Click the YAML tab.
In the
spec
section, change the value of themanagementState
field fortrustedCABundle
toUnmanaged
, as shown:spec: trustedCABundle: managementState: Unmanaged
Click Save.
Note that changing the
managementState
fromManaged
toUnmanaged
does not remove theodh-trusted-ca-bundle
ConfigMap, but the ConfigMap is not updated if you make changes to thecustomCABundle
field.
Verification
In the
spec
section, set or change the value of thecustomCABundle
field fortrustedCABundle
, for example:spec: trustedCABundle: managementState: Unmanaged customCABundle: example123
- Click Save.
-
Click Workloads
ConfigMaps. - Select a project from the project list.
-
Click the
odh-trusted-ca-bundle
ConfigMap. -
Click the YAML tab and verify that the value of the
customCABundle
field did not update.
7.6. Using self-signed certificates with OpenShift AI components
Some OpenShift AI components have additional options or required configuration for self-signed certificates.
7.6.1. Using certificates with data science pipelines
If you want to use self-signed certificates, you have added them to a central Certificate Authority (CA) bundle as described in Working with certificates (for disconnected environments, see Working with certificates).
No additional configuration is necessary to use those certificates with data science pipelines.
7.6.1.1. Providing a CA bundle only for data science pipelines
Perform the following steps to provide a Certificate Authority (CA) bundle just for data science pipelines.
Procedure
- Log in to OpenShift.
From Workloads
ConfigMaps, create a ConfigMap with the required bundle in the same data science project or namespace as the target data science pipeline: kind: ConfigMap apiVersion: v1 metadata: name: custom-ca-bundle data: ca-bundle.crt: | # contents of ca-bundle.crt
Add the following snippet to the
.spec.apiserver.caBundle
field of the underlying Data Science Pipelines Application (DSPA):apiVersion: datasciencepipelinesapplications.opendatahub.io/v1alpha1 kind: DataSciencePipelinesApplication metadata: name: data-science-dspa spec: ... apiServer: ... cABundle: configMapName: custom-ca-bundle configMapKey: ca-bundle.crt
The pipeline server pod redeploys with the updated bundle and uses it in the newly created pipeline pods.
Verification
Perform the following steps to confirm that your CA bundle was successfully mounted.
- Log in to the OpenShift console.
- Go to the OpenShift project that corresponds to the data science project.
- Click the Pods tab.
-
Click the pipeline server pod with the
ds-pipeline-dspa-<hash>
prefix. - Click Terminal.
-
Enter
cat /dsp-custom-certs/dsp-ca.crt
. - Verify that your CA bundle is present within this file.
You can also confirm that your CA bundle was successfully mounted by using the CLI:
In a terminal window, log in to the OpenShift cluster where OpenShift AI is deployed.
oc login
Set the
dspa
value:dspa=dspa
Set the
dsProject
value, replacing$YOUR_DS_PROJECT
with the name of your data science project:dsProject=$YOUR_DS_PROJECT
Set the
pod
value:pod=$(oc get pod -n ${dsProject} -l app=ds-pipeline-${dspa} --no-headers | awk '{print $1}')
Display the contents of the
/dsp-custom-certs/dsp-ca.crt
file:oc -n ${dsProject} exec $pod -- cat /dsp-custom-certs/dsp-ca.crt
- Verify that your CA bundle is present within this file.
7.6.2. Using certificates with workbenches
Self-signed certificates apply to workbenches that you create after configuring self-signed certificates centrally as described in Working with certificates (for disconnected environments, see Working with certificates. There is no change to workbenches that you created before configuring self-signed certificates.
7.6.2.1. Creating data science pipelines with Elyra and self-signed certificates
To create pipelines using a workbench that contains the Elyra extension and which uses self-signed certificates, see the Workbench workaround for executing a pipeline using Elyra in a disconnected environment knowledgebase article.