Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.
Chapter 9. Working with certificates
Certificates are used by various components in OpenShift to validate access to the cluster. For clusters that rely on self-signed certificates, you can add those self-signed certificates to a cluster-wide Certificate Authority (CA) bundle and use the CA bundle in Red Hat OpenShift AI. You can also use self-signed certificates in a custom CA bundle that is separate from the cluster-wide bundle. Administrators can add a CA bundle, remove a CA bundle from all namespaces, remove a CA bundle from individual namespaces, or manually manage certificate changes instead of the system.
9.1. Understanding certificates in OpenShift AI Link kopierenLink in die Zwischenablage kopiert!
For OpenShift clusters that rely on self-signed certificates, you can add those self-signed certificates to a cluster-wide Certificate Authority (CA) bundle (ca-bundle.crt) and use the CA bundle in Red Hat OpenShift AI. You can also use self-signed certificates in a custom CA bundle (odh-ca-bundle.crt) that is separate from the cluster-wide bundle.
9.1.1. How CA bundles are injected Link kopierenLink in die Zwischenablage kopiert!
After installing OpenShift AI, the Red Hat OpenShift AI Operator automatically creates an empty odh-trusted-ca-bundle configuration file (ConfigMap), and the Cluster Network Operator (CNO) injects the cluster-wide CA bundle into the odh-trusted-ca-bundle configMap with the label "config.openshift.io/inject-trusted-cabundle". The components deployed in the affected namespaces are responsible for mounting this configMap as a volume in the deployment pods.
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/part-of: opendatahub-operator
config.openshift.io/inject-trusted-cabundle: 'true'
name: odh-trusted-ca-bundle
After the CNO operator injects the bundle, it updates the ConfigMap with the ca-bundle.crt file containing the certificates.
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/part-of: opendatahub-operator
config.openshift.io/inject-trusted-cabundle: 'true'
name: odh-trusted-ca-bundle
data:
ca-bundle.crt: |
<BUNDLE OF CLUSTER-WIDE CERTIFICATES>
9.1.2. How the ConfigMap is managed Link kopierenLink in die Zwischenablage kopiert!
By default, the Red Hat OpenShift AI Operator manages the odh-trusted-ca-bundle ConfigMap. If you want to manage or remove the odh-trusted-ca-bundle ConfigMap, or add a custom CA bundle (odh-ca-bundle.crt) separate from the cluster-wide CA bundle (ca-bundle.crt), you can use the trustedCABundle property in the Operator’s DSC Initialization (DSCI) object.
spec:
trustedCABundle:
managementState: Managed
customCABundle: ""
In the Operator’s DSCI object, you can set the spec.trustedCABundle.managementState field to the following values:
-
Managed: The Red Hat OpenShift AI Operator manages theodh-trusted-ca-bundleConfigMap and adds it to all non-reserved existing and new namespaces (the ConfigMap is not added to any reserved or system namespaces, such asdefault,openshift-\*orkube-*). The ConfigMap is automatically updated to reflect any changes made to thecustomCABundlefield. This is the default value after installing Red Hat OpenShift AI. -
Removed: The Red Hat OpenShift AI Operator removes theodh-trusted-ca-bundleConfigMap (if present) and disables the creation of the ConfigMap in new namespaces. If you change this field fromManagedtoRemoved, theodh-trusted-ca-bundleConfigMap is also deleted from namespaces. This is the default value after upgrading Red Hat OpenShift AI from 2.7 or earlier versions to 2.16. -
Unmanaged: The Red Hat OpenShift AI Operator does not manage theodh-trusted-ca-bundleConfigMap, allowing for an administrator to manage it instead. Changing themanagementStatefromManagedtoUnmanageddoes not remove theodh-trusted-ca-bundleConfigMap, but the ConfigMap is not updated if you make changes to thecustomCABundlefield.
In the Operator’s DSCI object, you can add a custom certificate to the spec.trustedCABundle.customCABundle field. This adds the odh-ca-bundle.crt file containing the certificates to the odh-trusted-ca-bundle ConfigMap, as shown in the following example:
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/part-of: opendatahub-operator
config.openshift.io/inject-trusted-cabundle: 'true'
name: odh-trusted-ca-bundle
data:
ca-bundle.crt: |
<BUNDLE OF CLUSTER-WIDE CERTIFICATES>
odh-ca-bundle.crt: |
<BUNDLE OF CUSTOM CERTIFICATES>
9.2. Adding a CA bundle Link kopierenLink in die Zwischenablage kopiert!
There are two ways to add a Certificate Authority (CA) bundle to OpenShift AI. You can use one or both of these methods:
-
For OpenShift clusters that rely on self-signed certificates, you can add those self-signed certificates to a cluster-wide Certificate Authority (CA) bundle (
ca-bundle.crt) and use the CA bundle in Red Hat OpenShift AI. To use this method, log in to the OpenShift as a cluster administrator and follow the steps as described in Configuring the cluster-wide proxy during installation. -
You can use self-signed certificates in a custom CA bundle (
odh-ca-bundle.crt) that is separate from the cluster-wide bundle. To use this method, follow the steps in this section.
Prerequisites
-
You have admin access to the
DSCInitializationresources in the OpenShift cluster. -
You installed the OpenShift command line interface (
oc) as described in Installing the OpenShift CLI. - You are working in a new installation of Red Hat OpenShift AI. If you upgraded Red Hat OpenShift AI, see Adding a CA bundle after upgrading.
Procedure
- Log in to the OpenShift.
-
Click Operators
Installed Operators and then click the Red Hat OpenShift AI Operator. - Click the DSC Initialization tab.
- Click the default-dsci object.
- Click the YAML tab.
In the
specsection, add the custom certificate to thecustomCABundlefield fortrustedCABundle, as shown in the following example:spec: trustedCABundle: managementState: Managed customCABundle: | -----BEGIN CERTIFICATE----- examplebundle123 -----END CERTIFICATE------ Click Save.
Verification
If you are using a cluster-wide CA bundle, run the following command to verify that all non-reserved namespaces contain the
odh-trusted-ca-bundleConfigMap:$ oc get configmaps --all-namespaces -l app.kubernetes.io/part-of=opendatahub-operator | grep odh-trusted-ca-bundleIf you are using a custom CA bundle, run the following command to verify that a non-reserved namespace contains the
odh-trusted-ca-bundleConfigMap and that the ConfigMap contains yourcustomCABundlevalue. In the following command, example-namespace is the non-reserved namespace and examplebundle123 is the customCABundle value.$ oc get configmap odh-trusted-ca-bundle -n example-namespace -o yaml | grep examplebundle123
9.3. Removing a CA bundle Link kopierenLink in die Zwischenablage kopiert!
You can remove a Certificate Authority (CA) bundle from all non-reserved namespaces in OpenShift AI. This process changes the default configuration and disables the creation of the odh-trusted-ca-bundle configuration file (ConfigMap), as described in Understanding certificates in OpenShift AI.
The odh-trusted-ca-bundle ConfigMaps are only deleted from namespaces when you set the managementState of trustedCABundle to Removed; deleting the DSC Initialization does not delete the ConfigMaps.
To remove a CA bundle from a single namespace only, see Removing a CA bundle from a namespace.
Prerequisites
- You have cluster administrator privileges for your OpenShift cluster.
-
You installed the OpenShift command line interface (
oc) as described in Installing the OpenShift CLI.
Procedure
-
In the OpenShift web console, click Operators
Installed Operators and then click the Red Hat OpenShift AI Operator. - Click the DSC Initialization tab.
- Click the default-dsci object.
- Click the YAML tab.
In the
specsection, change the value of themanagementStatefield fortrustedCABundletoRemoved:spec: trustedCABundle: managementState: Removed- Click Save.
Verification
Run the following command to verify that the
odh-trusted-ca-bundleConfigMap has been removed from all namespaces:$ oc get configmaps --all-namespaces | grep odh-trusted-ca-bundleThe command should not return any ConfigMaps.
9.4. Removing a CA bundle from a namespace Link kopierenLink in die Zwischenablage kopiert!
You can remove a custom Certificate Authority (CA) bundle from individual namespaces in OpenShift AI. This process disables the creation of the odh-trusted-ca-bundle configuration file (ConfigMap) for the specified namespace only.
To remove a certificate bundle from all namespaces, see Removing a CA bundle.
Prerequisites
- You have cluster administrator privileges for your OpenShift cluster.
-
You installed the OpenShift command line interface (
oc) as described in Installing the OpenShift CLI.
Procedure
Run the following command to remove a CA bundle from a namespace. In the following command, example-namespace is the non-reserved namespace.
$ oc annotate ns example-namespace security.opendatahub.io/inject-trusted-ca-bundle=false
Verification
Run the following command to verify that the CA bundle has been removed from the namespace. In the following command, example-namespace is the non-reserved namespace.
$ oc get configmap odh-trusted-ca-bundle -n example-namespaceThe command should return
configmaps "odh-trusted-ca-bundle" not found.
9.5. Managing certificates Link kopierenLink in die Zwischenablage kopiert!
After installing OpenShift AI, the Red Hat OpenShift AI Operator creates the odh-trusted-ca-bundle configuration file (ConfigMap) that contains the trusted CA bundle and adds it to all new and existing non-reserved namespaces in the cluster. By default, the Red Hat OpenShift AI Operator manages the odh-trusted-ca-bundle ConfigMap and automatically updates it if any changes are made to the CA bundle. You can choose to manage the odh-trusted-ca-bundle ConfigMap instead of allowing the Red Hat OpenShift AI Operator to manage it.
Prerequisites
- You have cluster administrator privileges for your OpenShift cluster.
Procedure
-
In the OpenShift web console, click Operators
Installed Operators and then click the Red Hat OpenShift AI Operator. - Click the DSC Initialization tab.
- Click the default-dsci object.
- Click the YAML tab.
In the
specsection, change the value of themanagementStatefield fortrustedCABundletoUnmanaged, as shown:spec: trustedCABundle: managementState: UnmanagedClick Save.
Note that changing the
managementStatefromManagedtoUnmanageddoes not remove theodh-trusted-ca-bundleConfigMap, but the ConfigMap is not updated if you make changes to thecustomCABundlefield.
Verification
In the
specsection, set or change the value of thecustomCABundlefield fortrustedCABundle, for example:spec: trustedCABundle: managementState: Unmanaged customCABundle: example123- Click Save.
-
Click Workloads
ConfigMaps. - Select a project from the project list.
-
Click the
odh-trusted-ca-bundleConfigMap. -
Click the YAML tab and verify that the value of the
customCABundlefield did not update.
9.6. Accessing S3-compatible object storage with self-signed certificates Link kopierenLink in die Zwischenablage kopiert!
To use object storage solutions or databases that are deployed in an OpenShift cluster that uses self-signed certificates, you must configure OpenShift AI to trust the cluster’s certificate authority (CA).
Each namespace has a ConfigMap called kube-root-ca.crt that contains the CA certificates of the internal API Server. Use the following steps to configure OpenShift AI to trust the certificates issued by kube-root-ca.crt.
Alternatively, you can add a custom CA bundle by using the OpenShift console, as described in Adding a CA bundle.
Prerequisites
- You have cluster administrator privileges for your OpenShift cluster.
- You have downloaded and installed the OpenShift command-line interface (CLI). See Installing the OpenShift CLI.
- You have an object storage solution or database deployed in your OpenShift cluster.
Procedure
In a terminal window, log in to the OpenShift CLI as shown in the following example:
oc login api.<cluster_name>.<cluster_domain>:6443 --webRun the following command to fetch the current OpenShift AI trusted CA configuration and store it in a new file:
oc get dscinitializations.dscinitialization.opendatahub.io default-dsci -o json | jq -r '.spec.trustedCABundle.customCABundle' > /tmp/my-custom-ca-bundles.crtAdd the cluster’s
kube-root-ca.crtConfigMap to the OpenShift AI trusted CA configuration:oc get configmap kube-root-ca.crt -o jsonpath="{['data']['ca\.crt']}" >> /tmp/my-custom-ca-bundles.crtUpdate the OpenShift AI trusted CA configuration to trust certificates issued by the certificate authorities in
kube-root-ca.crt:oc patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"$(awk '{printf "%s\\n", $0}' /tmp/my-custom-ca-bundles.crt)"'"}]'
Verification
- You can successfully deploy components that are configured to use object storage solutions or databases that are deployed in the OpenShift cluster. For example, a pipeline server that is configured to use a database deployed in the cluster starts successfully.
You can verify your new certificate configuration by following the steps in the OpenShift AI fraud detection tutorial. Run the script to install local object storage buckets and create connections, and then enable data science pipelines.
For more information about running the script to install local object storage buckets, see Running a script to install local object storage buckets and create connections.
For more information about enabling data science pipelines, see Enabling data science pipelines.
9.7. Using self-signed certificates with OpenShift AI components Link kopierenLink in die Zwischenablage kopiert!
Some OpenShift AI components have additional options or required configuration for self-signed certificates.
9.7.1. Using certificates with data science pipelines Link kopierenLink in die Zwischenablage kopiert!
If you want to use self-signed certificates, you have added them to a central Certificate Authority (CA) bundle as described in Working with certificates (for disconnected environments, see Working with certificates).
No additional configuration is necessary to use those certificates with data science pipelines.
9.7.1.1. Providing a CA bundle only for data science pipelines Link kopierenLink in die Zwischenablage kopiert!
Perform the following steps to provide a Certificate Authority (CA) bundle just for data science pipelines.
Procedure
- Log in to OpenShift.
From Workloads
ConfigMaps, create a ConfigMap with the required bundle in the same data science project or namespace as the target data science pipeline: kind: ConfigMap apiVersion: v1 metadata: name: custom-ca-bundle data: ca-bundle.crt: | # contents of ca-bundle.crtAdd the following snippet to the
.spec.apiserver.caBundlefield of the underlying Data Science Pipelines Application (DSPA):apiVersion: datasciencepipelinesapplications.opendatahub.io/v1 kind: DataSciencePipelinesApplication metadata: name: data-science-dspa spec: ... apiServer: ... cABundle: configMapName: custom-ca-bundle configMapKey: ca-bundle.crt
The pipeline server pod redeploys with the updated bundle and uses it in the newly created pipeline pods.
Verification
Perform the following steps to confirm that your CA bundle was successfully mounted.
- Log in to the OpenShift console.
- Go to the OpenShift project that corresponds to the data science project.
- Click the Pods tab.
-
Click the pipeline server pod with the
ds-pipeline-dspa-<hash>prefix. - Click Terminal.
-
Enter
cat /dsp-custom-certs/dsp-ca.crt. - Verify that your CA bundle is present within this file.
You can also confirm that your CA bundle was successfully mounted by using the CLI:
In a terminal window, log in to the OpenShift cluster where OpenShift AI is deployed.
oc loginSet the
dspavalue:dspa=dspaSet the
dsProjectvalue, replacing$YOUR_DS_PROJECTwith the name of your data science project:dsProject=$YOUR_DS_PROJECTSet the
podvalue:pod=$(oc get pod -n ${dsProject} -l app=ds-pipeline-${dspa} --no-headers | awk '{print $1}')Display the contents of the
/dsp-custom-certs/dsp-ca.crtfile:oc -n ${dsProject} exec $pod -- cat /dsp-custom-certs/dsp-ca.crt- Verify that your CA bundle is present within this file.
9.7.2. Using certificates with workbenches Link kopierenLink in die Zwischenablage kopiert!
Self-signed certificates apply by default to workbenches that you create after configuring the certificates centrally as described in Working with certificates (for disconnected environments, see Working with certificates). To apply centrally configured certificates to an existing workbench, stop and then restart the workbench.
Self-signed certificates are stored in /etc/pki/tls/custom-certs/ca-bundle.crt. Workbenches are preset with an environment variable that points packages to this path, and that covers many popular HTTP client packages. For packages that are not included by default, you can provide this certificate path. For example, for the kfp package to connect to the data science pipeline server:
from kfp.client import Client
with open(sa_token_file_path, 'r') as token_file:
bearer_token = token_file.read()
client = Client(
host='https://<GO_TO_ROUTER_OF_DS_PROJECT>/',
existing_token=bearer_token,
ssl_ca_cert='/etc/pki/tls/custom-certs/ca-bundle.crt'
)
print(client.list_experiments())