Chapter 8. Working with certificates

Procedure

1. In a terminal window, log in to the OpenShift CLI (oc) as shown in the following example:

   oc login api.<cluster_name>.<cluster_domain>:6443 --web

2. Retrieve the current OpenShift AI trusted CA configuration and store it in a new file. This step preserves any existing custom CA bundle, so that the later steps add the internal CA for the cluster to it instead of replacing it:

   oc get dscinitializations.dscinitialization.opendatahub.io default-dsci -o json | jq -r '.spec.trustedCABundle.customCABundle' > /tmp/my-custom-ca-bundles.crt

3. Add the kube-root-ca.crt config map for the cluster to the OpenShift AI trusted CA configuration:

   oc get configmap kube-root-ca.crt -o jsonpath="{['data']['ca\.crt']}" >> /tmp/my-custom-ca-bundles.crt

   Note

   The kube-root-ca.crt config map is identical in every namespace, so this command returns the same CA certificate regardless of your current project.

4. Update the OpenShift AI trusted CA configuration to trust certificates issued by the certificate authorities in kube-root-ca.crt:

   oc patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"$(awk '{printf "%s\\n", $0}' /tmp/my-custom-ca-bundles.crt)"'"}]'

Procedure

1. Log in to the OpenShift web console as a cluster administrator.

2. Create a config map with the required bundle for your target AI pipeline:

   1. In the Administrator perspective, click Workloads ConfigMaps.
   2. From the Project list, select the project that contains the target AI pipeline.
   3. Click Create ConfigMap.

   4. In the Configure via section, select the YAML view option.

      The Create ConfigMap page opens, with default YAML code automatically added.

   5. Update the default YAML code for the required bundle for your target AI pipeline:

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: custom-ca-bundle
      data:
        ca-bundle.crt: |
          # contents of ca-bundle.crt

   6. Click Create.

3. Update the .spec.apiServer.cABundle field of the underlying DataSciencePipelinesApplication (DSPA):

   1. Click Home Search.
   2. From the Resources list, search for DataSciencePipelinesApplication and then select it from the list of results.
   3. Click the YAML tab.

   4. Add or update the following snippet to the .spec.apiServer.cABundle field:

      apiVersion: datasciencepipelinesapplications.opendatahub.io/v1
      kind: DataSciencePipelinesApplication
      metadata:
        name: data-science-dspa
      spec:
        apiServer:
          cABundle:
            configMapName: custom-ca-bundle
            configMapKey: ca-bundle.crt

      where:

      configMapName

      Required. Specifies the name of the source config map that you created.

      configMapKey

      Required. Specifies the key in the source config map that holds the CA bundle.

   5. Optional: To mount the bundle at a path or file name other than the default /dsp-custom-certs/dsp-ca.crt, set the caBundleFileMountPath and caBundleFileName fields under spec.apiServer:

      spec:
        apiServer:
          caBundleFileMountPath: /tmp/custom-dsp-certs
          caBundleFileName: custom-dsp-ca.crt

      where:

      caBundleFileMountPath

      Optional. Overrides the default mount directory, /dsp-custom-certs.

      caBundleFileName

      Optional. Overrides the default file name, dsp-ca.crt.

   6. Click Save to save your changes. The pipeline server pod automatically redeploys with the updated bundle.

      Note

      The pipeline controller generates a managed config map that is named dsp-trusted-ca-__<dspa-name>__. For example, a DSPA named dspa produces a managed config map that is named dsp-trusted-ca-dspa. The running pipeline server pod mounts this managed config map, not any source config map that you create.

      The controller creates this managed config map even when you do not set cABundle. When podToPodTLS is enabled, which is the default, the controller always adds the cluster service CA to the managed config map. When you set cABundle, the controller adds the CA certificates from your source config map to the same managed config map.

Procedure

1. Apply the certificate to the workbench:

   • For a new workbench, create the workbench after you configure cluster-wide certificates. The workbench trusts the bundle automatically. For more information about how to create workbenches, see Creating a workbench.
   • For an existing workbench, stop and then restart the workbench. For more information, see Starting a workbench.
   Important

   By default, self-signed certificates apply only to workbenches that you create after you configure cluster-wide certificates. Restarting an existing workbench applies the current bundle to it.

2. Optional: For a package that does not read any of the preset environment variables, pass the certificate path explicitly. For example, the kfp package connects to the AI pipeline server by using the ssl_ca_cert parameter:

   from kfp.client import Client
   
   with open(sa_token_file_path, 'r') as token_file:
       bearer_token = token_file.read()
   
       client = Client(
           host='https://<pipeline_server_route>/',
           existing_token=bearer_token,
           ssl_ca_cert='/etc/pki/tls/custom-certs/ca-bundle.crt'
       )
       print(client.list_experiments())

   Note

   The Elyra extension reads the preset KF_PIPELINES_SSL_SA_CERTS variable and applies this bundle automatically when you build and submit pipelines in the visual editor. When you call the kfp client directly, as in this example, pass ssl_ca_cert explicitly so that the client trusts the bundle.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.

2. Go to the Installed Operators page. The navigation path depends on your OpenShift version:

   • On OpenShift 4.20 and later, click Ecosystem Installed Operators.
   • On OpenShift 4.19, click Operators Installed Operators.
3. From the list of projects, open the openshift-ingress project.
4. Click YAML.
5. Search for "cert" to find a secret with a name that includes "cert". For example, rhods-internal-primary-cert-bundle-secret. The contents of the secret should contain two items that are used for all OpenShift Routes: tls.cert (the certificate) and tls.key (the key).
6. Copy the reference to the secret.
7. From the list of projects, open the istio-system project.
8. Create a YAML file and paste the reference to the secret that you copied from the openshift-ingress YAML file.

9. Edit the YAML code to keep only the relevant content, as shown in the following example. Replace rhods-internal-primary-cert-bundle-secret with the name of your secret:

   kind: Secret
   apiVersion: v1
   metadata:
   name: rhods-internal-primary-cert-bundle-secret
   data:
   tls.crt: >-
       LS0tLS1CRUd...
   tls.key: >-
       LS0tLS1CRUd...
   type: kubernetes.io/tls

10. Save the YAML file in the istio-system project.
11. Log in to the OpenShift web console as a cluster administrator.

12. From the OpenShift web console, go to the Installed Operators page. The navigation path depends on your OpenShift version:

    • On OpenShift 4.20 and later, click Ecosystem Installed Operators.
    • On OpenShift 4.19, click Operators Installed Operators.
13. Click Data Science Cluster, and then click default-dsc YAML.

14. Edit the kserve configuration section to refer to your secret as shown in the following example. Replace rhods-internal-primary-cert-bundle-secret with the name of the secret that you created in Step 8.

    kserve:
    devFlags: {}
    managementState: Managed
    serving:
        ingressGateway:
        certificate:
            secretName: rhods-internal-primary-cert-bundle-secret
            type: Provided
        managementState: Managed
        name: knative-serving

1. Reads the CA certificates from the source config map and validates each certificate.
2. Concatenates the valid certificates into a single bundle and stores the bundle in a managed config map that is named <instance-name>-ca-bundle.
3. Mounts the managed config map into the OGX server pod at /etc/ssl/certs/ca-bundle/ca-bundle.crt.
4. Sets the SSL_CERT_FILE environment variable on the server container so that TLS clients in the server use the bundle automatically.

Procedure

1. Log in to the OpenShift AI web console as a cluster administrator.
2. From the Project list, select the project that contains your OGXServer CR.

3. Confirm that the odh-trusted-ca-bundle config map is present in your project:

   1. In the Administrator perspective, click Workloads ConfigMaps.
   2. In the list of config maps, locate odh-trusted-ca-bundle and click its name.

   3. Confirm that the config map contains the following two keys, both of which the OGX Operator reads:

      ca-bundle.crt

      The cluster-wide CA bundle that the Cluster Network Operator (CNO) injects.

      odh-ca-bundle.crt

      Custom CAs that you have added through the customCABundle field of the DSCI object.

4. Reference the odh-trusted-ca-bundle config map from your OGXServer CR.

   1. Click Home Search.
   2. From the Resources list, search for OGXServer and select it. The cluster also exposes a OGX resource, which is an internal OpenShift AI resource that is managed by the Red Hat OpenShift AI Operator. Do not select OGX.
   3. From the list of OGXServer instances, click the name of the instance that you want to update.
   4. Click the YAML tab.

   5. Add a tlsConfig.caBundle field to the spec, as shown in the following example:

      apiVersion: ogx.io/v1beta1
      kind: OGXServer
      metadata:
        name: my-ogx
        namespace: my-ogx-namespace
      spec:
        distribution:
          name: rh-dev
        workload:
          replicas: 1
        tlsConfig:
          caBundle:
            configMapName: odh-trusted-ca-bundle 

      1

      
            configMapKeys: 

      2

      
              - ca-bundle.crt
              - odh-ca-bundle.crt

      1

      Specifies the name of the source config map.

      2

      Specifies the keys in the source config map that the OGX Operator reads. The OGX Operator concatenates the certificates from all listed keys into a single bundle.

   6. Click Save. The OGX server pod automatically redeploys with the updated bundle.

Verification

1. In a terminal window, log in to your OpenShift cluster from the OpenShift CLI (oc):

   $ oc login --token=<token> --server=<openshift_cluster_url>

2. Confirm that the OGX Operator created the managed CA bundle config map. The managed config map is named <instance-name>-ca-bundle:

   $ oc get configmap <instance-name>-ca-bundle -n <namespace>

   The output shows the managed config map.

3. Confirm that the OGXServer CR reports a successful CA bundle configuration in its status:

   $ oc get ogxserver <instance-name> -n <namespace> -o yaml

   In the status.conditions field of the output, verify that the DeploymentReady condition has status: "True" and that no condition reports a CA bundle validation failure in its message field.

4. Confirm that the OGX server pod is running with the CA bundle mounted:

   $ oc get pods -n <namespace> -l app.kubernetes.io/instance=<instance-name>

5. Confirm that the CA bundle file is present in the server container at the expected mount path and that the SSL_CERT_FILE environment variable points to it:

   $ oc exec -n <namespace> <pod-name> -- ls -l /etc/ssl/certs/ca-bundle/ca-bundle.crt

   $ oc exec -n <namespace> <pod-name> -- printenv SSL_CERT_FILE

   The output of the printenv command shows /etc/ssl/certs/ca-bundle/ca-bundle.crt.

6. Confirm that the bundle file contains the expected CA certificates:

   $ oc exec -n <namespace> <pod-name> -- \
     head -20 /etc/ssl/certs/ca-bundle/ca-bundle.crt

   The output begins with a -----BEGIN CERTIFICATE----- line, followed by the encoded certificate data of the first certificate in the bundle.

7. Confirm that the OGX server can establish a trusted TLS connection to your external endpoint by sending a test request from inside the pod:

   $ oc exec -n <namespace> <pod-name> -- \
     curl -sS -o /dev/null -w "%{http_code}\n" <external-endpoint-url>

   A successful HTTP status code, such as 200, indicates that the certificate chain is validated. A curl error that mentions a self-signed certificate or a certificate verification failure indicates that the bundle does not include the correct issuing CA.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.

2. Go to the Installed Operators page. The navigation path depends on your OpenShift version:

   • On OpenShift 4.20 and later, click Ecosystem Installed Operators.
   • On OpenShift 4.19, click Operators Installed Operators.
3. Click the Red Hat OpenShift AI Operator.
4. Click the DSC Initialization tab.
5. Click the default-dsci object.
6. Click the YAML tab.

7. In the spec section, change the value of the managementState field for trustedCABundle to Removed:

   spec:
     trustedCABundle:
       managementState: Removed

8. Click Save.