Installing and uninstalling OpenShift AI Self-Managed

Procedure

1. Open a new terminal window.

2. Follow these steps to log in to your OpenShift cluster as a cluster administrator:

   1. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   2. After you have logged in, click Display token.

   3. Copy the Log in with this token command and paste it in your terminal.

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy to Clipboard Toggle word wrap

3. Create a namespace for installation of the Operator by performing the following actions:

   Note

   If you have already created a custom namespace for the Operator, you can skip this step.

   1. Create a namespace YAML file named rhods-operator-namespace.yaml.

      apiVersion: v1
      kind: Namespace
      metadata:
        name: redhat-ods-operator 

      1

      Copy to Clipboard Toggle word wrap

      1

      Defines the operator namespace.

   2. Create the namespace in your OpenShift cluster.

      $ oc create -f rhods-operator-namespace.yaml

      Copy to Clipboard Toggle word wrap

      You see output similar to the following:

      namespace/redhat-ods-operator created

      Copy to Clipboard Toggle word wrap

4. Create an operator group for installation of the Operator by performing the following actions:

   1. Create an OperatorGroup object custom resource (CR) file, for example, rhods-operator-group.yaml.

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: rhods-operator
        namespace: redhat-ods-operator 

      1

      Copy to Clipboard Toggle word wrap

      1

      Defines the operator namespace.

   2. Create the OperatorGroup object in your OpenShift cluster.

      $ oc create -f rhods-operator-group.yaml

      Copy to Clipboard Toggle word wrap

      You see output similar to the following:

      operatorgroup.operators.coreos.com/rhods-operator created

      Copy to Clipboard Toggle word wrap

5. Create a subscription for installation of the Operator by performing the following actions:

   1. Create a Subscription object CR file, for example, rhods-operator-subscription.yaml.

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: rhods-operator
        namespace: redhat-ods-operator 

      1

      
      spec:
        name: rhods-operator
        channel: <channel> 

      2

      
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: rhods-operator.x.y.z 

      3

      Copy to Clipboard Toggle word wrap

      1

      Defines the operator namespace.

      2

      Sets the update channel. You must specify a value of fast, fast-x.y, stable, stable-x.y eus-x.y, or alpha. For more information, see Understanding update channels.

      3

      Optional: Sets the operator version. If you do not specify a value, the subscription defaults to the latest operator version. For more information, see the Red Hat OpenShift AI Self-Managed Life Cycle Knowledgebase article.

   2. Create the Subscription object in your OpenShift cluster to install the Operator.

      $ oc create -f rhods-operator-subscription.yaml

      Copy to Clipboard Toggle word wrap

      You see output similar to the following:

      subscription.operators.coreos.com/rhods-operator created

      Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators → OperatorHub.
3. On the OperatorHub page, locate the Red Hat OpenShift AI Operator by scrolling through the available Operators or by typing Red Hat OpenShift AI into the Filter by keyword box.
4. Click the Red Hat OpenShift AI tile. The Red Hat OpenShift AI information pane opens.
5. Select a Channel. For information about subscription update channels, see Understanding update channels.
6. Select a Version.
7. Click Install. The Install Operator page opens.
8. Review or change the selected channel and version as needed.
9. For Installation mode, note that the only available value is All namespaces on the cluster (default). This installation mode makes the Operator available to all namespaces in the cluster.

10. For Installed Namespace, choose one of the following options:

    • To use the predefined operator namespace, select the Operator recommended Namespace: redhat-ods-operator option.
    • To use the custom operator namespace that you created, select the Select a Namespace option, and then select the namespace from the drop-down list.

11. For Update approval, select one of the following update strategies:

    • Automatic: New updates in the update channel are installed as soon as they become available.

    • Manual: A cluster administrator must approve any new updates before installation begins.

      Important

      By default, the Red Hat OpenShift AI Operator follows a sequential update process. This means that if there are several versions between the current version and the target version, Operator Lifecycle Manager (OLM) upgrades the Operator to each of the intermediate versions before it upgrades it to the final, target version.

      If you configure automatic upgrades, OLM automatically upgrades the Operator to the latest available version. If you configure manual upgrades, a cluster administrator must manually approve each sequential update between the current version and the final, target version.

      For information about supported versions, see the Red Hat OpenShift AI Life Cycle Knowledgebase article.

12. Click Install.

    The Installing Operators pane appears. When the installation finishes, a checkmark appears next to the Operator name.

Procedure

1. Open a new terminal window.

2. Follow these steps to log in to your OpenShift cluster as a cluster administrator:

   1. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   2. After you have logged in, click Display token.

   3. Copy the Log in with this token command and paste it in your terminal.

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy to Clipboard Toggle word wrap

3. Create a DataScienceCluster object custom resource (CR) file, for example, rhods-operator-dsc.yaml.

   apiVersion: datasciencecluster.opendatahub.io/v2
   kind: DataScienceCluster
   metadata:
     name: default-dsc
   spec:
     components:
       aipipelines:
         argoWorkflowsControllers:
           managementState: Removed 

   1

   
         managementState: Removed
       dashboard:
         managementState: Removed
       feastoperator:
         managementState: Removed
       kserve:
         managementState: Removed
       kueue:
         defaultClusterQueueName: default
         defaultLocalQueueName: default
         managementState: Removed
       llamastackoperator:
         managementState: Removed
       modelregistry:
         managementState: Removed
         registriesNamespace: rhoai-model-registries
       ray:
         managementState: Removed
       trainingoperator:
         managementState: Removed
       trustyai:
         managementState: Removed
       workbenches:
         managementState: Removed
         workbenchNamespace: rhods-notebooks 

   2

   Copy to Clipboard Toggle word wrap

   1

   To use your own Argo Workflows instance with the aipipelines component, set argoWorkflowsControllers.managementState to Removed. This allows you to integrate with a managed Argo Workflows installation already on your OpenShift cluster and avoid conflicts with the embedded controller. See Configuring pipelines with your own Argo Workflows instance.

   2

   To use the predefined workbench namespace, set this value to rhods-notebooks or omit this line. To use a custom workbench namespace, set this value to your namespace.

4. In the spec.components section of the CR, for each OpenShift AI component shown, set the value of the managementState field to either Managed or Removed. These values are defined as follows:

   Managed

   The Operator actively manages the component, installs it, and tries to keep it active. The Operator will upgrade the component only if it is safe to do so.

   Removed

   The Operator actively manages the component but does not install it. If the component is already installed, the Operator will try to remove it.

   Important
   • To learn how to install the distributed workloads components, see Installing the distributed workloads components.

5. Create the DataScienceCluster object in your OpenShift cluster to install the specified OpenShift AI components.

   $ oc create -f rhods-operator-dsc.yaml

   Copy to Clipboard Toggle word wrap

   You see output similar to the following:

   datasciencecluster.datasciencecluster.opendatahub.io/default created

   Copy to Clipboard Toggle word wrap

Verification

1. Confirm that there is at least one running pod for each component:

   1. In the OpenShift web console, click Workloads → Pods.
   2. In the Project list at the top of the page, select redhat-ods-applications.
   3. In the applications namespace, confirm that there are one or more running pods for each of the OpenShift AI components that you installed.

2. Confirm the status of all installed components:

   1. In the OpenShift web console, click Operators → Installed Operators.
   2. Click the Red Hat OpenShift AI Operator.
   3. Click the Data Science Cluster tab.

   4. For the DataScienceCluster object called default-dsc, verify that the status is Phase: Ready.

      Note

      When you edit the spec.components section to change the installation status of a component, the default-dsc status also changes. During the initial installation, it might take a few minutes for the status phase to change from Progressing to Ready. You can access the OpenShift AI dashboard before the default-dsc status phase is Ready, but all components might not be ready.

   5. Click the default-dsc link to display the data science cluster details.
   6. Select the YAML tab.

   7. In the status.installedComponents section, confirm that the components you installed have a status value of true.

      Note

      If a component shows with the component-name: {} format in the spec.components section of the CR, the component is not installed.

3. In the OpenShift AI dashboard, users can view the list of the installed OpenShift AI components, their corresponding source (upstream) components, and the versions of the installed components, as described in Viewing installed OpenShift AI components.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators → Installed Operators and then click the Red Hat OpenShift AI Operator.
3. Click the Data Science Cluster tab.
4. Click Create DataScienceCluster.

5. For Configure via, select YAML view.

   An embedded YAML editor opens showing a default custom resource (CR) for the DataScienceCluster object, similar to the following example:

   apiVersion: datasciencecluster.opendatahub.io/v2
   kind: DataScienceCluster
   metadata:
     name: default-dsc
   spec:
     components:
       aipipelines:
         argoWorkflowsControllers:
           managementState: Removed 

   1

   
         managementState: Removed
       dashboard:
         managementState: Removed
       feastoperator:
         managementState: Removed
       kserve:
         managementState: Removed
       kueue:
         defaultClusterQueueName: default
         defaultLocalQueueName: default
         managementState: Removed
       llamastackoperator:
         managementState: Removed
       modelregistry:
         managementState: Removed
         registriesNamespace: rhoai-model-registries
       ray:
         managementState: Removed
       trainingoperator:
         managementState: Removed
       trustyai:
         managementState: Removed
       workbenches:
         managementState: Removed
         workbenchNamespace: rhods-notebooks 

   2

   Copy to Clipboard Toggle word wrap

   1

   To use your own Argo Workflows instance with the aipipelines component, set argoWorkflowsControllers.managementState to Removed. This allows you to integrate with a managed Argo Workflows installation already on your OpenShift cluster and avoid conflicts with the embedded controller. See Configuring pipelines with your own Argo Workflows instance.

   2

   To use the predefined workbench namespace, set this value to rhods-notebooks or omit this line. To use a custom workbench namespace, set this value to your namespace.

6. In the spec.components section of the CR, for each OpenShift AI component shown, set the value of the managementState field to either Managed or Removed. These values are defined as follows:

   Managed

   The Operator actively manages the component, installs it, and tries to keep it active. The Operator will upgrade the component only if it is safe to do so.

   Removed

   The Operator actively manages the component but does not install it. If the component is already installed, the Operator will try to remove it.

   Important
   • To learn how to fully install the KServe component, which is used by the single-model serving platform to serve large models, see Installing the model serving platform.
   • To learn how to install the distributed workloads components, see Installing the distributed workloads components.
7. Click Create.

Verification

1. Confirm the status of all installed components:

   1. In the OpenShift web console, click Operators → Installed Operators.
   2. Click the Red Hat OpenShift AI Operator.
   3. Click the Data Science Cluster tab.

   4. For the DataScienceCluster object called default-dsc, verify that the status is Phase: Ready.

      Note

      When you edit the spec.components section to change the installation status of a component, the default-dsc status also changes. During the initial installation, it might take a few minutes for the status phase to change from Progressing to Ready. You can access the OpenShift AI dashboard before the default-dsc status phase is Ready, but all components might not be ready.

   5. Click the default-dsc link to display the data science cluster details.
   6. Select the YAML tab.

   7. In the status.installedComponents section, confirm that the components you installed have a status value of true.

      Note

      If a component shows with the component-name: {} format in the spec.components section of the CR, the component is not installed.

2. Confirm that there is at least one running pod for each component:

   1. In the OpenShift web console, click Workloads → Pods.
   2. In the Project list at the top of the page, select redhat-ods-applications or your custom applications namespace.
   3. In the applications namespace, confirm that there are one or more running pods for each of the OpenShift AI components that you installed.
3. In the OpenShift AI dashboard, users can view the list of the installed OpenShift AI components, their corresponding source (upstream) components, and the versions of the installed components, as described in Viewing installed OpenShift AI components.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators → Installed Operators and then click the Red Hat OpenShift AI Operator.
3. Click the Data Science Cluster tab.
4. On the DataScienceClusters page, click the default-dsc object.

5. Click the YAML tab.

   An embedded YAML editor opens showing the default custom resource (CR) for the DataScienceCluster object, similar to the following example:

   apiVersion: datasciencecluster.opendatahub.io/v2
   kind: DataScienceCluster
   metadata:
     name: default-dsc
   spec:
     components:
       aipipelines:
         argoWorkflowsControllers:
           managementState: Removed
         managementState: Removed
       dashboard:
         managementState: Removed
       feastoperator:
         managementState: Removed
       kserve:
         managementState: Removed
       kueue:
         defaultClusterQueueName: default
         defaultLocalQueueName: default
         managementState: Removed
       llamastackoperator:
         managementState: Removed
       modelregistry:
         managementState: Removed
         registriesNamespace: rhoai-model-registries
       ray:
         managementState: Removed
       trainingoperator:
         managementState: Removed
       trustyai:
         managementState: Removed
       workbenches:
         managementState: Removed
         workbenchNamespace: rhods-notebooks

   Copy to Clipboard Toggle word wrap

6. In the spec.components section of the CR, for each OpenShift AI component shown, set the value of the managementState field to either Managed or Removed. These values are defined as follows:

   Managed

   The Operator actively manages the component, installs it, and tries to keep it active. The Operator will upgrade the component only if it is safe to do so.

   Removed

   The Operator actively manages the component but does not install it. If the component is already installed, the Operator will try to remove it.

   Important
   • To learn how to install the KServe component, which is used by the single-model serving platform to serve large models, see Installing the model serving platform.
   • To learn how to install the distributed workloads feature, see Installing the distributed workloads components.

7. Click Save.

   For any components that you updated, OpenShift AI initiates a rollout that affects all pods to use the updated image.

8. If you are upgrading from OpenShift AI 2.19 or earlier, upgrade the Authorino Operator to the stable update channel, version 1.2.1 or later.

   1. Update Authorino to the latest available release in the tech-preview-v1 channel (1.1.2), if you have not done so already.

   2. Switch to the stable channel:

      1. Navigate to the Subscription settings of the Authorino Operator.
      2. Under Update channel, click on the highlighted tech-preview-v1.
      3. Change the channel to stable.
   3. Select the update option for Authorino 1.2.1.

Verification

1. Confirm that there is at least one running pod for each component:

   1. In the OpenShift web console, click Workloads → Pods.
   2. In the Project list at the top of the page, select redhat-ods-applications or your custom applications namespace.
   3. In the applications namespace, confirm that there are one or more running pods for each of the OpenShift AI components that you installed.

2. Confirm the status of all installed components:

   1. In the OpenShift web console, click Operators → Installed Operators.
   2. Click the Red Hat OpenShift AI Operator.
   3. Click the Data Science Cluster tab and select the DataScienceCluster object called default-dsc.
   4. Select the YAML tab.

   5. In the status.installedComponents section, confirm that the components you installed have a status value of true.

      Note

      If a component shows with the component-name: {} format in the spec.components section of the CR, the component is not installed.

3. In the OpenShift AI dashboard, users can view the list of the installed OpenShift AI components, their corresponding source (upstream) components, and the versions of the installed components, as described in Viewing installed OpenShift AI components.

Procedure

1. Log in to the OpenShift AI dashboard.
2. In the top navigation bar, click the help icon ( ) and then select About.

Procedure

1. In a terminal window, log in to the OpenShift CLI (oc) as shown in the following example:

   oc login api.<cluster_name>.<cluster_domain>:6443 --web

   Copy to Clipboard Toggle word wrap

2. Retrieve the current OpenShift AI trusted CA configuration and store it in a new file:

   oc get dscinitializations.dscinitialization.opendatahub.io default-dsci -o json | jq -r '.spec.trustedCABundle.customCABundle' > /tmp/my-custom-ca-bundles.crt

   Copy to Clipboard Toggle word wrap

3. Add the cluster’s kube-root-ca.crt ConfigMap to the OpenShift AI trusted CA configuration:

   oc get configmap kube-root-ca.crt -o jsonpath="{['data']['ca\.crt']}" >> /tmp/my-custom-ca-bundles.crt

   Copy to Clipboard Toggle word wrap

4. Update the OpenShift AI trusted CA configuration to trust certificates issued by the certificate authorities in kube-root-ca.crt:

   oc patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"$(awk '{printf "%s\\n", $0}' /tmp/my-custom-ca-bundles.crt)"'"}]'

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift console.

2. From Workloads → ConfigMaps, create a ConfigMap with the required bundle in the same project as the target AI pipeline:

   kind: ConfigMap
   apiVersion: v1
   metadata:
       name: custom-ca-bundle
   data:
       ca-bundle.crt: |
       # contents of ca-bundle.crt

   Copy to Clipboard Toggle word wrap

3. Add the following snippet to the .spec.apiserver.caBundle field of the underlying DataSciencePipelinesApplication (DSPA):

   apiVersion: datasciencepipelinesapplications.opendatahub.io/v1
   kind: DataSciencePipelinesApplication
   metadata:
       name: data-science-dspa
   spec:
       ...
       apiServer:
       ...
       cABundle:
           configMapName: custom-ca-bundle
           configMapKey: ca-bundle.crt

   Copy to Clipboard Toggle word wrap
4. Save the ConfigMap. The pipeline server pod automatically redeploys with the updated bundle.

1. Log in to the OpenShift console.
2. Go to the project that has the target AI pipeline.
3. Click the Pods tab.
4. Click the pipeline server pod with the ds-pipeline-dspa-<hash> prefix.
5. Click Terminal.
6. Enter cat /dsp-custom-certs/dsp-ca.crt.
7. Verify that your CA bundle is present within this file.

Procedure

1. Log in to the OpenShift console.
2. From the list of projects, open the openshift-ingress project.
3. Click YAML.
4. Search for "cert" to find a secret with a name that includes "cert". For example, rhods-internal-primary-cert-bundle-secret. The contents of the secret should contain two items that are used for all OpenShift Routes: tls.cert (the certificate) and tls.key (the key).
5. Copy the reference to the secret.
6. From the list of projects, open the istio-system project.
7. Create a YAML file and paste the reference to the secret that you copied from the openshift-ingress YAML file.

8. Edit the YAML code to keep only the relevant content, as shown in the following example. Replace rhods-internal-primary-cert-bundle-secret with the name of your secret:

   kind: Secret
   apiVersion: v1
   metadata:
   name: rhods-internal-primary-cert-bundle-secret
   data:
   tls.crt: >-
       LS0tLS1CRUd...
   tls.key: >-
       LS0tLS1CRUd...
   type: kubernetes.io/tls

   Copy to Clipboard Toggle word wrap
9. Save the YAML file in the istio-system project.
10. Navigate to Operators → Installed Operators → Red Hat OpenShift AI.
11. Click Data Science Cluster*, and then click default-dsc → YAML.

12. Edit the kserve configuration section to refer to your secret as shown in the following example. Replace rhods-internal-primary-cert-bundle-secret with the name of the secret that you created in Step 8.

    kserve:
    devFlags: {}
    managementState: Managed
    serving:
        ingressGateway:
        certificate:
            secretName: rhods-internal-primary-cert-bundle-secret
            type: Provided
        managementState: Managed
        name: knative-serving

    Copy to Clipboard Toggle word wrap

Procedure

1. In the OpenShift web console, click Operators → Installed Operators and then click the Red Hat OpenShift AI Operator.
2. Click the DSC Initialization tab.
3. Click the default-dsci object.
4. Click the YAML tab.

5. In the spec section, change the value of the managementState field for trustedCABundle to Removed:

   spec:
     trustedCABundle:
       managementState: Removed

   Copy to Clipboard Toggle word wrap
6. Click Save.

1. Log in to the OpenShift CLI (oc).

2. Run the following command to stream logs from all Operator pods:

   for pod in $(oc get pods -l name=rhods-operator -n redhat-ods-operator -o name); do
     oc logs -f "$pod" -n redhat-ods-operator &
   done

   Copy to Clipboard Toggle word wrap

   The Operator pod logs open in your terminal.

   Tip

   Press Ctrl+C to stop viewing. To fully stop all log streams, run kill $(jobs -p).