Chapter 1. Managing AI pipelines

Procedure

1. Before configuring your pipeline server, from Amazon RDS: Certificate bundles by AWS Region, download the PEM certificate bundle for the region that the database was created in.

   For example, if the database was created in the us-east-1 region, download us-east-1-bundle.pem.

2. In a terminal window, log in to the OpenShift cluster where OpenShift AI is deployed.

   oc login api.<cluster_name>.<cluster_domain>:6443 --web

   Copy to Clipboard Toggle word wrap

3. Run the following command to fetch the current OpenShift AI trusted CA configuration and store it in a new file:

   oc get dscinitializations.dscinitialization.opendatahub.io default-dsci -o json | jq '.spec.trustedCABundle.customCABundle' > /tmp/my-custom-ca-bundles.crt

   Copy to Clipboard Toggle word wrap

4. Run the following command to append the PEM certificate bundle that you downloaded to the new custom CA configuration file:

   cat us-east-1-bundle.pem >> /tmp/my-custom-ca-bundles.crt

   Copy to Clipboard Toggle word wrap

5. Run the following command to update the OpenShift AI trusted CA configuration to trust certificates issued by the CAs included in the new custom CA configuration file:

   oc patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"$(awk '{printf "%s\\n", $0}' /tmp/my-custom-ca-bundles.crt)"'"}]'

   Copy to Clipboard Toggle word wrap
6. Configure a pipeline server, as described in Configuring a pipeline server.

Procedure

1. Save the following code as a new file named compile.py in your working directory.

   The example uses the KubernetesManifestOptions class from the kfp.compiler.compiler_utils module to define pipeline metadata such as the name, version, and namespace.

   Example compile script

   from kfp import dsl, compiler
   from kfp.compiler.compiler_utils import KubernetesManifestOptions
   
   @dsl.pipeline(name="<pipeline_name>")
   def my_pipeline():
       pass  # define your tasks
   
   compiler.Compiler().compile(
       pipeline_func=my_pipeline,
       package_path="<output_file>.yaml",
       kubernetes_manifest_format=True,
       kubernetes_manifest_options=KubernetesManifestOptions(
           pipeline_name="<pipeline_name>",
           pipeline_version_name="<version_name>",
           namespace="<namespace>",
           include_pipeline_manifest=True,
       ),
   )

   Copy to Clipboard Toggle word wrap

2. Run the script to compile your pipeline and generate the Kubernetes manifests:

   $ python compile.py

   Copy to Clipboard Toggle word wrap

Procedure

1. Set environment variables for your project and pipeline server route:

   export NAMESPACE=<project_namespace>
   export DSPA_NAME=$(oc -n "$NAMESPACE" get dspa -o jsonpath='{.items[0].metadata.name}')
   export API_URL="https://$(oc -n "$NAMESPACE" get route "ds-pipeline-${DSPA_NAME}" -o jsonpath='{.spec.host}')"

   Copy to Clipboard Toggle word wrap

   Replace <project_namespace> with the name of your project.

2. Obtain an OpenShift access token for the current user:

   export OCP_TOKEN=$(oc whoami --show-token)

   Copy to Clipboard Toggle word wrap
   Note

   Avoid pasting the access token directly into commands or scripts. The token can appear in your shell history or in process listings if you pass it as a literal argument.

   To reduce this risk, store the token in an environment variable and reference it from your code or commands. For example:

   ./.venv/bin/python my_script.py --kfp-server-host "$API_URL" --namespace "$NAMESPACE" --token "$OCP_TOKEN"

   Copy to Clipboard Toggle word wrap

   Alternatively, use a prompt with read -s to input the token securely at runtime.

3. Optional: If you are running outside the cluster or you use a custom or self-signed certificate, set an environment variable for your trusted certificate bundle:

   export SSL_CA_CERT=/etc/pki/tls/custom-certs/ca-bundle.crt

   Copy to Clipboard Toggle word wrap

   Adjust the path if your environment uses a different certificate location.

4. In your Python environment, create a KFP client that uses the pipeline server route and OpenShift access token:

   import os
   from kfp.client import Client
   
   api_url = os.environ["API_URL"]
   token = os.environ["OCP_TOKEN"]
   namespace = os.environ["NAMESPACE"]
   
   # Optional: Use a custom certificate bundle if required
   ssl_ca_cert = os.environ.get("SSL_CA_CERT", None)
   
   client_args = {
       "host": api_url,
       "existing_token": token,
       "namespace": namespace,
   }
   
   if ssl_ca_cert:
       client_args["ssl_ca_cert"] = ssl_ca_cert
   
   client = Client(**client_args)

   Copy to Clipboard Toggle word wrap

5. Verify the connection by calling the API. For example, list experiments or pipelines:

   print(client.list_experiments())
   # or
   print(client.list_pipelines())

   Copy to Clipboard Toggle word wrap

Procedure

1. In a terminal window, log in to your OpenShift cluster by using the OpenShift CLI (oc):

   $ oc login -u <user_name>

   Copy to Clipboard Toggle word wrap

   When prompted, enter the OpenShift server URL, connection type, and your password.

2. To configure the pipeline server to use Kubernetes API storage instead of the default database option, set the spec.apiServer.pipelineStore field to kubernetes in your project’s DataSciencePipelinesApplication (DSPA) custom resource.

   In the following command, replace <dspa_name> with the name of your DSPA custom resource, and replace <namespace> with the name of your project:

   $ oc patch dspa <dspa_name> -n <namespace> \
     --type=merge \
     -p {"spec": {"apiServer": {"pipelineStore": "kubernetes"}}}

   Copy to Clipboard Toggle word wrap
   Warning

   When you switch the pipeline server from database storage to Kubernetes API storage, existing pipelines that were stored in the internal database are no longer visible in the OpenShift AI dashboard or REST API. To view or manage those pipelines again, change the spec.apiServer.pipelineStore field back to database.

3. Define a Pipeline custom resource in a YAML file with the following contents:

   Example pipeline definition

   apiVersion: pipelines.kubeflow.org/v2beta1
   kind: Pipeline
   metadata:
     name: <name>
     namespace: <namespace>
   spec:
     displayName: <displayName>

   Copy to Clipboard Toggle word wrap

   • name: The immutable Kubernetes resource name of your pipeline.
   • namespace: The name of your project.
   • displayName: The user-friendly display name of your pipeline, which is shown in the dashboard and REST API.

4. Apply the pipeline definition to create the Pipeline custom resource in your cluster.

   In the following command, replace <pipeline_yaml_file> with the name of your YAML file:

   Example command

   $ oc apply -f <pipeline_yaml_file>.yaml

   Copy to Clipboard Toggle word wrap

5. Alternatively, if you compiled Kubernetes-native manifests with the KFP SDK, you can apply the generated file directly without manually creating separate YAML files:

   $ oc apply -f <output_file>.yaml

   Copy to Clipboard Toggle word wrap

   The generated file includes both Pipeline and PipelineVersion resources. You can skip the following manual definition steps and proceed to the verification step.

6. Define a PipelineVersion custom resource in a YAML file with the following contents:

   Example pipeline version definition

   apiVersion: pipelines.kubeflow.org/v2beta1
   kind: PipelineVersion
   metadata:
     name: <name>
     namespace: <namespace>
   spec:
     pipelineName: <pipelineName>
     displayName: <displayName>
     description: This is the first version of the pipeline.
     pipelineSpec:
           # ... YAML generated by compiling Python pipeline with KFP SDK ...

   Copy to Clipboard Toggle word wrap

   • name: The name of your pipeline version.
   • namespace: The name of your project.
   • pipelineName: The immutable Kubernetes resource name of your pipeline. This value must match the metadata.name value in the Pipeline custom resource.
   • displayName: The user-friendly display name of your pipeline version, which is shown in the dashboard and REST API.
   • pipelineSpec: The YAML content that you generated by using the Kubeflow Pipelines (KFP) SDK.

7. Apply the pipeline version definition to create the PipelineVersion custom resource in your cluster.

   In the following command, replace <pipeline_version_yaml_file> with the name of your YAML file:

   Example command

   $ oc apply -f <pipeline_version_yaml_file>.yaml

   Copy to Clipboard Toggle word wrap

   After creating the pipeline version, the system automatically applies the following labels to the pipeline version for easier filtering:

   Example automatic labels

   pipelines.kubeflow.org/pipeline-id: <metadata.uid of the pipeline>
   pipelines.kubeflow.org/pipeline: <pipeline name>

   Copy to Clipboard Toggle word wrap

Verification

1. Check that the Pipeline custom resource was successfully created:

   $ oc get pipeline <pipeline_name> -n <namespace>

   Copy to Clipboard Toggle word wrap

2. Check that the PipelineVersion custom resource was successfully created:

   $ oc get pipelineversion <pipeline_version_name> -n <namespace>

   Copy to Clipboard Toggle word wrap

Procedure

1. In a terminal window, log in to your OpenShift cluster by using the OpenShift CLI (oc):

   $ oc login -u <user_name>

   Copy to Clipboard Toggle word wrap

   When prompted, enter the OpenShift server URL, connection type, and your password.

2. Set environment variables for your project and get the pipeline API route.

   In the export command, replace <namespace> with the name of your project:

   echo "Setting the prerequisite variables"
   export NAMESPACE=<namespace>
   export DSPA_NAME=$(oc -n $NAMESPACE get dspa -o jsonpath={.items[0].metadata.name})
   export API_URL="https://$(oc -n $NAMESPACE get route "ds-pipeline-$DSPA_NAME" -o jsonpath={.spec.host})"

   Copy to Clipboard Toggle word wrap

3. Create a Python virtual environment and install the required dependencies.

   echo "Set up the Python prerequisites"
   python3.11 -m venv .venv
   ./.venv/bin/pip install kfp requests PyYAML

   Copy to Clipboard Toggle word wrap

4. Download and run the Kubeflow Pipelines community migration script.

   The script connects to the AI Pipelines API, exports all pipelines and versions from the specified project, and generates one YAML file per pipeline in a local kfp-exported-pipelines/ directory. Each file includes a Pipeline resource followed by all associated PipelineVersion resources.

   1. Run the following command:

      curl -L https://raw.githubusercontent.com/kubeflow/pipelines/refs/heads/master/tools/k8s-native/migration.py -o migration.py
      ./.venv/bin/python migration.py --skip-tls-verify --kfp-server-host $API_URL --namespace $NAMESPACE --token "$(oc whoami --show-token)"

      Copy to Clipboard Toggle word wrap
      Note

      The --skip-tls-verify option disables certificate validation and should be used only in development environments or when connecting to a server with a self-signed certificate. In production environments, provide a valid certificate bundle instead.

      Additionally, passing the access token directly on the command line might expose it in shell history or process lists. To reduce this risk, store the token in an environment variable and reference it in your command:

      export KFP_TOKEN=$(oc whoami --show-token)
      ./.venv/bin/python migration.py --kfp-server-host $API_URL --namespace $NAMESPACE --token "$KFP_TOKEN"

      Copy to Clipboard Toggle word wrap

      Alternatively, use a prompt with read -s to input the token securely at runtime.

   2. Optional: For more information about the script, run the following command:

      ./.venv/bin/python migration.py --help

      Copy to Clipboard Toggle word wrap
   3. If you plan to create new or updated PipelineVersion custom resources after migration, you can compile your pipeline code by using the Kubeflow Pipelines SDK. For more information, see Compiling the pipeline YAML with the Kubeflow Pipelines SDK and Compiling Kubernetes-native manifests with the Kubeflow Pipelines SDK.

5. Apply the exported Kubernetes custom resources to your cluster.

   oc apply -f ./kfp-exported-pipelines

   Copy to Clipboard Toggle word wrap

6. Change the pipeline server to use Kubernetes API storage.

   oc -n "$NAMESPACE" patch dspa "$DSPA_NAME" --type=merge -p {"spec":{"apiServer":{"pipelineStore":"kubernetes"}}}

   Copy to Clipboard Toggle word wrap
   Note

   To view pipelines that were stored in the internal database and not migrated, you can temporarily change the pipeline server back to database storage.

   oc -n $NAMESPACE patch dspa $DSPA_NAME --type=merge -p {"spec":{"apiServer":{"pipelineStore":"database"}}}

   Copy to Clipboard Toggle word wrap
7. Repeat this procedure for each additional project that you want to migrate, changing NAMESPACE to the appropriate project name.

8. Optional: Clean up the local environment.

   rm -rf .venv migration.py

   Copy to Clipboard Toggle word wrap

Verification

1. Check that the Pipeline and PipelineVersion custom resources were created in your project:

   $ oc -n <namespace> get pipelines.pipelines.kubeflow.org
   $ oc -n <namespace> get pipelineversions.pipelines.kubeflow.org

   Copy to Clipboard Toggle word wrap

2. Verify that the pipeline server is using Kubernetes API storage:

   $ oc -n <namespace> get dspa <dspa_name> -o jsonpath={.spec.apiServer.pipelineStore}{"\n"}

   Copy to Clipboard Toggle word wrap

   The command should return kubernetes.