Chapter 1. Managing data science pipelines

Procedure

1. Before configuring your pipeline server, from Amazon RDS: Certificate bundles by AWS Region, download the PEM certificate bundle for the region that the database was created in.

   For example, if the database was created in the us-east-1 region, download us-east-1-bundle.pem.

2. In a terminal window, log in to the OpenShift cluster where OpenShift AI is deployed.

   oc login api.<cluster_name>.<cluster_domain>:6443 --web

3. Run the following command to fetch the current OpenShift AI trusted CA configuration and store it in a new file:

   oc get dscinitializations.dscinitialization.opendatahub.io default-dsci -o json | jq '.spec.trustedCABundle.customCABundle' > /tmp/my-custom-ca-bundles.crt

4. Run the following command to append the PEM certificate bundle that you downloaded to the new custom CA configuration file:

   cat us-east-1-bundle.pem >> /tmp/my-custom-ca-bundles.crt

5. Run the following command to update the OpenShift AI trusted CA configuration to trust certificates issued by the CAs included in the new custom CA configuration file:

   oc patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"$(awk '{printf "%s\\n", $0}' /tmp/my-custom-ca-bundles.crt)"'"}]'

6. Configure a pipeline server, as described in Configuring a pipeline server.

Procedure

1. Save the following code as a new file named compile.py in your working directory.

   The example uses the KubernetesManifestOptions class from the kfp.compiler.compiler_utils module to define pipeline metadata such as the name, version, and namespace.

   Example compile script

   from kfp import dsl, compiler
   from kfp.compiler.compiler_utils import KubernetesManifestOptions
   
   @dsl.pipeline(name="<pipeline_name>")
   def my_pipeline():
       pass  # define your tasks
   
   compiler.Compiler().compile(
       pipeline_func=my_pipeline,
       package_path="<output_file>.yaml",
       kubernetes_manifest_format=True,
       kubernetes_manifest_options=KubernetesManifestOptions(
           pipeline_name="<pipeline_name>",
           pipeline_version_name="<version_name>",
           namespace="<namespace>",
           include_pipeline_manifest=True,
       ),
   )

2. Run the script to compile your pipeline and generate the Kubernetes manifests:

   $ python compile.py

Procedure

1. In a terminal window, log in to your OpenShift cluster by using the OpenShift CLI (oc):

   $ oc login -u <user_name>

   When prompted, enter the OpenShift server URL, connection type, and your password.

2. To configure the pipeline server to use Kubernetes API storage instead of the default database option, set the spec.apiServer.pipelineStore field to kubernetes in your project’s DataSciencePipelinesApplication (DSPA) custom resource.

   In the following command, replace <dspa_name> with the name of your DSPA custom resource, and replace <namespace> with the name of your project:

   $ oc patch dspa <dspa_name> -n <namespace> \
     --type=merge \
     -p {"spec": {"apiServer": {"pipelineStore": "kubernetes"}}}

   Warning

   When you switch the pipeline server from database storage to Kubernetes API storage, existing pipelines that were stored in the internal database are no longer visible in the OpenShift AI dashboard or REST API. To view or manage those pipelines again, change the spec.apiServer.pipelineStore field back to database.

3. Define a Pipeline custom resource in a YAML file with the following contents:

   Example pipeline definition

   apiVersion: pipelines.kubeflow.org/v2beta1
   kind: Pipeline
   metadata:
     name: <name>
     namespace: <namespace>
   spec:
     displayName: <displayName>

   • name: The immutable Kubernetes resource name of your pipeline.
   • namespace: The name of your project.
   • displayName: The user-friendly display name of your pipeline, which is shown in the dashboard and REST API.

4. Apply the pipeline definition to create the Pipeline custom resource in your cluster.

   In the following command, replace <pipeline_yaml_file> with the name of your YAML file:

   Example command

   $ oc apply -f <pipeline_yaml_file>.yaml

5. Alternatively, if you compiled Kubernetes-native manifests with the KFP SDK, you can apply the generated file directly without manually creating separate YAML files:

   $ oc apply -f <output_file>.yaml

   The generated file includes both Pipeline and PipelineVersion resources. You can skip the following manual definition steps and proceed to the verification step.

6. Define a PipelineVersion custom resource in a YAML file with the following contents:

   Example pipeline version definition

   apiVersion: pipelines.kubeflow.org/v2beta1
   kind: PipelineVersion
   metadata:
     name: <name>
     namespace: <namespace>
   spec:
     pipelineName: <pipelineName>
     displayName: <displayName>
     description: This is the first version of the pipeline.
     pipelineSpec:
           # ... YAML generated by compiling Python pipeline with KFP SDK ...

   • name: The name of your pipeline version.
   • namespace: The name of your project.
   • pipelineName: The immutable Kubernetes resource name of your pipeline. This value must match the metadata.name value in the Pipeline custom resource.
   • displayName: The user-friendly display name of your pipeline version, which is shown in the dashboard and REST API.
   • pipelineSpec: The YAML content that you generated by using the Kubeflow Pipelines (KFP) SDK.

7. Apply the pipeline version definition to create the PipelineVersion custom resource in your cluster.

   In the following command, replace <pipeline_version_yaml_file> with the name of your YAML file:

   Example command

   $ oc apply -f <pipeline_version_yaml_file>.yaml

   After creating the pipeline version, the system automatically applies the following labels to the pipeline version for easier filtering:

   Example automatic labels

   pipelines.kubeflow.org/pipeline-id: <metadata.uid of the pipeline>
   pipelines.kubeflow.org/pipeline: <pipeline name>

Verification

1. Check that the Pipeline custom resource was successfully created:

   $ oc get pipeline <pipeline_name> -n <namespace>

2. Check that the PipelineVersion custom resource was successfully created:

   $ oc get pipelineversion <pipeline_version_name> -n <namespace>

Procedure

1. In a terminal window, log in to your OpenShift cluster by using the OpenShift CLI (oc):

   $ oc login -u <user_name>

   When prompted, enter the OpenShift server URL, connection type, and your password.

2. Set environment variables for your data science project and get the pipeline API route.

   In the export command, replace <namespace> with the name of your project:

   echo "Setting the prerequisite variables"
   export NAMESPACE=<namespace>
   export DSPA_NAME=$(oc -n $NAMESPACE get dspa -o jsonpath={.items[0].metadata.name})
   export API_URL="https://$(oc -n $NAMESPACE get route "ds-pipeline-$DSPA_NAME" -o jsonpath={.spec.host})"

3. Create a Python virtual environment and install the required dependencies.

   echo "Set up the Python prerequisites"
   python3.11 -m venv .venv
   ./.venv/bin/pip install kfp requests PyYAML

4. Download and run the Kubeflow Pipelines community migration script.

   The script connects to the Data Science Pipelines API, exports all pipelines and versions from the specified data science project, and generates one YAML file per pipeline in a local kfp-exported-pipelines/ directory. Each file includes a Pipeline resource followed by all associated PipelineVersion resources.

   1. Run the following command:

      curl -L https://raw.githubusercontent.com/kubeflow/pipelines/refs/heads/master/tools/k8s-native/migration.py -o migration.py
      ./.venv/bin/python migration.py --skip-tls-verify --kfp-server-host $API_URL --namespace $NAMESPACE --token "$(oc whoami --show-token)"

      Note

      The --skip-tls-verify option disables certificate validation and should be used only in development environments or when connecting to a server with a self-signed certificate. In production environments, provide a valid certificate bundle instead.

      Additionally, passing the access token directly on the command line might expose it in shell history or process lists. To reduce this risk, store the token in an environment variable and reference it in your command:

      export KFP_TOKEN=$(oc whoami --show-token)
      ./.venv/bin/python migration.py --kfp-server-host $API_URL --namespace $NAMESPACE --token "$KFP_TOKEN"

      Alternatively, use a prompt with read -s to input the token securely at runtime.

   2. Optional: For more information about the script, run the following command:

      ./.venv/bin/python migration.py --help

   3. If you plan to create new or updated PipelineVersion custom resources after migration, you can compile your pipeline code by using the Kubeflow Pipelines SDK. For more information, see Compiling the pipeline YAML with the Kubeflow Pipelines SDK and Compiling Kubernetes-native manifests with the Kubeflow Pipelines SDK.

5. Apply the exported Kubernetes custom resources to your cluster.

   oc apply -f ./kfp-exported-pipelines

6. Change the pipeline server to use Kubernetes API storage.

   oc -n "$NAMESPACE" patch dspa "$DSPA_NAME" --type=merge -p {"spec":{"apiServer":{"pipelineStore":"kubernetes"}}}

   Note

   To view pipelines that were stored in the internal database and not migrated, you can temporarily change the pipeline server back to database storage.

   oc -n $NAMESPACE patch dspa $DSPA_NAME --type=merge -p {"spec":{"apiServer":{"pipelineStore":"database"}}}

7. Repeat this procedure for each additional data science project that you want to migrate, changing NAMESPACE to the appropriate project name.

8. Optional: Clean up the local environment.

   rm -rf .venv migration.py

Verification

1. Check that the Pipeline and PipelineVersion custom resources were created in your project:

   $ oc -n <namespace> get pipelines.pipelines.kubeflow.org
   $ oc -n <namespace> get pipelineversions.pipelines.kubeflow.org

2. Verify that the pipeline server is using Kubernetes API storage:

   $ oc -n <namespace> get dspa <dspa_name> -o jsonpath={.spec.apiServer.pipelineStore}{"\n"}

   The command should return kubernetes.