Chapter 4. Working with data science pipelines

Procedure

1. From the OpenShift AI dashboard, click Data Science Projects.

   The Data science projects page opens.

2. Click the name of the project that you want to configure a pipeline server for.

   A project details page opens.

3. In the Pipelines section, click Configure a pipeline server.

   The Configure pipeline server dialog appears.

4. In the Object storage connection section, provide values for the mandatory fields:

   1. In the Access key field, enter the access key ID for the S3-compatible object storage provider.
   2. In the Secret key field, enter the secret access key for the S3-compatible object storage account that you specified.
   3. In the Endpoint field, enter the endpoint of your S3-compatible object storage bucket.

   4. In the Bucket field, enter the name of your S3-compatible object storage bucket.

      Important

      If you specify incorrect data connection settings, you cannot update these settings on the same pipeline server. Therefore, you must delete the pipeline server and configure another one.

5. In the Database section, click Show advanced database options to specify the database to store your pipeline data and select one of the following sets of actions:

   • Select Use default database stored on your cluster to deploy a MariaDB database in your project.

   • Select Connect to external MySQL database to add a new connection to an external database that your pipeline server can access.

     1. In the Host field, enter the database’s host name.
     2. In the Port field, enter the database’s port.
     3. In the Username field, enter the default user name that is connected to the database.
     4. In the Password field, enter the password for the default user account.
     5. In the Database field, enter the database name.
6. Click Configure.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project that you want to import a pipeline to.

3. Click Import pipeline.

   The Import pipeline dialog opens.

4. Enter the details for the pipeline that you are importing.

   1. In the Pipeline name field, enter a name for the pipeline that you are importing.
   2. In the Pipeline description field, enter a description for the pipeline that you are importing.

   3. Click Upload. Alternatively, drag the file from your local machine’s file system and drop it in the designated area in the Import pipeline dialog.

      A file browser opens.

   4. Navigate to the file containing the pipeline code and click Select.
   5. Click Import pipeline.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project whose pipeline that you want to download.

3. In the Pipeline name column, click the name of the pipeline that you want to download.

   The Pipeline details page opens displaying the Graph tab.

4. Click the YAML tab.

   The page reloads to display an embedded YAML editor showing the pipeline code.

5. Click the Download button ( ) to download the YAML file containing your pipeline’s code to your local machine.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project that contains the pipeline that you want to delete.

3. Click the action menu (⋮) beside the pipeline that you want to delete and click Delete pipeline.

   The Delete pipeline dialog opens.

4. Enter the pipeline name in the text field to confirm that you intend to delete it.
5. Click Delete pipeline.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project whose pipeline server you want to delete.

3. From the Pipeline server actions list, select Delete pipeline server.

   The Delete pipeline server dialog opens.

4. Enter the pipeline server’s name in the text field to confirm that you intend to delete it.
5. Click Delete.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project whose pipeline server you want to view.
3. From the Pipeline server actions list, select View pipeline server configuration.
4. When you have finished inspecting the pipeline server’s details, click Done.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the relevant project whose pipelines you want to view.
3. Study the pipelines on the list.
4. Optional: Click Expand ( ) on the relevant row to view the pipeline’s executed runs. If the pipeline does not contain any runs, click Create run to create one.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project that you want to upload a pipeline version to.

3. Click the Import pipeline dropdown list and select Upload new version.

   The Upload new version dialog opens.

4. Enter the details for the pipeline version that you are uploading.

   1. From the Pipeline list, select the pipeline that you want to upload your pipeline version to.
   2. In the Pipeline version name field, confirm the name for the pipeline version, and change it if necessary.
   3. In the Pipeline version description field, enter a description for the pipeline version.

   4. Click Upload. Alternatively, drag the file from your local machine’s file system and drop it in the designated area in the Upload new version dialog.

      A file browser opens.

   5. Navigate to the file containing the pipeline version code and click Select.
   6. Click Upload.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project that contains a version of a pipeline that you want to delete.
3. On the row containing the pipeline, click Expand ( ).
4. On the row containing the pipeline version that you want to delete, select the checkbox.

5. Click the action menu (⋮) next to the Import pipeline dropdown and select Delete selected from the list.

   The Delete pipeline version dialog opens.

6. Enter the name of the pipeline version in the text field to confirm that you intend to delete it.
7. Click Delete.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project containing the pipeline versions that you want to view.
3. Click Expand ( ) on the row containing the pipeline that you want to view versions for.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project containing the pipeline versions that you want to view details for.
3. Click Expand ( ) on the row containing the pipeline that you want to view versions for.

4. Click the pipeline version that you want to view the details of.

   The Pipeline details page opens, displaying the Graph and YAML tabs.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project that you want to create a run for.

3. Click the action menu (⋮) beside the relevant pipeline and click Create run.

   The Create run page opens.

4. In the Name field, enter a name for the run.
5. In the Description field, enter a description for the run.
6. From the Pipeline list, select the pipeline that you want to create a run for. Alternatively, to create a new pipeline, click Create new pipeline and complete the relevant fields in the Import pipeline dialog.
7. From the Pipeline version list, select the pipeline version to create a run for. Alternatively, to upload a new version, click Upload new version and complete the relevant fields in the Upload new version dialog.

8. Configure the run type by performing one of the following sets of actions:

   • Select Run once immediately after creation to specify the run executes once, and immediately after its creation.

   • Select Schedule recurring run to schedule the run to recur.

     1. Configure the run’s trigger type.

        1. Select Periodic to specify an execution frequency. Enter a numerical value and select an execution frequency from the list.
        2. Select Cron to specify the execution schedule in cron format. This creates a cron job to execute the run. Click the Copy button ( ) to copy the cron job schedule to the clipboard. The field furthest to the left represents seconds. For more information about scheduling tasks using the supported cron format, see Cron Expression Format.

     2. Configure the run’s duration.

        1. Select the Start date check box to specify a start date for the run. Select the run’s start date using the Calendar and the start time from the list of times.
        2. Select the End date check box to specify an end date for the run. Select the run’s end date using the Calendar and the end time from the list of times.
9. Configure the input parameters for the run by selecting the parameters from the list.
10. Click Create.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Runs.

   The Runs page opens.

2. Click the action menu (⋮) beside the relevant run and click Clone.

   The Clone page opens.

3. From the Project list, select the project that contains the pipeline whose run that you want to clone.
4. In the Name field, enter a name for the run that you want to clone.
5. In the Description field, enter a description for the run that you want to clone.
6. From the Pipeline list, select the pipeline containing the run that you want to clone.

7. To configure the run type for the run that you are cloning, in the Run type section, perform one of the following sets of actions:

   • Select Run once immediately after create to specify the run that you are cloning executes once, and immediately after its creation. If you selected this option, skip to step 10.
   • Select Schedule recurring run to schedule the run that you are cloning to recur.

8. If you selected Schedule recurring run in the previous step, to configure the trigger type for the run, perform one of the following actions:

   • Select Periodic and select the execution frequency from the Run every list.
   • Select Cron to specify the execution schedule in cron format. This creates a cron job to execute the run. Click the Copy button ( ) to copy the cron job schedule to the clipboard. The field furthest to the left represents seconds. For more information about scheduling tasks using the supported cron format, see Cron Expression Format.

9. If you selected Schedule recurring run in step 7, configure the duration for the run that you are cloning.

   1. Select the Start date check box to specify a start date for the run. Select the start date using the calendar tool and the start time from the list of times.
   2. Select the End date check box to specify an end date for the run. Select the end date using the calendar tool and the end time from the list of times.
10. In the Parameters section, configure the input parameters for the run that you are cloning by selecting the appropriate parameters from the list.
11. Click Create.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Runs.

   The Runs page opens.

2. From the Project list, select the project that contains the pipeline whose triggered run you want to stop.

   The page refreshes to show the pipeline’s triggered runs on the Triggered tab.

3. Click the action menu (⋮) beside the triggered run that you want to delete and click Stop.

   There might be a short delay while the run stops.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Runs.

   The Runs page opens.

2. From the Project list, select the project that contains the pipeline whose scheduled run you want to delete.

   The page refreshes to show the pipeline’s scheduled runs on the Scheduled tab.

3. Click the action menu (⋮) beside the scheduled run that you want to delete and click Delete.

   The Delete scheduled run dialog opens.

4. Enter the run’s name in the text field to confirm that you intend to delete it.
5. Click Delete scheduled run.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Runs.

   The Runs page opens.

2. From the Project list, select the project that contains the pipeline whose triggered run you want to delete.

   The page refreshes to show the pipeline’s triggered runs on the Triggered tab.

3. Click the action menu (⋮) beside the triggered run that you want to delete and click Delete.

   The Delete triggered run dialog opens.

4. Enter the run’s name in the text field to confirm that you intend to delete it.
5. Click Delete triggered run.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Runs.

   The Runs page opens.

2. From the Project list, select the project whose scheduled pipeline runs you want to view.
3. Click the Scheduled tab.

4. Study the table showing a list of scheduled runs.

   After a run has been scheduled, the run’s status is displayed in the Status column in the table, indicating whether the run is ready for execution or unavailable for execution. To enable or disable a previously imported notebook image, on the row containing the relevant notebook image, click the toggle in the Enabled column.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Runs.

   The Runs page opens.

2. From the Project list, select the project whose previously executed pipeline runs you want to view.

   The Run details page opens.

3. Click the Triggered tab.

   A table opens that shows list of triggered runs. After a run has completed its execution, the run’s status is displayed in the Status column in the table, indicating whether the run has succeeded or failed.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project that you want to view run details for.
3. For a pipeline that you want to view run details for, click Expand ( ).

4. Click View runs on the row containing the project version that you want to view run details for.

   The Runs page opens displaying a list of previously executed pipeline runs.

5. Click the name of the run that you want to view the details of.

   The Run details page opens.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project that you want to view logs for.
3. For the pipeline that you want to view logs for, click Expand ( ).

4. Click View runs on the row containing the pipeline version that you want to view logs for.

   The Runs page opens displaying a list of previously executed pipeline runs.

5. Click the name of the run that you want to view logs for.

   The Run details page opens.

6. On the graph, click the pipeline step that you want to view logs for.

   A pane opens displaying information about the pipeline step.

7. Click the Logs tab.

   The log viewer opens.

8. To view the logs of another pipeline step, from the Steps list, select the step that you want to view logs for.

9. Analyze the log using the log viewer.

   • To search for a specific log message, enter at least part of the message in the search bar.
   • To view the full log in a separate browser window, click the action menu (⋮) and select View raw logs. Alternatively, to expand the size of the log viewer, click the action menu (⋮) and select Expand.

Procedure

1. From the OpenShift AI dashboard, click Data Science Pipelines Pipelines.

   The Pipelines page opens.

2. From the Project list, select the project that you want to download logs for.
3. For the pipeline that you want to download logs for, click Expand ( ).

4. Click View runs on the row containing the pipeline version that you want to download logs for.

   The Runs page opens displaying a list of previously executed pipeline runs.

5. Click the name of the run that you want to download logs for.

   The Run details page opens.

6. On the graph, click the pipeline step that you want to download logs for.

   A pane opens displaying information about the pipeline step.

7. Click the Logs tab.

   The log viewer opens.

8. Click the Download button ( ).

   1. Select Download current stop log to download the log for the current pipeline step.
   2. Select Download all step logs to download the logs for all steps in your pipeline run.

Procedure

1. After you open JupyterLab, confirm that the JupyterLab launcher is automatically displayed.

2. In the Elyra section of the JupyterLab launcher, click the Pipeline Editor tile.

   The Pipeline Editor opens.

Procedure

1. In the left sidebar of JupyterLab, click Runtimes ( ).

2. Click the Create new runtime configuration button ( ).

   The Add new Data Science Pipelines runtime configuration page opens.

3. Complete the relevant fields to define your runtime configuration.

   1. In the Display Name field, enter a name for your runtime configuration.
   2. Optional: In the Description field, enter a description to define your runtime configuration.
   3. Optional: In the Tags field, click Add Tag to define a category for your pipeline instance. Enter a name for the tag and press Enter.

   4. Define the credentials of your data science pipeline:

      1. In the Data Science Pipelines API Endpoint field, enter the API endpoint of your data science pipeline. Do not specify the pipelines namespace in this field.

      2. In the Public Data Science Pipelines API Endpoint field, enter the public API endpoint of your data science pipeline.

         Important

         You can obtain the Data Science Pipelines API endpoint from the Data Science Pipelines Runs page in the dashboard. Copy the relevant end point and enter it in the Public Data Science Pipelines API Endpoint field.

      3. Optional: In the Data Science Pipelines User Namespace field, enter the relevant user namespace to run pipelines.
      4. From the Data Science Pipelines engine list, select Tekton.

      5. From the Authentication Type list, select the authentication type required to authenticate your pipeline.

         Important

         If you created a notebook directly from the Jupyter tile on the dashboard, select EXISTING_BEARER_TOKEN from the Authentication Type list.

      6. In the Data Science Pipelines API Endpoint Username field, enter the user name required for the authentication type.

      7. In the Data Science Pipelines API Endpoint Password Or Token, enter the password or token required for the authentication type.

         Important

         To obtain the Data Science Pipelines API endpoint token, in the upper-right corner of the OpenShift web console, click your user name and select Copy login command. After you have logged in, click Display token and copy the value of --token= from the Log in with this token command.

   5. Define the connectivity information of your S3-compatible storage:

      1. In the Cloud Object Storage Endpoint field, enter the endpoint of your S3-compatible storage. For more information about Amazon s3 endpoints, see Amazon Simple Storage Service endpoints and quotas.
      2. Optional: In the Public Cloud Object Storage Endpoint field, enter the URL of your S3-compatible storage.
      3. In the Cloud Object Storage Bucket Name field, enter the name of the bucket where your pipeline artifacts are stored. If the bucket name does not exist, it is created automatically.
      4. From the Cloud Object Storage Authentication Type list, select the authentication type required to access to your S3-compatible cloud storage. If you use AWS S3 buckets, select KUBERNETES_SECRET from the list.
      5. In the Cloud Object Storage Credentials Secret field, enter the secret that contains the storage user name and password. This secret is defined in the relevant user namespace, if applicable. In addition, it must be stored on the cluster that hosts your pipeline runtime.
      6. In the Cloud Object Storage Username field, enter the user name to connect to your S3-compatible cloud storage, if applicable. If you use AWS S3 buckets, enter your AWS Secret Access Key ID.
      7. In the Cloud Object Storage Password field, enter the password to connect to your S3-compatible cloud storage, if applicable. If you use AWS S3 buckets, enter your AWS Secret Access Key.
   6. Click Save & Close.

Procedure

1. In the left sidebar of JupyterLab, click Runtimes ( ).

2. Hover the cursor over the runtime configuration that you want to update and click the Edit button ( ).

   The Data Science Pipelines runtime configuration page opens.

3. Fill in the relevant fields to update your runtime configuration.

   1. In the Display Name field, update name for your runtime configuration, if applicable.
   2. Optional: In the Description field, update the description of your runtime configuration, if applicable.
   3. Optional: In the Tags field, click Add Tag to define a category for your pipeline instance. Enter a name for the tag and press Enter.

   4. Define the credentials of your data science pipeline:

      1. In the Data Science Pipelines API Endpoint field, update the API endpoint of your data science pipeline, if applicable. Do not specify the pipelines namespace in this field.
      2. In the Public Data Science Pipelines API Endpoint field, update the API endpoint of your data science pipeline, if applicable.
      3. Optional: In the Data Science Pipelines User Namespace field, update the relevant user namespace to run pipelines, if applicable.
      4. From the Data Science Pipelines engine list, select Tekton.

      5. From the Authentication Type list, select a new authentication type required to authenticate your pipeline, if applicable.

         Important

         If you created a notebook directly from the Jupyter tile on the dashboard, select EXISTING_BEARER_TOKEN from the Authentication Type list.

      6. In the Data Science Pipelines API Endpoint Username field, update the user name required for the authentication type, if applicable.

      7. In the Data Science Pipelines API Endpoint Password Or Token, update the password or token required for the authentication type, if applicable.

         Important

         To obtain the Data Science Pipelines API endpoint token, in the upper-right corner of the OpenShift web console, click your user name and select Copy login command. After you have logged in, click Display token and copy the value of --token= from the Log in with this token command.

   5. Define the connectivity information of your S3-compatible storage:

      1. In the Cloud Object Storage Endpoint field, update the endpoint of your S3-compatible storage, if applicable. For more information about Amazon s3 endpoints, see Amazon Simple Storage Service endpoints and quotas.
      2. Optional: In the Public Cloud Object Storage Endpoint field, update the URL of your S3-compatible storage, if applicable.
      3. In the Cloud Object Storage Bucket Name field, update the name of the bucket where your pipeline artifacts are stored, if applicable. If the bucket name does not exist, it is created automatically.
      4. From the Cloud Object Storage Authentication Type list, update the authentication type required to access to your S3-compatible cloud storage, if applicable. If you use AWS S3 buckets, you must select USER_CREDENTIALS from the list.
      5. Optional: In the Cloud Object Storage Credentials Secret field, update the secret that contains the storage user name and password, if applicable. This secret is defined in the relevant user namespace. You must save the secret on the cluster that hosts your pipeline runtime.
      6. Optional: In the Cloud Object Storage Username field, update the user name to connect to your S3-compatible cloud storage, if applicable. If you use AWS S3 buckets, update your AWS Secret Access Key ID.
      7. Optional: In the Cloud Object Storage Password field, update the password to connect to your S3-compatible cloud storage, if applicable. If you use AWS S3 buckets, update your AWS Secret Access Key.
   6. Click Save & Close.

Procedure

1. In the left sidebar of JupyterLab, click Runtimes ( ).

2. Hover the cursor over the runtime configuration that you want to delete and click the Delete Item button ( ).

   A dialog box appears prompting you to confirm the deletion of your runtime configuration.

3. Click OK.

Procedure

1. In the left sidebar of JupyterLab, click Runtimes ( ).
2. Hover the cursor over the runtime configuration that you want to duplicate and click the Duplicate button ( ).

Procedure

1. In the Pipeline Editor user interface, click Run Pipeline ( ).

   The Run Pipeline dialog appears. The Pipeline Name field is automatically populated with the pipeline file name.

   Important

   You must enter a unique pipeline name. The pipeline name that you enter must not match the name of any previously executed pipelines.

2. Define the settings for your pipeline run.

   1. From the Runtime Configuration list, select the relevant runtime configuration to run your pipeline.
   2. Optional: Configure your pipeline parameters, if applicable. If your pipeline contains nodes that reference pipeline parameters, you can change the default parameter values. If a parameter is required and has no default value, you must enter a value.
3. Click OK.

Procedure

1. In the Pipeline Editor user interface, click Export Pipeline ( ).

   The Export Pipeline dialog appears. The Pipeline Name field is automatically populated with the pipeline file name.

2. Define the settings to export your pipeline.

   1. From the Runtime Configuration list, select the relevant runtime configuration to export your pipeline.
   2. From the Export Pipeline as select an appropriate file format
   3. In the Export Filename field, enter a file name for the exported pipeline.
   4. Select the Replace if file already exists check box to replace an existing file of the same name as the pipeline you are exporting.
   5. Optional: Configure your pipeline parameters, if applicable. If your pipeline contains nodes that reference pipeline parameters, you can change the default parameter values. If a parameter is required and has no default value, you must enter a value.
3. Click OK.