Chapter 5. Implementing pipelines

Procedure

1. Open your workbench’s JupyterLab environment. If the launcher is not visible, click + to open it.

   

2. Click Pipeline Editor.

   

   You have created a blank pipeline.

3. Set the default runtime image for when you run your notebook or Python code.

   1. In the pipeline editor, click Open Panel.

      

   2. Select the Pipeline Properties tab.

      

   3. In the Pipeline Properties panel, scroll down to Generic Node Defaults and Runtime Image. Set the value to Runtime | Tensorflow | Cuda | Python 3.12.

      
4. Select File Save Pipeline.

Procedure

1. From the JupyterLab file-browser panel, drag the 1_experiment_train.ipynb and 2_save_model.ipynb notebooks onto the pipeline canvas.

   

2. Click the output port of 1_experiment_train.ipynb and drag a connecting line to the input port of 2_save_model.ipynb.

   
3. Save the pipeline.

Procedure

1. Click the 1_experiment_train.ipynb node.

   
2. In the Properties panel, click the Node Properties tab.

3. Scroll down to the File Dependencies section and then click Add.

   
4. Set the value to data/*.csv which has the data to train your model.

5. Select the Include Subdirectories option.

   
6. Save the pipeline.

Procedure

1. Select node 1.
2. Select the Node Properties tab.
3. Scroll down to the Output Files section, and then click Add.

4. Set the value to models/fraud/1/model.onnx.

   
5. Repeat steps 2-4 for node 2.
6. Click Save Pipeline.

Procedure

1. Remove any pre-filled environment variables.

   1. Select node 2, and then select the Node Properties tab.

      Under Additional Properties, note that some environment variables have been pre-filled. The pipeline editor inferred that you need them from the notebook code.

      Because you do not want to save the value in your pipelines, remove all of these environment variables.

   2. Click Remove for each of the pre-filled environment variables.

      

2. Add the S3 bucket and keys by using the Kubernetes secret.

   1. Under Kubernetes Secrets, click Add.

      

   2. Enter the following values and then click Add.

      • Environment Variable: AWS_ACCESS_KEY_ID

        • Secret Name: my-storage

        • Secret Key: AWS_ACCESS_KEY_ID

          

3. Repeat Step 2 for each of the following Kubernetes secrets:

   • Environment Variable: AWS_SECRET_ACCESS_KEY

     • Secret Name: my-storage
     • Secret Key: AWS_SECRET_ACCESS_KEY

   • Environment Variable: AWS_S3_ENDPOINT

     • Secret Name: my-storage
     • Secret Key: AWS_S3_ENDPOINT

   • Environment Variable: AWS_DEFAULT_REGION

     • Secret Name: my-storage
     • Secret Key: AWS_DEFAULT_REGION

   • Environment Variable: AWS_S3_BUCKET

     • Secret Name: my-storage
     • Secret Key: AWS_S3_BUCKET
4. Select File Save Pipeline As to save and rename the pipeline. For example, rename it to My Train Save.pipeline.

Procedure

1. In the JupyterLab pipeline editor toolbar, click Run Pipeline.

   
2. Enter a name for your pipeline.
3. Verify that the Runtime Configuration: is set to Pipeline.

4. Click OK.

   NOTE: If you see an error message stating that "no runtime configuration for Pipelines is defined", you might have created your workbench before the pipeline server was available. To address this error, you must verify that you configured the pipeline server and then restart the workbench.

5. In the OpenShift AI dashboard, open your project and expand the newly created pipeline.

   

6. Click View runs.

   

7. Click your run and then view the pipeline run in progress.