Chapter 4. Track experiments with MLflow in workbenches

Procedure

1. Log in to your OpenShift cluster as a cluster administrator:

   $ oc login --token=__<token>__ --server=__<openshift_cluster_url>__

2. Identify the name of the DataScienceCluster object:

   $ oc get datasciencecluster

3. Edit the DataScienceCluster object to enable the MLflow operator component:

   $ oc edit datasciencecluster __<dsc_name>__

4. In the spec.components section, add or update the mlflowoperator field and set managementState to Managed:

   spec:
     components:
       # ...
       mlflowoperator:
         managementState: Managed
       # ...

   Note

   The default managementState for the mlflowoperator component is Removed. You must explicitly set it to Managed to enable MLflow on the platform.

5. Save and close the editor.

   When you set mlflowoperator to Managed, the Red Hat OpenShift AI Operator automatically configures the notebook controller with MLFLOW_ENABLED=true and sets the appropriate GATEWAY_URL for the cluster.

6. Create an MLflow CR to deploy a tracking server instance. The MLflow CR structure and namespace requirements depend on the MLflow operator version. For more information, consult the MLflow operator documentation for the CR specification.

Verification

1. Confirm that the MLflow operator pod is running:

   $ oc get pods -n redhat-ods-applications -l app=mlflow-operator --field-selector status.phase=Running

   The output shows one or more MLflow operator pods in Running status. If the command returns no results, verify that the mlflowoperator component managementState is set to Managed and wait for the operator pods to start.

2. Confirm that the mlflowoperator component status is true in the DataScienceCluster:

   $ oc get datasciencecluster __<dsc_name>__ -o jsonpath='{.status.installedComponents.mlflowoperator}'

   The expected output is true.

Procedure

1. Log in to your OpenShift cluster:

   $ oc login --token=__<token>__ --server=__<openshift_cluster_url>__

2. Add the opendatahub.io/mlflow-instance annotation to the notebook resource.

   In the following command, replace __<notebook_name>__ with the name of your workbench notebook and __<mlflow_instance_name>__ with the name of your MLflow instance CR:

   $ oc annotate notebook -n __<namespace>__ __<notebook_name>__ opendatahub.io/mlflow-instance=__<mlflow_instance_name>__

   If the MLflow instance name is mlflow, the tracking URI path is /mlflow. If the instance name is different, the path is /mlflow-__<mlflow_instance_name>__.

   Alternatively, you can add the annotation directly in the notebook resource YAML:

   apiVersion: kubeflow.org/v1
   kind: Notebook
   metadata:
     name: __<notebook_name>__
     namespace: __<namespace>__
     annotations:
       opendatahub.io/mlflow-instance: "__<mlflow_instance_name>__"
   spec:
     # ...

3. Start or restart the workbench so that the environment variables are injected into the new pod.

Verification

1. Verify that the MLflow environment variables are present in the running workbench pod.

   First, find the pod name for your notebook:

   $ oc get pods -n __<namespace>__ -l notebook-name=__<notebook_name>__

   Then verify that the MLflow environment variables are present, where __<notebook_pod_name>__ is the pod name from the previous command and __<notebook_name>__ is the notebook resource name:

   $ oc exec -n __<namespace>__ __<notebook_pod_name>__ -c __<notebook_name>__ -- env | grep MLFLOW

   The expected output shows the following environment variables:

   MLFLOW_TRACKING_URI=https://<gateway_hostname>/mlflow
   MLFLOW_K8S_INTEGRATION=true
   MLFLOW_TRACKING_AUTH=kubernetes-namespaced

   The <gateway_hostname> value is cluster-specific and matches the GatewayAPI public endpoint for your cluster.

2. Verify that the MLflow RoleBinding exists in the namespace:

   $ oc get rolebinding -n __<namespace>__ __<notebook_name>__-mlflow

   The output shows a RoleBinding referencing the mlflow-operator-mlflow-integration ClusterRole.

Procedure

1. Open a notebook in the workbench.

2. Verify that the MLflow environment variables are available:

   import os
   print(os.environ.get("MLFLOW_TRACKING_URI"))
   print(os.environ.get("MLFLOW_K8S_INTEGRATION"))
   print(os.environ.get("MLFLOW_TRACKING_AUTH"))

   All three variables display their configured values.

3. Import the MLflow SDK and set an experiment:

   import mlflow
   
   mlflow.set_experiment("my-experiment")

   The set_experiment() call returns an Experiment object and prints the experiment ID. If you see a ConnectionError, verify that the MLFLOW_TRACKING_URI environment variable is set correctly by re-running the check in step 2.

4. Start a run and log parameters and metrics:

   with mlflow.start_run():
       mlflow.log_param("learning_rate", 0.01)
       mlflow.log_param("epochs", 10)
       mlflow.log_metric("accuracy", 0.95)
       mlflow.log_metric("loss", 0.05)

   You can log any number of parameters, metrics, and tags within a run. If start_run() raises a ConnectionError or an authentication or authorization error, verify that the MLflow tracking server is running and that the workbench RoleBinding exists.

5. View the experiment results in the MLflow UI, which is accessible from the OpenShift AI dashboard.

Procedure

1. Log in to your OpenShift cluster:

   $ oc login --token=__<token>__ --server=__<openshift_cluster_url>__

2. Verify that the workbench pod has stopped:

   $ oc get pods -n __<namespace>__ -l notebook-name=__<notebook_name>__

   The command produces no output when the pod has stopped.

3. Remove the opendatahub.io/mlflow-instance annotation from the notebook resource.

   In the following command, replace __<notebook_name>__ with the name of your workbench notebook:

   $ oc annotate notebook -n __<namespace>__ __<notebook_name>__ opendatahub.io/mlflow-instance-

   The trailing dash (-) removes the annotation.

Verification

1. Start the workbench and verify that the MLflow environment variables are no longer present.

   First, find the pod name for your notebook:

   $ oc get pods -n __<namespace>__ -l notebook-name=__<notebook_name>__

   Then verify that the MLflow environment variables are no longer present, where __<notebook_pod_name>__ is the pod name from the previous command and __<notebook_name>__ is the notebook resource name:

   $ oc exec -n __<namespace>__ __<notebook_pod_name>__ -c __<notebook_name>__ -- env | grep MLFLOW

   The command produces no output, confirming that the MLflow variables have been removed.

2. Verify that the MLflow RoleBinding resource has been removed:

   $ oc get rolebinding -n __<namespace>__ __<notebook_name>__-mlflow

   The expected output is:

   Error from server (NotFound): rolebindings.rbac.authorization.k8s.io "__<notebook_name>__-mlflow" not found

Procedure

1. Stop the workbench. You can stop the workbench from the OpenShift AI dashboard or by applying the stop annotation:

   $ oc annotate notebook -n __<namespace>__ __<notebook_name>__ kubeflow-resource-stopped=true

   The value of the kubeflow-resource-stopped annotation is not significant for this purpose. The validating webhook checks only for the presence of the annotation key.

2. Wait for the notebook pod to stop fully:

   $ oc get pods -n __<namespace>__ -l notebook-name=__<notebook_name>__

   The command produces no output when the pod has stopped.

3. Remove the MLflow annotation:

   $ oc annotate notebook -n __<namespace>__ __<notebook_name>__ opendatahub.io/mlflow-instance-