Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 3. Install and authenticate the MLflow SDK

Install the MLflow SDK and configure authentication for your OpenShift cluster to track machine learning experiments in Red Hat OpenShift AI.

Prerequisites

  • You have access to a OpenShift cluster.
  • You have installed the OpenShift CLI (oc) and are able to access MLflow in your OpenShift AI project or MLflow workspace.
  • For the automated authentication method, you have a service account with the appropriate permissions or are running locally.

Procedure

  1. Install the MLflow SDK: 
pip install "mlflow[kubernetes]>=3.11"

  1. Authenticate the SDK by using one of the following methods:

    1. To use automated authentication, enable the kubernetes-namespaced client-side authentication plugin.

      Note

      This plugin reads credentials from the mounted service account token when running in a pod, or from the active kubeconfig context when running on a workstation. The MLflow workspace is automatically configured by reading either the service account’s namespace or active kubeconfig namespace.

      Enter the following commands: 

      export MLFLOW_TRACKING_URI="https://<dashboard-url>/mlflow"
export MLFLOW_TRACKING_AUTH=kubernetes-namespaced

    2. To authenticate manually, export your tracking token and project workspace as environment variables. Enter the following commands: 

      export MLFLOW_TRACKING_URI="https://<dashboard-url>/mlflow"
export MLFLOW_TRACKING_TOKEN="$(oc whoami --show-token)"
export MLFLOW_WORKSPACE="<project-name>"
      Important

      This configuration example is not recommended for production environments.

  2. If your OpenShift cluster does not use trusted TLS certificates, enter the following command to disable TLS verification: 

    export MLFLOW_TRACKING_INSECURE_TLS=true

Verification

Use the following command to verify connectivity python -c "import mlflow; print(mlflow.list_workspaces())" This command lists the workspaces you can access on the OpenShift cluster.

When you run the MLflow SDK on a local workstation, the authentication plugin uses the active kubeconfig context. The plugin uses the namespace from the kubeconfig context as the workspace and resolves the authentication token from your kubeconfig credentials, including exec-based authentication providers used by oc login command.

+ .Prerequisites

  • You have installed the MLflow SDK.
  • You have an active kubeconfig context.

Procedure

+ . Set the following environment variables to configure the tracking URI and authentication method:

+ 

export MLFLOW_TRACKING_URI="https://<dashboard-url>/mlflow"
export MLFLOW_TRACKING_AUTH=kubernetes-namespaced

  1. Optional: If you prefer to set the token and workspace manually, export the following variables: 

    export MLFLOW_TRACKING_URI="https://<dashboard-url>/mlflow"
export MLFLOW_TRACKING_TOKEN="$(oc whoami --show-token)"
export MLFLOW_WORKSPACE="<project-name>"

+ When you run a pod, the Kubernetes authentication plugin uses the mounted service account token and namespace to automatically set the workspace.

You do not have to call mlflow.set_workspace() when you enable the authentication plugin. The plugin derives the workspace from the pod’s service account namespace. You can override the workspace explicitly if you need to target a different project, provided that your service account has the necessary RBAC permissions in that project.

Prerequisites

  • You have installed the MLflow SDK.

    Procedure

    1. Set the following environment variables in your pod configuration: 

      export MLFLOW_TRACKING_URI="https://<dashboard-url>/mlflow"
export MLFLOW_TRACKING_AUTH=kubernetes-namespaced

Verification

  1. Run a Python script using the following code to confirm that the MLflow SDK successfully connects to the tracking server and logs data: 

    import mlflow

mlflow.set_experiment("demo-experiment")

with mlflow.start_run():
    mlflow.log_param("framework", "pytorch")
    mlflow.log_metric("accuracy", 0.95)

3.3. Upstream MLflow SDK reference
Copy link

Additional resources

For more information about the upstream MLflow SDK, see the following resources:

3.4. MLflow SDK troubleshooting reference
Copy link

If you encounter errors when working with experiments or artifacts in the MLflow SDK, use the following information to resolve common issues and error messages.

If your issue is not described here, contact Red Hat support.

Common issues

403 or permission denied
Problem
The active project is missing the required role-based access control (RBAC) permissions.
Resolution
Verify that your user or service account has the necessary role binding in the active project.
Workspace not found

Problem
The SDK cannot locate the workspace because the project name is incorrect, the namespace is filtered, or no workspace was selected.

Resolution
Verify that your project name is correct and that the namespace is not restricted. Ensure that you have selected a workspace in your MLflow environment settings by using one of the following methods: * The MLFLOW_TRACKING_AUTH=kubernetes-namespaced environment variable. * The MLFLOW_WORKSPACE=<workspace_name> environment variable. * The mlflow.set_workspace("team-a") Python function.

Artifact override is not applied
Problem
The MLflowConfig resource is missing, has the wrong name, or exists in the wrong project. Alternatively, the associated secret does not exist.
Resolution
Ensure that an MLflowConfig resource exists and is named mlflow. Verify that the mlflow-artifact-connection secret is present in the namespace.
Kubernetes-namespaced authentication plugin cannot resolve credentials
Problem
The authentication plugin is missing a service account token (when running in-cluster) or an active kubeconfig context (when running locally).
Resolution
If running in-cluster, ensure the service account has a valid token. If running locally, verify that your kubeconfig context is active and points to the correct cluster and project by running oc project.
Artifact writes go to the default storage location
Problem
The MLflowConfig resource does not exist in the active project, or the artifactRootSecret is invalid.
Resolution
Create the MLflowConfig resource in your active project and verify that the artifactRootSecret contains the correct connection credentials.
JSON decode error: Expecting value
Problem
The Red Hat OpenShift AI authentication token is missing or invalid. Consequently, OpenShift AI receives an HTML response from MLflow instead of the expected JSON, causing OpenShift AI to prompt for a login.
Resolution
Log in to the OpenShift cluster by running the oc login command. Ensure that your environment uses the MLFLOW_TRACKING_AUTH=kubernetes-namespaced environment variable to authenticate requests.

3.5. MLflow version compatibility
Copy link

The following information describes the compatible versions of MLflow and Red Hat OpenShift AI 3.4 GA.

Expand
Table 3.1. MLflow version compatibility and configuration
ItemDescription

Deployed MLflow server version

3.10.1

Required MLflow SDK version

3.11 or later

Authentication plugin name

kubernetes-namespaced

Environment variable

MLFLOW_TRACKING_AUTH=kubernetes-namespaced

The following command installs the compatible version of the MLflow SDK:

+ 

pip install "mlflow[kubernetes]>=3.11"

+

Note

MLflow SDK version 3.11 and later includes the kubernetes-namespaced authentication plugin by default.

3.6. MLflow storage and database compatibility
Copy link

The following table lists the MLflow storage and database configurations. The configuration settings vary according to your environment, for example production, development, or testing.

Expand
Table 3.2. Supported storage and database options
Storage areaSupported options

Artifact storage

S3 compatible object storage for production. File system for development and testing.

Database

PostgreSQL for production. SQLite for development and testing.

Artifact repository plugins

S3 and file.

3.7. Tracking experiments with MLflow SDK
Copy link

Use the MLflow software development kit (SDK) to log and track machine learning experiments. With the SDK, you can record parameters, metrics, and artifacts to a centralized tracking server for later analysis.

Prerequisites

  • You have installed the MLflow SDK version 3.11 or later.
  • You have access to a Red Hat OpenShift AI data science project that has MLflow permissions configured.

  • You have configured a tracking URI and authentication. For more information, see MLflow Kubernetes Authentication. .Procedure

    1. In your notebook, log experiments, parameters, metrics, and artifacts by using the MLflow SDK:

      When the kubernetes-namespaced authentication plugin is configured, the tracking URI and workspace are resolved automatically. 

      import random
import time
import mlflow

mlflow.set_experiment("demo-experiment")

with mlflow.start_run(run_name="demo-run"):
    mlflow.log_param("model_type", "baseline")
    mlflow.log_param("feature_count", 3)
    for step in range(5):
        mlflow.log_metric("accuracy", 0.8 + random.random() * 0.2, step=step)
        mlflow.log_metric("loss", 0.5 - random.random() * 0.2, step=step)
        time.sleep(0.2)

    2. Optional: To target a project other than the one associated with your credentials, set the workspace explicitly: 

      mlflow.set_workspace("<project-name>")

Troubleshooting

If you encounter errors when using the MLflow SDK to manage experiments or artifacts, see Troubleshooting MLflow SDK.

By default, all projects use the artifact storage configured in the MLflow resource. You can override the artifact storage for a specific project by creating an MLflowConfig resource and a connection in that project. After you create these resources, MLflow resolves the artifact root from the override for any new experiments and runs that you create in that project. However, MLflow does not serve artifacts when you configure a per-project override. The client accesses the S3 bucket directly, so the client must have valid S3 credentials.

The MLflowConfig resource is namespace-scoped and must be named mlflow. It points to an S3 compatible object storage connection that holds the credentials and bucket information for the project.

Prerequisites

  • You have installed MLflow.
  • You have an S3-compatible object storage bucket with credentials.

Procedure

  1. In your project, add a connection of type S3 compatible object storage. Set the resource name of the connection to mlflow-artifact-connection. Provide the following connection details:

    • Access key: The S3 access key ID.
    • Secret key: The S3 secret access key.
    • Endpoint: The S3 compatible endpoint URL.
    • Region: The S3 region. Optional if your storage provider does not require it.
    • Bucket: The S3 bucket name.

    If you are using the connections API instead of the dashboard, create a Secret with the annotation opendatahub.io/connection-type-protocol: "s3" and name it mlflow-artifact-connection. Setting an AWS_DEFAULT_REGION is optional. The required keys are:

    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY
    • AWS_S3_BUCKET
    • AWS_S3_ENDPOINT

  2. Create an MLflowConfig resource named mlflow in the same project: 

    apiVersion: mlflow.kubeflow.org/v1
kind: MLflowConfig
metadata:
  name: mlflow
spec:
  artifactRootSecret: mlflow-artifact-connection
  artifactRootPath: mlflow-artifacts

    artifactRootSecret must be mlflow-artifact-connection. The Custom Resource Definition (CRD) enforces this validation.

    Note

    artifactRootPath is an optional relative path that the system appends to the bucket root from the secret. For example, if the bucket is ds-team-bucket and artifactRootPath is mlflow-artifacts, the resolved artifact root becomes s3://ds-team-bucket/mlflow-artifacts. The path must be relative, must not use backslashes, and must not contain path traversal such as ...

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top