SupportConsoleDevelopersStart a trialContact

Chapter 9. Viewing logs and audit records

As a cluster administrator, you can use the OpenShift AI Operator logger to monitor and troubleshoot issues. You can also use OpenShift audit records to review a history of changes made to the OpenShift AI Operator configuration.

9.1. Configure the OpenShift AI Operator logger
Copy link

You can change the log level for the OpenShift AI Operator by setting the .spec.devFlags.logLevel flag for the DSC Initialization (DSCI) custom resource during runtime. If you do not set a logLevel value, the logger uses the info log level by default.

The log level that you set with .spec.devFlags.logLevel applies only to the rhods-operator itself, not to the individual OpenShift AI components that the Operator manages.

The following table describes the available log levels:

Expand
Table 9.1. Available log levels for the OpenShift AI Operator
Log levelSeverityVerbosityOutput formatDescription

error

N/A

Low

JSON

Restricts logging output to error messages only.

info

0

Medium

JSON

Enables standard informational logs and error. Default when not set.

debug

-1

High

JSON

Enables all logs: debug, info, and error.

Custom numeric

1 or higher

Very High

JSON

Fine-grained controller execution logging for advanced troubleshooting. Higher integers yield progressively more detailed tracing output.

Prerequisites

  • You have administrator access to the DSCInitialization resources in the OpenShift cluster.

  • You have installed the OpenShift CLI (oc) as described in the appropriate documentation for your cluster:

Procedure

  1. Log in to the OpenShift as a cluster administrator.

  2. Go to the Installed Operators page. The navigation path depends on your OpenShift version:

    • On OpenShift 4.20 and later, click Ecosystem Installed Operators.
    • On OpenShift 4.19, click Operators Installed Operators.
  3. Click the DSC Initialization tab.
  4. Click the default-dsci object.
  5. Click the YAML tab.

  6. In the spec section, update the .spec.devFlags.logLevel flag with the log level that you want to set. 

    apiVersion: dscinitialization.opendatahub.io/v2
kind: DSCInitialization
metadata:
  name: default-dsci
spec:
  devFlags:
    logLevel: debug
  7. Click Save.

  8. Alternatively, to configure the log level from the OpenShift CLI (oc), run the following command: 

    $ oc patch dsci default-dsci -p {"spec":{"devFlags":{"logLevel":"debug"}}} --type=merge

Verification

  • If you set the log level to error, logs generate infrequently and only capture critical execution failures and errors.
  • If you set the log level to info (or leave it unset), logs generate at a standard frequency and include info and error messages.
  • If you set the log level to debug, logs generate often and include all debug, info, and error messages.
  • If you set the log level to a numeric value (such as 1, 2, or 3), logs generate at an extremely high frequency, exposing deep, fine-grained controller execution data for advanced troubleshooting. Higher integers yield progressively more detailed tracing output.

9.1.1. View the OpenShift AI Operator logs
Copy link

  1. Log in to the OpenShift CLI (oc).

  2. Run the following command to stream logs from all Operator pods: 

    for pod in $(oc get pods -l name=rhods-operator -n redhat-ods-operator -o name); do oc logs -f "$pod" -n redhat-ods-operator & done

    The Operator pod logs open in your terminal.

    Tip

    Press Ctrl+C to stop viewing. To fully stop all log streams, run kill $(jobs -p).

You can also view each Operator pod log in the OpenShift console by navigating to Workloads Pods, selecting the redhat-ods-operator project, clicking a pod name, and then clicking the Logs tab.

9.2. Viewing audit records
Copy link

Cluster administrators can use OpenShift auditing to see changes made to the OpenShift AI Operator configuration by reviewing modifications to the DataScienceCluster (DSC) and DSCInitialization (DSCI) custom resources. Audit logging is enabled by default in standard OpenShift cluster configurations. For more information, see Viewing audit logs in the OpenShift documentation.

Note

In Red Hat OpenShift Service on AWS, audit logging is disabled by default because the Elasticsearch log store does not provide secure storage for audit logs. To configure log forwarding, see Logging in the Red Hat OpenShift Service on AWS documentation.

The following example shows how to use the OpenShift audit logs to see the history of changes made (by users) to the DSC and DSCI custom resources.

Prerequisites

  • You have cluster administrator privileges for your OpenShift cluster.

  • You have installed the OpenShift CLI (oc) as described in the appropriate documentation for your cluster:

Procedure

  1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example: 

    $ oc login <openshift_cluster_url> -u <admin_username> -p <password>
  2. To access the full content of the changed custom resources, set the OpenShift audit log policy to WriteRequestBodies or a more comprehensive profile. For more information, see Configuring the audit log policy.

  3. Fetch the audit log files that are available for the relevant control plane nodes. For example: 

    oc adm node-logs --role=master --path=kube-apiserver/ \
  | awk '{ print $1 }' | sort -u \
  | while read node ; do
      oc adm node-logs $node --path=kube-apiserver/audit.log < /dev/null
    done \
  | grep opendatahub > /tmp/kube-apiserver-audit-opendatahub.log

  4. Search the files for the DSC and DSCI custom resources. For example: 

    jq 'select((.objectRef.apiGroup == "dscinitialization.opendatahub.io"
                or .objectRef.apiGroup == "datasciencecluster.opendatahub.io")
              and .user.username != "system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager"
              and .verb != "get" and .verb != "watch" and .verb != "list")' < /tmp/kube-apiserver-audit-opendatahub.log

Verification

  • The commands return relevant log entries.
Tip

To configure the log retention time, see the Logging section in the OpenShift documentation.

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top