Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 4. Configuring metrics for user workload monitoring

Configure the collection of metrics to monitor how cluster components and your own workloads are performing.

You can send ingested metrics to remote systems for long-term storage and add cluster ID labels to the metrics to identify the data coming from different clusters.

4.1. Configuring remote write storage
Copy link

Extend metrics retention and centralize monitoring data by sending metrics to external systems, supporting compliance requirements and long-term analytics. Doing so has no impact on how or for how long Prometheus stores metrics.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.
  • A cluster administrator has enabled monitoring for user-defined projects.
  • You have installed the OpenShift CLI (oc).

  • You have set up a remote write compatible endpoint (such as Thanos) and know the endpoint URL. See the Prometheus remote endpoints and storage documentation for information about endpoints that are compatible with the remote write feature.

    Important

    Red Hat only provides information for configuring remote write senders and does not offer guidance on configuring receiver endpoints. Customers are responsible for setting up their own endpoints that are remote-write compatible. Issues with endpoint receiver configurations are not included in Red Hat production support.

  • You have set up authentication credentials in a Secret object for the remote write endpoint. You must create the secret in the openshift-user-workload-monitoring namespace.

    Warning

    To reduce security risks, use HTTPS and authentication to send metrics to an endpoint.

Procedure

  1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project: 

    $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

  2. Add a remoteWrite: section under data/config.yaml/prometheus, as shown in the following example: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        <endpoint_authentication_credentials>

    where:

    url
    Defines the URL of the remote write endpoint.
    <endpoint_authentication_credentials>
    Specifies the authentication method and credentials for the endpoint. Currently supported authentication methods are AWS Signature Version 4, authentication using HTTP in an Authorization request header, Basic authentication, OAuth 2.0, and TLS client. See Supported remote write authentication settings for sample configurations of supported authentication methods.

  3. Add write relabel configuration values after the authentication credentials: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        <endpoint_authentication_credentials>
        writeRelabelConfigs:
        - <your_write_relabel_configs>

    where:

    <your_write_relabel_configs>
    Specifies configuration for metrics that you want to send to the remote endpoint.

    The following example forwards a single metric called my_metric: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        writeRelabelConfigs:
        - sourceLabels: [__name__]
          regex: 'my_metric'
          action: keep

    The following example forwards metrics called my_metric_1 and my_metric_2 in my_namespace namespace: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        writeRelabelConfigs:
        - sourceLabels: [__name__,namespace]
          regex: '(my_metric_1|my_metric_2);my_namespace'
          action: keep
  4. Save the file to apply the changes. The new configuration is applied automatically.

Use different methods to securely authenticate with a remote write endpoint.

The following authentication methods are supported:

  • AWS Signature Version 4
  • Basic authentication
  • Authorization
  • OAuth 2.0
  • TLS client

The following table provides details about supported authentication methods for remote write.

Expand
Authentication methodConfig map fieldDescription

AWS Signature Version 4

sigv4

This method uses AWS Signature Version 4 authentication to sign requests. You cannot use this method together with authorization, OAuth 2.0, or Basic authentication.

Basic authentication

basicAuth

Basic authentication sets an authorization header with the configured username and password on each remote write request.

authorization

authorization

Authorization sets the Authorization header on each remote write request by using the configured token.

OAuth 2.0

oauth2

An OAuth 2.0 configuration uses the client credentials grant type. Prometheus fetches an access token from tokenUrl with the specified client ID and client secret to access the remote write endpoint. You cannot use this method together with authorization, AWS Signature Version 4, or Basic authentication.

TLS client

tlsConfig

A TLS client configuration specifies the CA certificate, client certificate, and client key file used to authenticate with the remote write endpoint server using TLS.

The sample configuration requires that you have already created a CA certificate file, a client certificate file, and a client key file.

Learn about different authentication settings you can use to securely connect to a remote write endpoint.

Each sample shows how to configure a corresponding Secret object that contains authentication credentials and other relevant settings. Each sample configures authentication for use with monitoring for user-defined projects in the openshift-user-workload-monitoring namespace.

The following shows the settings for a sigv4 secret named sigv4-credentials in the openshift-user-workload-monitoring namespace. 

apiVersion: v1
kind: Secret
metadata:
  name: sigv4-credentials
  namespace: openshift-user-workload-monitoring
stringData:
  accessKey: <AWS_access_key>
  secretKey: <AWS_secret_key>
type: Opaque

where:

<AWS_access_key>
Specifies the AWS API access key.
<AWS_secret_key>
Specifies the AWS API secret key.

The following shows sample AWS Signature Version 4 remote write authentication settings that use a Secret object named sigv4-credentials in the openshift-user-workload-monitoring namespace: 

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://authorization.example.com/api/write"
        sigv4:
          region: <AWS_region>
          accessKey:
            name: sigv4-credentials
            key: accessKey
          secretKey:
            name: sigv4-credentials
            key: secretKey
          profile: <AWS_profile_name>
          roleArn: <AWS_role_arn>

where:

<AWS_region>
Specifies the AWS region.
accessKey.name, secretKey.name
Define the name of the Secret object containing the AWS API access credentials.
accessKey.key
Defines the key that contains the AWS API access key in the specified Secret object.
secretKey.key
Defines the key that contains the AWS API secret key in the specified Secret object.
<AWS_profile_name>
Specifies the name of the AWS profile that is being used to authenticate.
<AWS_role_arn>
Specifies the unique identifier for the Amazon Resource Name (ARN) assigned to your role.

4.1.2.2. Sample YAML for Basic authentication
Copy link

The following shows sample Basic authentication settings for a Secret object named rw-basic-auth in the openshift-user-workload-monitoring namespace: 

apiVersion: v1
kind: Secret
metadata:
  name: rw-basic-auth
  namespace: openshift-user-workload-monitoring
stringData:
  user: <basic_username>
  password: <basic_password>
type: Opaque

where:

<basic_username>
Specifies the username.
<basic_password>
Specifies the password.

The following sample shows a basicAuth remote write configuration that uses a Secret object named rw-basic-auth in the openshift-user-workload-monitoring namespace. It assumes that you have already set up authentication credentials for the endpoint. 

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://basicauth.example.com/api/write"
        basicAuth:
          username:
            name: rw-basic-auth
            key: user
          password:
            name: rw-basic-auth
            key: password

where:

username.name, password.name
Define the name of the Secret object that contains the authentication credentials.
username.key
Defines the key that contains the username in the specified Secret object.
password.key
Defines the key that contains the password in the specified Secret object.

The following shows bearer token settings for a Secret object named rw-bearer-auth in the openshift-user-workload-monitoring namespace: 

apiVersion: v1
kind: Secret
metadata:
  name: rw-bearer-auth
  namespace: openshift-user-workload-monitoring
stringData:
  token: <authentication_token>
type: Opaque

where:

<authentication_token>
Specifies the authentication token.

The following shows sample bearer token config map settings that use a Secret object named rw-bearer-auth in the openshift-user-workload-monitoring namespace: 

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://authorization.example.com/api/write"
        authorization:
          type: Bearer
          credentials:
            name: rw-bearer-auth
            key: token

where:

authorization.type
Defines the authentication type of the request. The default value is Bearer.
credentials.name
Defines the name of the Secret object that contains the authentication credentials.
credentials.key
Defines the key that contains the authentication token in the specified Secret object.

4.1.2.4. Sample YAML for OAuth 2.0 authentication
Copy link

The following shows sample OAuth 2.0 settings for a Secret object named oauth2-credentials in the openshift-user-workload-monitoring namespace: 

apiVersion: v1
kind: Secret
metadata:
  name: oauth2-credentials
  namespace: openshift-user-workload-monitoring
stringData:
  id: <oauth2_id>
  secret: <oauth2_secret>
type: Opaque

where:

<oauth2_id>
Specifies the Oauth 2.0 ID.
<oauth2_secret>
Specifies the OAuth 2.0 secret.

The following shows an oauth2 remote write authentication sample configuration that uses a Secret object named oauth2-credentials in the openshift-user-workload-monitoring namespace: 

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://test.example.com/api/write"
        oauth2:
          clientId:
            secret:
              name: oauth2-credentials
              key: id
          clientSecret:
            name: oauth2-credentials
            key: secret
          tokenUrl: https://example.com/oauth2/token
          scopes:
          - <scope_1>
          - <scope_2>
          endpointParams:
            param1: <parameter_1>
            param2: <parameter_2>

where:

oauth2.clientId.secret.name, oauth2.clientSecret.name
Define the name of the corresponding Secret object. Note that ClientId can alternatively refer to a ConfigMap object, although clientSecret must refer to a Secret object.
oauth2.clientId.secret.key, oauth2.clientSecret.key
Define the key that contains the OAuth 2.0 credentials in the specified Secret object.
tokenUrl
Defines the URL used to fetch a token with the specified clientId and clientSecret.
scopes
Defines the OAuth 2.0 scopes for the authorization request. These scopes limit what data the tokens can access.
endpointParams
Defines the OAuth 2.0 authorization request parameters required for the authorization server.

4.1.2.5. Sample YAML for TLS client authentication
Copy link

The following shows sample TLS client settings for a tls Secret object named mtls-bundle in the openshift-user-workload-monitoring namespace. 

apiVersion: v1
kind: Secret
metadata:
  name: mtls-bundle
  namespace: openshift-user-workload-monitoring
data:
  ca.crt: <ca_cert>
  client.crt: <client_cert>
  client.key: <client_key>
type: tls

where:

<ca_cert>
Specifies the CA certificate in the Prometheus container with which to validate the server certificate.
<client_cert>
Specifies the client certificate for authentication with the server.
<client_key>
Specifies the client key.

The following sample shows a tlsConfig remote write authentication configuration that uses a TLS Secret object named mtls-bundle. 

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        tlsConfig:
          ca:
            secret:
              name: mtls-bundle
              key: ca.crt
          cert:
            secret:
              name: mtls-bundle
              key: client.crt
          keySecret:
            name: mtls-bundle
            key: client.key

where:

ca.secret.name, cert.secret.name, keySecret.name
Define the name of the corresponding Secret object that contains the TLS authentication credentials. Note that ca and cert can alternatively refer to a ConfigMap object, though keySecret must refer to a Secret object.
ca.secret.key
Defines the key in the specified Secret object that contains the CA certificate for the endpoint.
cert.secret.key
Defines the key in the specified Secret object that contains the client certificate for the endpoint.
keySecret.key
Defines the key in the specified Secret object that contains the client key secret.

4.1.3. Example remote write queue configuration
Copy link

Optimize metrics delivery performance and reliability by tuning remote write queue parameters with the queueConfig object.

The following example shows the queue parameters with their default values for monitoring for user-defined projects in the openshift-user-workload-monitoring namespace.

Example configuration of remote write parameters with default values: 

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        <endpoint_authentication_credentials>
        queueConfig:
          capacity: 10000
          minShards: 1
          maxShards: 50
          maxSamplesPerSend: 2000
          batchSendDeadline: 5s
          minBackoff: 30ms
          maxBackoff: 5s
          retryOnRateLimit: false
          sampleAgeLimit: 0s

where:

capacity
Defines the number of samples to buffer per shard before they are dropped from the queue.
minShards
Defines the minimum number of shards.
maxShards
Defines the maximum number of shards.
maxSamplesPerSend
Defines the maximum number of samples per send.
batchSendDeadline
Defines the maximum time for a sample to wait in buffer.
minBackoff
Defines the initial time to wait before retrying a failed request. The time gets doubled for every retry up to the maxbackoff time.
maxBackoff
Defines the maximum time to wait before retrying a failed request.
retryOnRateLimit
When set to true, retries a request after receiving a 429 status code from the remote write storage.
sampleAgeLimit
Specifies that samples older than the sampleAgeLimit limit are dropped from the queue. If the value is undefined or set to 0s, the parameter is ignored.

4.1.4. Table of remote write metrics
Copy link

The following table contains remote write and remote write-adjacent metrics with further descriptions. The metrics help solve issues during remote write configuration.

Expand
MetricDescription

prometheus_remote_storage_highest_timestamp_in_seconds

Shows the newest timestamp that Prometheus stored in the write-ahead log (WAL) for any sample.

prometheus_remote_storage_queue_highest_sent_timestamp_seconds

Shows the newest timestamp that the remote write queue successfully sent.

prometheus_remote_storage_samples_retried_total

The number of samples that remote write failed to send and had to resend to remote storage. A steady high rate for this metric indicates problems with the network or remote storage endpoint.

prometheus_remote_storage_shards

Shows how many shards are currently running for each remote endpoint.

prometheus_remote_storage_shards_desired

Shows the calculated needed number of shards based on the current write throughput and the rate of incoming versus sent samples.

prometheus_remote_storage_shards_max

Shows the maximum number of shards based on the current configuration.

prometheus_remote_storage_shards_min

Shows the minimum number of shards based on the current configuration.

prometheus_tsdb_wal_segment_current

The WAL segment file that Prometheus is currently writing new data to.

prometheus_wal_watcher_current_segment

The WAL segment file that each remote write instance is currently reading from.

4.2. Creating cluster ID labels for metrics
Copy link

Create cluster ID labels for metrics to uniquely identify and track metrics across clusters and workloads by adding the write_relabel settings for remote write storage in the user-workload-monitoring-config config map in the openshift-user-workload-monitoring namespace.

Note

When Prometheus scrapes user workload targets that expose a namespace label, the system stores this label as exported_namespace. This behavior ensures that the final namespace label value is equal to the namespace of the target pod. You cannot override this default configuration by setting the value of the honorLabels field to true for PodMonitor or ServiceMonitor objects.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role, or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.
  • A cluster administrator has enabled monitoring for user-defined projects.
  • You have installed the OpenShift CLI (oc).
  • You have configured remote write storage.

Procedure

  1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project: 

    $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

  2. In the writeRelabelConfigs: section under data/config.yaml/prometheus/remoteWrite, add cluster ID relabel configuration values: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        <endpoint_authentication_credentials>
        writeRelabelConfigs:
          - <relabel_config>

    where:

    writeRelabelConfigs
    Specifies a list of write relabel configurations for metrics that you want to send to the remote endpoint.
    <relabel_config>
    Specifies the label configuration for the metrics sent to the remote write endpoint.

    The following sample shows how to forward a metric with the cluster ID label cluster_id: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        writeRelabelConfigs:
        - sourceLabels:
          - __tmp_openshift_cluster_id__
          targetLabel: cluster_id
          action: replace

    where:

    tmp_openshift_cluster_id
    Is a temporarily applied cluster ID source label. This temporary label gets replaced by the cluster ID label name that you specify.
    targetLabel
    Specifies the name of the cluster ID label for metrics sent to remote write storage. If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label. For the label name, do not use __tmp_openshift_cluster_id__. The final relabeling step removes labels that use this name.
    action
    Specifies the write relabel action. The replace write relabel action replaces the temporary label with the target label for outgoing metrics. This action is the default and is applied if no action is specified.
  3. Save the file to apply the changes. The new configuration is applied automatically.

Set up metrics collection for user-defined projects to monitor application performance and health by creating a ServiceMonitor resource that scrapes metrics from your service endpoints.

This assumes that your application uses a Prometheus client library to expose metrics to the /metrics canonical name.

You can deploy a sample service in a user-defined project and then create a ServiceMonitor resource that defines how that service should be monitored.

4.3.1. Deploying a sample service
Copy link

Deploy a sample service to verify user workload monitoring works correctly for your services and learn how to expose custom metrics.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role or as a user with administrative permissions for the namespace.

Procedure

  1. Create a YAML file for the service configuration. In this example, it is called prometheus-example-app.yaml.

  2. Add the following deployment and service configuration details to the file: 

    apiVersion: v1
kind: Namespace
metadata:
  name: ns1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus-example-app
  name: prometheus-example-app
  namespace: ns1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-example-app
  template:
    metadata:
      labels:
        app: prometheus-example-app
    spec:
      containers:
      - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
        imagePullPolicy: IfNotPresent
        name: prometheus-example-app
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus-example-app
  name: prometheus-example-app
  namespace: ns1
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
    name: web
  selector:
    app: prometheus-example-app
  type: ClusterIP

    This configuration deploys a service named prometheus-example-app in the user-defined ns1 project. This service exposes the custom version metric.

  3. Apply the configuration to the cluster: 

    $ oc apply -f prometheus-example-app.yaml

    It takes some time to deploy the service.

  4. You can check that the pod is running: 

    $ oc -n ns1 get pod

    Example output: 

    NAME                                      READY     STATUS    RESTARTS   AGE
prometheus-example-app-7857545cb7-sbgwq   1/1       Running   0          81m

4.3.2. Specifying how a service is monitored
Copy link

To monitor your custom application performance and health, configure the monitoring stack to scrape metrics from the /metrics endpoint. You can do this by using a ServiceMonitor custom resource definition (CRD) that specifies how a service should be monitored, or a PodMonitor CRD for pods.

The ServiceMonitor CRD requires a Service object, while the PodMonitor CRD does not, allowing Prometheus to directly scrape metrics from the metrics endpoint exposed by a pod.

The following example procedure shows you how to create a ServiceMonitor resource for a service in a user-defined project.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role or the monitoring-edit cluster role.
  • You have enabled monitoring for user-defined projects.

  • For this example, you have deployed the prometheus-example-app sample service in the ns1 project.

    Note

    The prometheus-example-app sample service does not support TLS authentication.

Procedure

  1. Create a new YAML configuration file named example-app-service-monitor.yaml.

  2. Add a ServiceMonitor resource to the YAML file. The following example creates a service monitor named prometheus-example-monitor to scrape metrics exposed by the prometheus-example-app service in the ns1 namespace: 

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-example-monitor
  namespace: ns1
spec:
  endpoints:
  - interval: 30s
    port: web
    scheme: http
  selector:
    matchLabels:
      app: prometheus-example-app

    where:

    namespace
    Defines a user-defined namespace where your service runs.
    port
    Defines endpoint ports to be scraped by Prometheus.
    selector
    Defines a selector to match your service based on its metadata labels.
    Note

    A ServiceMonitor resource in a user-defined namespace can only discover services in the same namespace. That is, the namespaceSelector field of the ServiceMonitor resource is always ignored.

  3. Apply the configuration to the cluster: 

    $ oc apply -f example-app-service-monitor.yaml

    It takes some time to deploy the ServiceMonitor resource.

Verification

  • Verify that the ServiceMonitor resource is running: 

    $ oc -n <namespace> get servicemonitor

    Example output: 

    NAME                         AGE
prometheus-example-monitor   81m

You can configure authentication to safely scrape metrics from service endpoints in a user-defined project by using ServiceMonitor and PodMonitor custom resource definitions (CRDs).

The following samples show different authentication settings for a ServiceMonitor resource. Each sample shows how to configure a corresponding Secret object that contains authentication credentials and other relevant settings.

The following sample shows bearer token settings for a Secret object named example-bearer-auth in the ns1 namespace:

Example bearer token secret: 

apiVersion: v1
kind: Secret
metadata:
  name: example-bearer-auth
  namespace: ns1
stringData:
  token: <authentication_token>

where:

<authentication_token>
Specifies an authentication token.

The following sample shows bearer token authentication settings for a ServiceMonitor CRD. The example uses a Secret object named example-bearer-auth:

Example bearer token authentication settings: 

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-example-monitor
  namespace: ns1
spec:
  endpoints:
  - authorization:
      credentials:
        key: token
        name: example-bearer-auth
    port: web
  selector:
    matchLabels:
      app: prometheus-example-app

where:

credentials.key
Defines the key that contains the authentication token in the specified Secret object.
credentials.name
Defines the name of the Secret object that contains the authentication credentials.
Important

Do not use bearerTokenFile to configure bearer token. If you use the bearerTokenFile configuration, the ServiceMonitor resource is rejected.

4.3.3.2. Sample YAML for Basic authentication
Copy link

The following sample shows Basic authentication settings for a Secret object named example-basic-auth in the ns1 namespace:

Example Basic authentication secret: 

apiVersion: v1
kind: Secret
metadata:
  name: example-basic-auth
  namespace: ns1
stringData:
  user: <basic_username>
  password: <basic_password>

where:

<basic_username>
Specifies a username for authentication.
<basic_password>
Specifies a password for authentication.

The following sample shows Basic authentication settings for a ServiceMonitor CRD. The example uses a Secret object named example-basic-auth:

Example Basic authentication settings: 

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-example-monitor
  namespace: ns1
spec:
  endpoints:
  - basicAuth:
      username:
        key: user
        name: example-basic-auth
      password:
        key: password
        name: example-basic-auth
    port: web
  selector:
    matchLabels:
      app: prometheus-example-app

where:

username.key
Defines the key that contains the username in the specified Secret object.
username.name, password.name
Define the name of the Secret object that contains the Basic authentication.
password.key
Defines the key that contains the password in the specified Secret object.

4.3.3.3. Sample YAML authentication with OAuth 2.0
Copy link

The following sample shows OAuth 2.0 settings for a Secret object named example-oauth2 in the ns1 namespace:

Example OAuth 2.0 secret: 

apiVersion: v1
kind: Secret
metadata:
  name: example-oauth2
  namespace: ns1
stringData:
  id: <oauth2_id>
  secret: <oauth2_secret>

where:

<oauth2_id>
Specifies an Oauth 2.0 ID.
<oauth2_secret>
Specifies an Oauth 2.0 secret.

The following sample shows OAuth 2.0 authentication settings for a ServiceMonitor CRD. The example uses a Secret object named example-oauth2:

Example OAuth 2.0 authentication settings: 

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-example-monitor
  namespace: ns1
spec:
  endpoints:
  - oauth2:
      clientId:
        secret:
          key: id
          name: example-oauth2
      clientSecret:
        key: secret
        name: example-oauth2
      tokenUrl: https://example.com/oauth2/token
    port: web
  selector:
    matchLabels:
      app: prometheus-example-app

where:

clientId.secret.key
Defines the key that contains the OAuth 2.0 ID in the specified Secret object.
clientId.secret.name, clientSecret.name
Define the name of the Secret object that contains the OAuth 2.0 credentials.
clientSecret.key
Defines the key that contains the OAuth 2.0 secret in the specified Secret object.
tokenUrl
Defines the URL used to fetch a token with the specified clientId and clientSecret.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top