Chapter 12. Accessing monitoring APIs by using the CLI

Procedure

1. Extract an authentication token by running the following command:

   $ TOKEN=$(oc whoami -t)

2. Extract the alertmanager-main API route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route alertmanager-main -ojsonpath={.status.ingress[].host})

3. Query the service API receivers for Alertmanager by running the following command:

   $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v2/receivers"

Procedure

1. Retrieve the bearer token by running the following the command:

   $ TOKEN=$(oc whoami -t)

2. Get the Prometheus federation route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route prometheus-k8s-federate -ojsonpath={.status.ingress[].host})

3. Query metrics from the /federate route. The following example command queries up metrics:

   $ curl -G -k -H "Authorization: Bearer $TOKEN" https://$HOST/federate --data-urlencode 'match[]=up'

   Example output

   # TYPE up untyped
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.143.148:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035322214
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.148.166:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035338597
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.173.16:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035343834
   ...

Procedure

1. Extract an authentication token to connect to Prometheus by running the following command:

   $ TOKEN=$(oc whoami -t)

2. Extract the thanos-querier API route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route thanos-querier -ojsonpath={.status.ingress[].host})

3. Set the namespace to the namespace in which your service is running by using the following command:

   $ NAMESPACE=ns1

4. Query the metrics of your own services in the command line by running the following command:

   $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v1/query?" --data-urlencode "query=up{namespace='$NAMESPACE'}"

   The output shows the status for each application pod that Prometheus is scraping:

   The formatted example output

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "__name__": "up",
             "endpoint": "web",
             "instance": "10.129.0.46:8080",
             "job": "prometheus-example-app",
             "namespace": "ns1",
             "pod": "prometheus-example-app-68d47c4fb6-jztp2",
             "service": "prometheus-example-app"
           },
           "value": [
             1591881154.748,
             "1"
           ]
         }
       ],
     }
   }

   Note
   • The formatted example output uses a filtering tool, such as jq, to provide the formatted indented JSON. See the jq Manual (jq documentation) for more information about using jq.
   • The command requests an instant query endpoint of the Thanos Querier service, which evaluates selectors at one point in time.