Chapter 8. Troubleshooting and debugging metering
Use the following sections to help troubleshoot and debug specific issues with metering.
In addition to the information in this section, be sure to review the following topics:
8.1. Troubleshooting metering
A common issue with metering is pods failing to start. Pods might fail to start due to lack of resources or if they have a dependency on a resource that does not exist, such as a StorageClass or Secret.
8.1.1. Not enough compute resources
A common issue when installing or running metering is lack of compute resources. Ensure that metering is allocated the minimum resource requirements described in the installation prerequisites.
To determine if the issue is with resources or scheduling, follow the troubleshooting instructions included in the Kubernetes document Managing Compute Resources for Containers.
8.1.2. StorageClass not configured
Metering requires that a default StorageClass be configured for dynamic provisioning.
See the documentation on configuring metering for information on how to check if there are any StorageClasses configured for the cluster, how to set the default, and how to configure metering to use a StorageClass other than the default.
8.1.3. Secret not configured correctly
A common issue with metering is providing the incorrect secret when configuring your persistent storage. Be sure to review the example configuration files and create you secret according to the guidelines for your storage provider.
8.2. Debugging metering
Debugging metering is much easier when you interact directly with the various components. The sections below detail how you can connect and query Presto and Hive as well as view the dashboards of the Presto and HDFS components.
All of the commands in this section assume you have installed metering through OperatorHub in the openshift-metering namespace.
8.2.1. Get reporting operator logs
The command below will follow the logs of the reporting-operator.
$ oc -n openshift-metering logs -f "$(oc -n openshift-metering get pods -l app=reporting-operator -o name | cut -c 5-)" -c reporting-operator
8.2.2. Query Presto using presto-cli
The following command opens an interactive presto-cli session where you can query Presto. This session runs in the same container as Presto and launches an additional Java instance, which can create memory limits for the pod. If this occurs, you should increase the memory request and limits of the Presto pod.
By default, Presto is configured to communicate using TLS. You must to run the following command to run Presto queries:
$ oc -n openshift-metering exec -it "$(oc -n openshift-metering get pods -l app=presto,presto=coordinator -o name | cut -d/ -f2)" -- /usr/local/bin/presto-cli --server https://presto:8080 --catalog hive --schema default --user root --keystore-path /opt/presto/tls/keystore.pem
Once you run this command, a prompt appears where you can run queries. Use the show tables from metering; query to view the list of tables:
$ presto:default> show tables from metering;
Table
datasource_your_namespace_cluster_cpu_capacity_raw
datasource_your_namespace_cluster_cpu_usage_raw
datasource_your_namespace_cluster_memory_capacity_raw
datasource_your_namespace_cluster_memory_usage_raw
datasource_your_namespace_node_allocatable_cpu_cores
datasource_your_namespace_node_allocatable_memory_bytes
datasource_your_namespace_node_capacity_cpu_cores
datasource_your_namespace_node_capacity_memory_bytes
datasource_your_namespace_node_cpu_allocatable_raw
datasource_your_namespace_node_cpu_capacity_raw
datasource_your_namespace_node_memory_allocatable_raw
datasource_your_namespace_node_memory_capacity_raw
datasource_your_namespace_persistentvolumeclaim_capacity_bytes
datasource_your_namespace_persistentvolumeclaim_capacity_raw
datasource_your_namespace_persistentvolumeclaim_phase
datasource_your_namespace_persistentvolumeclaim_phase_raw
datasource_your_namespace_persistentvolumeclaim_request_bytes
datasource_your_namespace_persistentvolumeclaim_request_raw
datasource_your_namespace_persistentvolumeclaim_usage_bytes
datasource_your_namespace_persistentvolumeclaim_usage_raw
datasource_your_namespace_persistentvolumeclaim_usage_with_phase_raw
datasource_your_namespace_pod_cpu_request_raw
datasource_your_namespace_pod_cpu_usage_raw
datasource_your_namespace_pod_limit_cpu_cores
datasource_your_namespace_pod_limit_memory_bytes
datasource_your_namespace_pod_memory_request_raw
datasource_your_namespace_pod_memory_usage_raw
datasource_your_namespace_pod_persistentvolumeclaim_request_info
datasource_your_namespace_pod_request_cpu_cores
datasource_your_namespace_pod_request_memory_bytes
datasource_your_namespace_pod_usage_cpu_cores
datasource_your_namespace_pod_usage_memory_bytes
(32 rows)
Query 20190503_175727_00107_3venm, FINISHED, 1 node
Splits: 19 total, 19 done (100.00%)
0:02 [32 rows, 2.23KB] [19 rows/s, 1.37KB/s]
presto:default>8.2.3. Query Hive using beeline
The following opens an interactive beeline session where you can query Hive. This session runs in the same container as Hive and launches an additional Java instance, which can create memory limits for the pod. If this occurs, you should increase the memory request and limits of the Hive pod.
$ oc -n openshift-metering exec -it $(oc -n openshift-metering get pods -l app=hive,hive=server -o name | cut -d/ -f2) -c hiveserver2 -- beeline -u 'jdbc:hive2://127.0.0.1:10000/default;auth=noSasl'
Once you run this command, a prompt appears where you can run queries. Use the show tables; query to view the list of tables:
$ 0: jdbc:hive2://127.0.0.1:10000/default> show tables from metering; +----------------------------------------------------+ | tab_name | +----------------------------------------------------+ | datasource_your_namespace_cluster_cpu_capacity_raw | | datasource_your_namespace_cluster_cpu_usage_raw | | datasource_your_namespace_cluster_memory_capacity_raw | | datasource_your_namespace_cluster_memory_usage_raw | | datasource_your_namespace_node_allocatable_cpu_cores | | datasource_your_namespace_node_allocatable_memory_bytes | | datasource_your_namespace_node_capacity_cpu_cores | | datasource_your_namespace_node_capacity_memory_bytes | | datasource_your_namespace_node_cpu_allocatable_raw | | datasource_your_namespace_node_cpu_capacity_raw | | datasource_your_namespace_node_memory_allocatable_raw | | datasource_your_namespace_node_memory_capacity_raw | | datasource_your_namespace_persistentvolumeclaim_capacity_bytes | | datasource_your_namespace_persistentvolumeclaim_capacity_raw | | datasource_your_namespace_persistentvolumeclaim_phase | | datasource_your_namespace_persistentvolumeclaim_phase_raw | | datasource_your_namespace_persistentvolumeclaim_request_bytes | | datasource_your_namespace_persistentvolumeclaim_request_raw | | datasource_your_namespace_persistentvolumeclaim_usage_bytes | | datasource_your_namespace_persistentvolumeclaim_usage_raw | | datasource_your_namespace_persistentvolumeclaim_usage_with_phase_raw | | datasource_your_namespace_pod_cpu_request_raw | | datasource_your_namespace_pod_cpu_usage_raw | | datasource_your_namespace_pod_limit_cpu_cores | | datasource_your_namespace_pod_limit_memory_bytes | | datasource_your_namespace_pod_memory_request_raw | | datasource_your_namespace_pod_memory_usage_raw | | datasource_your_namespace_pod_persistentvolumeclaim_request_info | | datasource_your_namespace_pod_request_cpu_cores | | datasource_your_namespace_pod_request_memory_bytes | | datasource_your_namespace_pod_usage_cpu_cores | | datasource_your_namespace_pod_usage_memory_bytes | +----------------------------------------------------+ 32 rows selected (13.101 seconds) 0: jdbc:hive2://127.0.0.1:10000/default>
8.2.4. Port-forward to the Hive web UI
Run the following command:
$ oc -n openshift-metering port-forward hive-server-0 10002
You can now open http://127.0.0.1:10002 in your browser window to view the Hive web interface.
8.2.5. Port-forward to hdfs
To the namenode:
$ oc -n openshift-metering port-forward hdfs-namenode-0 9870
You can now open http://127.0.0.1:9870 in your browser window to view the HDFS web interface.
To the first datanode:
$ oc -n openshift-metering port-forward hdfs-datanode-0 9864
To check other datanodes, run the above command, replacing hdfs-datanode-0 with the pod you want to view information on.
8.2.6. Metering Ansible Operator
Metering uses the Ansible Operator to watch and reconcile resources in a cluster environment. When debugging a failed metering installation, it can be helpful to view the Ansible logs or status of your MeteringConfig custom resource.
8.2.6.1. Accessing ansible logs
In the default installation, the metering Operator is deployed as a pod. In this case, we can check the logs of the ansible container within this pod:
$ oc -n openshift-metering logs $(oc -n openshift-metering get pods -l app=metering-operator -o name | cut -d/ -f2) -c ansible
Alternatively, you can view the logs of the Operator container (replace -c ansible with -c operator) for condensed output.
8.2.6.2. Checking the MeteringConfig Status
It can be helpful to view the .status field of your MeteringConfig custom resource to debug any recent failures. The following command shows status messages with type Invalid:
$ oc -n openshift-metering get meteringconfig operator-metering -o=jsonpath='{.status.conditions[?(@.type=="Invalid")].message}'