Chapter 6. Troubleshoot and common questions

The following information for Red Hat OpenShift On-Demand subscriptions includes an explanation of the applicable units of measurement, a detailed scenario that shows the steps that the subscriptions service and the other Hybrid Cloud Console and monitoring stack tools use to calculate core hour usage, and additional information that can help you understand how core hour usage is reported in the subscriptions service. You can use this information to help you understand the basic principles of how the subscriptions service calculates usage for the time-based units of measurement that also use sampling.

Units of measurement for Red Hat OpenShift On-Demand subscriptions

The following table provides additional details about the derived units of measurement that are used for the Red Hat OpenShift On-Demand products. These details include the name and definition of the unit of measurement along with examples of usage that would equal one of that unit of measurement. In addition, a sample Prometheus query language (PromQL) query is provided for each unit. This example query is not the complete set of processes by which the subscriptions service calculates usage, but it is a query that you can run locally in a cluster to help you understand some of those processes.

sum_over_time((max by (_id) (cluster:usage:workload:capacity_physical_cpu_cores:min:5m))[1h:1s])

group(cluster:usage:workload:capacity_physical_cpu_cores:max:5m[1h:5m]) by (_id)

Example core hour usage calculation

The following example describes the process for calculating core hour usage for a Red Hat OpenShift On-Demand subscription. You can use this example to help you understand other derived units of measurement where time is one of the base units of the usage calculation and sampling is used as part of the measurement. For example, the vCPU hour calculation for Red Hat OpenShift AI On-Demand is done in the same way, except that the measurement is for virtual cores.

To obtain usage in core hours, the subscriptions service uses numerical integration. Numerical integration is also commonly known as an "area under the curve" calculation, where the area of a complex shape is calculated by using the area of a series of rectangles.

The tools in the Red Hat OpenShift monitoring stack contain the Prometheus query language (PromQL) function sum_over_time, a function that aggregates data for a time interval. This function is the foundation of the core hours calculation in the subscriptions service.

sum_over_time((max by (_id) (cluster:usage:workload:capacity_physical_cpu_cores:min:5m))[1h:1s])

Every 2 minutes, a cluster reports its size in cores to the monitoring stack tools, including Telemetry. One of the Hybrid Cloud Console tools, the Tally engine, reviews this information every hour in 5 minute intervals. Because the cluster reports to the monitoring stack tools every 2 minutes, each 5 minute interval might contain up to three values for cluster size. The Tally engine selects the smallest cluster size value to represent the full 5 minute interval.

The following example shows how a sample cluster size is collected every 2 minutes and how the smallest size is selected for the 5 minute interval.

Figure 6.2. Calculating the cluster size

Then, for each cluster, the Tally engine uses the selected value and creates a box of usage for each 5 minute interval. The area of the 5-minute box is 300 seconds times the height in cores. For every 5 minute box, this core seconds value is stored and eventually used to calculate the daily, account-wide aggregation of core hour usage.

The following example shows a graphical representation of how an area under the curve is calculated, with cluster size and time used to create usage boxes, and the area of each box used as building blocks to create daily core hour usage totals.

Figure 6.3. Calculating the core hours

Every day, each 5 minute usage value is added to create the total usage of a cluster on that day. Then the totals for each cluster are combined to create daily usage information for all clusters in the account. In addition, the core seconds are converted to core hours.

During the regular 24-hour update of the subscriptions service with the previous day’s data, the core hour usage information for pay-as-you-go subscriptions is updated. In the subscriptions service, the daily core hour usage for the account is plotted on the usage and utilization graph, and additional core hours used information shows the accumulated total for the account. The current instances table also lists each cluster in the account and shows the cumulative number of core hours used in that cluster.

Every month, the monthly core hour usage total for your account is supplied to Red Hat Marketplace for invoice preparation and billing. For subscription types that are offered with a four-to-one relationship of core hour to vCPU hour, the core hour total from the subscriptions service is divided by 4 for the Red Hat Marketplace billing activities. For subscription types that are offered with a one-to-one relationship of core hour to vCPU hour, no conversion in the total is made.

After the monthly total is sent to Red Hat Marketplace and the new month begins, the usage values for the subscriptions service display reset to 0 for the new current month. You can use filtering to view usage data for previous months for the span of one year.

Resolving questions about core hour usage

If you have questions about core hour usage, first use the following steps as a diagnostic tool:

From these initial troubleshooting steps, you might be able to find the cluster owner and discuss whether the unusual usage is due to an extremely high workload, problems with cluster configuration, or other issues.

If you continue to have questions after using these steps, you can contact your Red Hat account team to help you understand your core hour usage. For questions about billing, use the support instructions for Red Hat Marketplace.

October 2021: This change assumes that simultaneous multithreading on x86-64 architectures is enabled, resulting in more accurate usage data within the subscriptions service.

Across different technology vendors, the term vCPU can have different definitions. If you work with a number of different vendors, the definition that you use might not match the definition that is used by Red Hat. As a result, you might not be familiar with how Red Hat and the subscriptions service measures usage when vCPUs and simultaneous multithreading (also referred to as hyper-threading) are in use within your environment.

Some vendors offer hypervisors that do not expose to guests whether the CPUs of the guests use simultaneous multithreading. For example, recent versions of the VMware hypervisor do not show the simultaneous multithreading status to the kernel of the VM, and always report threads per core as 1. The effect of this counting method is that customers can interpret the subscriptions service reporting of Red Hat OpenShift usage data related to vCPUs to be artificially doubled.

To address customer concerns about vCPU counting, Red Hat has adjusted its assumptions related to simultaneous multithreading. Red Hat now assumes simultaneous multithreading of 2 threads per core for x86 architectures. For many hypervisors, that assumption results in an accurate counting of vCPUs per core, and customers who use those hypervisors will see no change in their Red Hat OpenShift usage data in the subscriptions service.

However, other customers who use hypervisors that do not expose simultaneous multithreading status to the kernel will see an abrupt change in subscriptions service data in October 2021. Those customers will see their related Red Hat OpenShift usage data in the subscriptions service reduced by 50% on the date that this change in counting is implemented. Past data will not be affected.

Customers who encounter this situation will not be penalized. Red Hat requires that the customer purchase enough subscriptions to cover the usage as counted in the subscriptions service only.

In the past, the discrepancies in the definitions for vCPUs have resulted in known problems with the interpretation of usage and capacity data for some subscriptions service users. This change in the assumptions for simultaneous multithreading is intended to improve the accuracy of vCPU usage data across a wider spectrum of customers, regardless of the hypervisor technology that is deployed.

If you have questions or concerns related to the usage and capacity data that is displayed in the subscriptions service, work with your Red Hat account team to help you understand your data and account status. For additional information about the resolution of this problem, you can also log in to your Red Hat account to view the issue in Additional resources.

January 2022: These changes improved capacity analysis for subscriptions that include extra entitlements or infrastructure subscriptions. These improvements resulted in a more accurate calculation of usage and capacity data for those subscriptions and a more accurate calculation of the subscription threshold within the subscriptions service for the Red Hat OpenShift portion of your Red Hat account.

December 2023 and March 2024: These changes involved analyzing the metrics used by Red Hat OpenShift Cluster Manager for internal operational purposes and determining whether these metrics were still the optimal metrics to use for subscription usage tracking purposes in the subscriptions service. The use of these metrics for two different purposes was determined to be ineffective, and increased the chances for inaccurate subscription usage reporting. Therefore, the subscriptions service changed to a service-level metric that is designed exclusively to track the subscription obligation in cores. The overall assumption for simultaneous multithreading in virtualized, x86 environments at a factor of 2 threads per core remains in effect.