Integrating OpenShift Container Platform data into cost management
Learn how to add and configure your OpenShift Container Platform integrations
Abstract
Preface Copy linkLink copied to clipboard!
To use cost management with OpenShift so that you can view your OpenShift on-premise and cloud costs, you must create an OpenShift Hybrid Cloud Console integration and install the Cost Management Metrics Operator. The Cost Management Metrics Operator gathers usage, capacity, and request information from OpenShift Container Platform. You can configure the Cost Management Metrics Operator integration from the OpenShift Container Platform web console or from the command line interface (CLI). You can create an OpenShift integration in the Hybrid Cloud Console, in the OpenShift Container Platform web console, or from the CLI.
There are three ways to transfer data to Red Hat:
- Direct connection (default): You have a direct Internet connection to Red Hat.
- CMMO-specific proxy: The Cost Management Metrics Operator sends data to Red Hat by connecting to the Internet through a CMMO-specific proxy.
- Restricted network mode: the Cost Management Metrics Operator never connects to the Internet. Data is generated locally and a user must push it to Red Hat.
To collect data, cost management uses Prometheus queries that you can find in the source code.
Chapter 1. Install the Cost Management Metrics Operator and create an OpenShift Container Platform integration Copy linkLink copied to clipboard!
To view your OpenShift Container Platform and cloud costs, you must install the Cost Management Metrics Operator and integrate your OpenShift cluster with the Red Hat Hybrid Cloud Console. You can do this from the OpenShift web console or from the OpenShift command line.
Perform the following tasks to install the Cost Management Metrics Operator and begin using the cost management application in OpenShift Container Platform. In both cases the high-level steps are the same:
-
Install the Cost Management Metrics Operator (
costmanagement-metrics-operator) and use the default token authentication. -
Create a CostManagementMetricsConfig YAML file that configures
costmanagement-metrics-operator. - Create a cost management OpenShift Container Platform integration with a new installation, or confirm an existing integration with a replacement installation.
If you do not use token authentication, you must take additional steps to configure the secret that holds the client_id and client_secret credentials for your service account from the Red Hat Hybrid Cloud Console.
The Cost Management Metrics Operator collects usage, capacity, and request information in bytes per second, which measures how much data is used per second. To convert from bytes per second to gibibytes per month (GiB-Mo), Red Hat uses the following equation:
bytes_per_second_reported_by_operator / (seconds_in_a_day * days_in_month) * power(2, -30)
One GiB (gibibyte) equals 2^30 bytes (1,073,741,824 bytes). The calculation uses 2^-30 to convert from bytes to GiB.
1.1. Install the cost management operator with the OpenShift web console Copy linkLink copied to clipboard!
Install the Cost Management Metrics Operator from the OpenShift Container Platform web console so you can import OpenShift usage data into cost management.
Prerequisites
- You have an OpenShift Container Platform cluster.
- You are logged in to the OpenShift Container Platform web console with an account that has cluster administrator privileges.
- You can access the Red Hat Hybrid Cloud Console with an account that has access to the cost management service. See Limiting access to cost management resources for more information.
Procedure
Log in to the OpenShift Container Platform web console and navigate the Software Catalog page.
NoteIf you are using OpenShift Container Platform 4.18 or earlier, navigate to the Operator Hub page.
- Enter Cost Management in the seach box and then select Cost Managment Metrics Operator. The Cost Management Metrics Operator install page opens.
- Click Install. The Install Operator page opens.
- Click Install at the bottom of the Install Operator page.
- Click Installed Operators in the left-hand navigation. A page with a list of installed operators opens.
- Click Cost Management Metrics Operator.
- Click Create CostManagementMetricsConfig. The configuration YAML file opens.
-
Scroll to the bottom of the YAML file and change
create_source: falsetocreate_source: true. Whencreate_sourceis set totrue, an OpenShift integration is created. Note that if you previously created an OpenShift integration in the Red Hat Hybrid Cloud Console, you should leave the value ofcreate_sourceasfalse. Change
name:''to the name of your integration, for example:name: my-openshift-cost-sourceIf you do not provide a name, the operator name defaults to the cluster ID.
- Click Create.
Next steps
- If you did not use token authentication, follow the steps in Section 1.1.1, “Configure service account authentication for the Cost Management Metrics Operator”.
- If a proxy has a custom CA certificate, you must create additional configurations to inject this certificate into the Cost Management Metrics Operator. For more details, see Injecting a custom CA certificate in the OpenShift Container Platform documentation.
1.1.1. Configure service account authentication for the Cost Management Metrics Operator Copy linkLink copied to clipboard!
If you manually configured the cost management metrics operator to use basic authentication, you must set up service account authentication. Basic authentication is no longer supported.
Prerequisites
- You are logged into the OpenShift Container Platform web console and have cluster administrator privileges.
- The Cost Management Metrics Operator is in the Installed Operators tab.
Procedure
Create a service account and get your client ID and client secret:
-
From Red Hat Hybrid Cloud Console, click
.
- Click .
- Click the tab Service Accounts.
- Click .
Enter a name and description.
The name must start with a letter and end with either a letter or a number. Use only alphanumeric characters and hyphens.
- Click .
Copy the client ID and client secret and store them in a safe location.
The secret will not be shown again after you close the window.
- Select the checkbox I have copied the client ID and secret and then click . The account that you created is added to the list.
-
From Red Hat Hybrid Cloud Console, click
Associate the new service account with a group:
- Click the tab User Access and then click Groups.
- Find your desired group in the list and click its name.
- In the window that opens, click the tab Service Accounts.
- Click .
- Select the service account that you made in the previous section, or whichever service account that you want to associate with the group.
Click .
Your service account will now inherit permissions associated with the roles in your group. To create an integration, the service account that you created for the operator needs the Cloud Administrator role. For more information, see Default user roles in cost management.
Create the secret key/value pair for service account authentication:
- In the OpenShift Container Platform web console, click the tab Workloads and then click Secrets.
- In the Secrets window, click the drop-down and then select Key/value secret. You will make two keys: one for your Client ID and one for your Client Secret.
Enter the following information in the Create key/value secret window:
-
In the Secret Name box, enter
service-account-auth-secret. -
In the Key box, enter
client_id. -
In the Value box for the first key
client_id, upload the value for your authorized Red Hat Hybrid Cloud Console user account or paste it into the text box. This is the client ID that you saved when you made your service account.
-
In the Secret Name box, enter
- Click Add Key/Value to add the second key/value pair for your client secret.
-
In the Key box, enter
client_secret: -
In the Value box for the second key
client_secret, upload the Value for your authorized Red Hat Hybrid Cloud Console user account or paste it into the text box. - After you verify that the key/value details for the secret are correct, click to complete the creation of your service account authorization secret.
- Copy the name of your secret. You will use it in the following section.
Modify the YAML file:
- Click the Operators tab and then click Installed Operators.
- Find the Cost Management Metrics Operator and click its name.
-
Click the tab Cost Management Metrics Config and then click the configuration file in Name. The default name is
costmanagementmetricscfg-sample. - Click the tab YAML to open the file.
Locate the following lines in the YAML file:
authentication: type: token-
Change
type: tokentotype: service-account. Insert a new line for
secret_name. Enter the secret that you copied in the previous section, for example:authentication: secret_name: service-account-auth-secret #Change this line to match your secret name. type: service-account- Click .
1.2. Install the cost management operator with the OpenShift CLI Copy linkLink copied to clipboard!
If you are automating cluster creation, you can install the Cost Management Metrics Operator by using the OpenShift CLI instead of the OpenShift web console. In the OpenShift Container Platform CLI, you can create an integration for your OpenShift Container Platform cluster in cost management. If you use service authentication, you must configure your Operator to use it.
Prerequisites
-
You installed the OpenShift CLI,
oc. - You have cluster administrator privileges for your OpenShift Container Platform cluster.
- You can access the Red Hat Hybrid Cloud Console with an account that has access to the cost management service. See Limiting access to cost management resources for more information.
Procedure
To verify the package manifests have the supported install modes and available channels, enter the following command:
oc describe packagemanifests costmanagement-metrics-operator -n openshift-marketplaceCreate an
OperatorGroupobject and a subscription object.- To create a subscription object, see the OpenShift documentation Installing from OperatorHub using the CLI.
If you are not using token authentication, configure your Operator to use service account authorization. To use this method, add your service account to a User Access Group that has a Cloud Administrator role. Ensure the service account inherits the permissions of the user group. For more information, see Limiting access to cost managment resources.
-
Retrieve your
client_idandclient_secretfrom the Red Hat Hybrid Cloud Console service account. Encode the value of your service account’s
client_idin base64. In your terminal, enter:echo -n "<red_hat_service_account_client_id>" | base64Encode the value of your service account’s
client_secretin base64. In your terminal, enter:echo -n "<red_hat_service_account_client_secret>" | base64
-
Retrieve your
Create a YAML file to store your secrets. Paste the
client_idandclient_secretin thedata.client_idanddata.client_secretfields.Example.yaml
kind: Secret apiVersion: v1 metadata: name: service-account-auth-secret namespace: costmanagement-metrics-operator data: client_id: <base64_encoded_red_hat_service_account_client_id> client_secret: <base64_encoded_red_hat_service_account_client_secret>Deploy your secret YAML file with the following command:
oc apply -f example.yamlTo use service authentication for the cost management Operator, edit the custom resource definition for the Operator. You must edit the custom resource example YAML so that
authentication.typeis set toservice-account. You must also add a line so thatauthentication.secret_nameis set to the name of your secret. In this earlier example, the name of the secret isservice-account-auth-secret.Custom resource example
kind: CostManagementMetricsConfig apiVersion: costmanagement-metrics-cfg.openshift.io/v1beta1 metadata: name: costmanagementmetricscfg-sample-v1beta1 namespace: costmanagement-metrics-operator spec: authentication: type: service-account secret_name: service-account-auth-secret packaging: max_reports_to_store: 30 max_size_MB: 100 prometheus_config: collect_previous_data: true context_timeout: 120 disable_metrics_collection_cost_management: false disable_metrics_collection_resource_optimization: false source: check_cycle: 1440 create_source: false name: '' upload: upload_cycle: 360 upload_toggle: trueTo create an integration automatically without using the wizard in Red Hat Hybrid Cloud Console, edit the custom resource example YAML so that
source.create_sourceis set totrueandsource.nameis set to a name. In this example, the name is set tocluster2.Source creation example
kind: CostManagementMetricsConfig apiVersion: costmanagement-metrics-cfg.openshift.io/v1beta1 metadata: name: costmanagementmetricscfg-sample-v1beta1 namespace: costmanagement-metrics-operator spec: authentication: type: service-account secret_name: service-account-auth-secret packaging: max_reports_to_store: 30 max_size_MB: 100 prometheus_config: collect_previous_data: true context_timeout: 120 disable_metrics_collection_cost_management: false disable_metrics_collection_resource_optimization: false source: check_cycle: 1440 create_source: true name: 'cluster2' upload: upload_cycle: 360 upload_toggle: true
Next steps
If you use a proxy with a custom CA certificate, you must create additional configurations to inject this certificate into Cost Management Metrics Operator. For more details, see Injecting a custom CA certificate in the OpenShift Container Platform documentation.
1.3. Troubleshoot issues with the Cost Management Metrics Operator Copy linkLink copied to clipboard!
If the Cost Management Metrics Operator is not working as expected, verify the content of the Cost Management Metrics Operator and check the memory allocation for the OpenShift cluster.
1.3.1. Verify that the Cost Management Metrics Operator YAML file is configured correctly Copy linkLink copied to clipboard!
If the Cost Management Metrics Operator is not working as expected, verify that your YAML file is configured correctly.
Prerequisites
- You have access to the OpenShift Container Platform web console.
- You can view the content in the Installed Operators tab.
Procedure
- Click the Installed Operators tab.
- In the list of installed operators, click Cost Management Metrics Operator.A metrics operator window opens.
- Click the CostManagementMetricsConfig tab to show a list of the configuration file names.
-
In the file name list, click the configuration file that you want to verify. In the default installation, the file name is
costmanagementmetricscfg-sample. A Details window opens. Click and check the following items:
prometheus_configuredandprometheus_connectedshould be set totrue:prometheus: last_query_start_time: '2021-01-25T20:59:06Z' last_query_success_time: '2021-01-25T20:59:06Z' prometheus_configured: true prometheus_connected: true service_address: 'https://thanos-querier.openshift-monitoring.svc:9091' skip_tls_verification: falseingress_path,last_successful_upload_time,last_upload_status, andlast_upload_timeshould all have content:upload: ingress_path: /api/ingress/v1/upload last_successful_upload_time: '2021-01-25T20:59:35Z' last_upload_status: 202 Accepted last_upload_time: '2021-01-25T20:59:35Z' upload: true upload_cycle: 360 upload_wait: 28 validate_cert: true
1.3.2. Adjust pod memory size Copy linkLink copied to clipboard!
If your OpenShift deployment is large and uses the default resource requests, the OpenShift pod might stop with an OOMkilled message.
Procedure
Increase the pod memory to 2GiB or more for the initial data ingestion.
After the initial data ingestion completes, pod memory can be reduced. The exact memory requirements for the pod vary based on the size of the OpenShift cluster.
Chapter 2. Integrate and use cost management with OpenShift Container Platform on a restricted network Copy linkLink copied to clipboard!
A restricted or disconnected network is a network that does not have access to the internet. If your OpenShift Container Platform cluster is deployed on a restricted network, you can configure the Operator Lifecycle Manager to manage local integrations. The following steps outline the process to integrate and use cost management on a restricted network: . Create a OpenShift Container Platform integration manually. . Configure the Operator Lifecycle Manager to install and run local integrations. . Configure the costmanagement-metrics-operator to store cost report CSV files locally using a persistent volume claim (PVC). . Download cost reports stored in the PVC to a workstation. . Upload cost reports to Red Hat Hybrid Cloud Console from your workstation.
2.1. Manually create an Openshift Container Platform integration Copy linkLink copied to clipboard!
You can automatically create your OpenShift Container Platform integration by following the steps in Installing a cost operator. However, some situations, such as restricted network installations, require that you create an OpenShift Container Platform integration manually on Red Hat Hybrid Cloud Console.
Prerequisites
- You have an OpenShift Container Platform cluster installed.
- You are logged into the OpenShift Container Platform web console and have cluster administrator privileges.
Procedure
-
From Red Hat Hybrid Cloud Console, click Settings Menu
> Integrations.
- Click the Red Hat tab.
- Click , which opens the Add a cloud integration wizard.
- In Select your integration type, click .
- In Application, click cost management. Then click .
- Enter a name in Integration name. Then click .
- In a new tab, access the OpenShift Container Platform web console. Go to → and copy your Cluster Identifier.
- Back in cost management, enter your Cluster Identifier. Then click .
- Review the details and click to create the integration.
2.2. Install the cost management operator on a restricted network Copy linkLink copied to clipboard!
Because remote integrations require full Internet connectivity, Operator Lifecycle Manager (OLM) cannot access OpenShift Container Platform clusters that are installed on restricted (disconnected) networks. You must install and configure OLM to run locally.
Prerequisites
- You installed an OpenShift Container Platform cluster.
- You have a workstation with unrestricted network access.
- You logged in to the OpenShift Container Platform web console and have cluster administrator privileges.
Procedure
Complete the following OpenShift Container Platform procedure to create a local mirror of the
costmanagement-metrics-operator: Using Operator Lifecycle Manager in disconnected environments.NoteThe
costmanagement-metrics-operatoris in theredhat-operatorscatalog in theregistry.redhat.io/redhat/redhat-operator-index:OCP_VERSIONwhereOCP_VERSIONmatches the cluster version.Prune unwanted objects from the index before you push to the mirrored registry, but do not delete the
costmanagement-metrics-operatorpackage.Log in to the OpenShift Container Platform web console and navigate the Software Catalog page.
NoteIf you are using OpenShift Container Platform 4.18 or earlier, navigate to the Operator Hub page.
- Click Cost Management Metrics Operator.
-
The Install Operator window opens. Select the
costmanagement-metrics-operatornamespace that you want to install. If the namespace does not exist, it gets created. - Click .
Verification steps
- After a short wait, Cost Management Metrics Operator appears in the Installed Operators tab in Project: all projects or Project: costmanagement-metrics-operator.
- For more details about the Operator Lifecycle Manager, see What is Operator Lifecycle Manager?
2.3. Configure the Cost Operator on a restricted network Copy linkLink copied to clipboard!
Learn how to run the costmanagement-metrics-operator on a restricted network.
Prerequisites
-
You installed
costmanagement-metrics-operator. - You logged into the OpenShift Container Platform web console and have cluster administrator privileges.
Procedure
-
From the OpenShift Container Platform web console, select Operators > Installed Operators >
costmanagement-metrics-operator> CostManagementMetricsConfig > Create Instance. Set a storage amount. If you do not specify an amount, the operator creates a default persistent volume claim (PVC) called
costmanagement-metrics-operator-datawith 10Gi of storage.NoteTo configure the
costmanagement-metrics-operatorto use a different PVC, editvolume_claim_templatein YAML view.- Select YAML view.
-
Enter a value in
max_reports_to_storeto set the maximum number of reports that you want to store. Enter a value in
upload_cycleto set how many minutes you want to pass between each report generation.packaging: max_reports_to_store: 30 max_size_MB: 100upload: upload_cycle: 360ImportantThe
costmanagement-metrics-operatorcreates one report every 360 minutes by default. The default value of 30 reports and 360 minutes gives you 7.5 days of reports.After the maximum number of reports generate, any subsequent reports replace the oldest report in storage. To avoid losing reports, download them from your PVC.
Set
upload_toggletofalse:upload: upload_cycle: 360 upload_toggle: falseSet
sourceto empty braces:source: {}Set
authenticationto empty braces:authentication: {}- Click .
Verification
- Select the CostManagementMetricsConfig that you created.
- Click .
Verify that a report was created by viewing the data in
packaging:packaging: last_successful_packaging_time: `current date and time` max_reports_to_store: 30 max_size_MB: 100 number_of_reports_stored: 1 packaged_files: - >- /tmp/costmanagement-metrics-operator-reports/upload/YYYYMMDDTHHMMSS-cost-mgmt.tar.gzNoteAfter configuration,
costmanagement-metrics-operatorgenerates an initial report. These reports are inpackaged_files.
2.4. Download cost reports Copy linkLink copied to clipboard!
If you configured the costmanagement-metrics-operator to run on a restricted network, the reports from the persistent volume claims (PVC) are temporarily stored in a workstation. Copy the reports to an unrestricted network.
The default configuration saves one week of reports. To avoid losing metrics data, download the reports locally and upload them to Red Hat Hybrid Cloud Console weekly.
You can configure any PVC, but by default, most PVCs are ReadWriteOnce. For ReadWriteOnce PVCs, the volume-shell must be attached to the same node as the operator pod.
Prerequisites
- You have a workstation with unrestricted network access.
-
costmanagement-metrics-operatorreports in your PVC.
Procedure
Create the following pod and set
claimNameto the PVC with the report data:kind: Pod apiVersion: v1 metadata: name: volume-shell namespace: costmanagement-metrics-operator spec: volumes: - name: costmanagement-metrics-operator-reports persistentVolumeClaim: claimName: costmanagement-metrics-operator-data containers: - name: volume-shell image: busybox command: ['sleep', '3600'] volumeMounts: - name: costmanagement-metrics-operator-reports mountPath: /tmp/costmanagement-metrics-operator-reportsRun
rsyncto copy all of the files from the PVC to a local folder:$ oc rsync volume-shell:/tmp/costmanagement-metrics-operator-reports/upload local/path/to/save/folder- Confirm that the files were copied.
Run the following command to connect to the pod and delete the contents of the upload folder:
$ oc rsh volume-shell $ rm /tmp/costmanagement-metrics-operator-reports/upload/*(Optional) Run the following command to delete the pod that you used to connect to the PVC:
$ oc delete -f volume-shell.yamlView your PVC usage.
In the OpenShift tab in Red Hat Hybrid Cloud Console, your PVCs with the highest usage automatically populate under Persistent Volume Claims. To view all PVCs, click at the end of the section.
You can filter your PVC data by the following fields: * Persistent volume claim * Cluster * StorageClass
2.5. Upload cost reports to the Red Hat Hybrid Cloud Console Copy linkLink copied to clipboard!
Manually upload your locally stored cost reports from a restricted network to Red Hat Hybrid Cloud Console.
The default configuration saves one week of reports. Download the reports locally and upload them to Red Hat Hybrid Cloud Console weekly to avoid losing metrics data.
Prerequisites
- You created a Bearer token.
-
You have
costmanagement-metrics-operatorreports downloaded locally. - You are a Red Hat account user with Organization Administrator entitlements.
- You have a workstation with unrestricted network access.
Procedure
To upload your reports to Red Hat Hybrid Cloud Console, set your client_id and client_secret to your credentials, set FILE_NAME to the report that you want to upload, and enter your Bearer token in $TOKEN:
$ curl -vvvv -F "file=@$FILE_NAME.tar.gz;type=application/vnd.redhat.hccm.tar+tgz" -H "Authorization: Bearer $TOKEN" https://console.redhat.com/api/ingress/v1/upload
Verification steps
- From cost management, click .
- On the OpenShift details page, confirm that you have OpenShift usage data for your cluster.
Chapter 3. Change the Cost Management Metrics Operator resource allocation Copy linkLink copied to clipboard!
The Cost Management Metrics Operator includes a default CPU and memory resource allocation. On larger clusters, the Cost Management Metrics Operator might run out of memory when it processes all of the metric data from Prometheus. If the default resource allocation does not meet your needs, you can increase the resources that are available to the operator.
The Cost Management Metrics Operator has the following default resource allocation:
Limits:
- CPU: 500m
- memory: 500Mi
Requests:
- CPU: 100m
- memory: 20Mi
Procedure
- Log in to the OpenShift Container Platform web console.
- From Installed Operators, click .
- Click the Subscription tab and then click the Actions menue and select Edit Subscription.
In the YAML file that appears, edit the deployment resources with values that meet the CPU and memory needs of your cluster:
kind: Subscription metadata: ... spec: ... config: resources: limits: cpu: 500m memory: 500Mi requests: cpu: 200m memory: 100MiThe operator is now redeployed and Deployment shows the new resources.
Additional resources
Chapter 4. Configure a proxy for the Cost Management Metrics Operator Copy linkLink copied to clipboard!
You can configure a specific proxy for the Cost Management Metrics Operator (CMMO) to allow the CMMO to connect to Red Hat without granting access to the entire cluster.
If your OpenShift cluster is on an internal network with no direct Internet access, you must typically use a proxy to reach external services. However, configuring a cluster-wide proxy grants Internet access to all OpenShift components, including operators, telemetry, and updates. To maintain a restricted environment, configure a proxy specifically for the CMMO. This ensures that only the CMMO has the connectivity required to push cost data to the Red Hat Cost Management SaaS, while the rest of the cluster remains isolated.
Procedure
- Follow the instructions in the Configuring proxy support in Operator Lifecycle Manager section of the OpenShift Container Platform documentation.
- When you are prompted to select an operator, select Cost Management Metrics Operator.