Chapter 2. Integrating filtered Google Cloud data into hybrid committed spend

Procedure

1. From Red Hat Hybrid Cloud Console, click Settings Menu > Integrations.
2. On the Settings page, in the Cloud tab, click Add integration.
3. In the Add a cloud integration wizard, select Google Cloud as the cloud provider type and click Next.
4. Enter a name for your integration. Click Next.
5. In the Select application step, select Hybrid committed spend and click Next.

Procedure

1. In the Google Cloud Console click IAM & Admin Create a Project.
2. Enter a Project name in the new page that appears and select your billing account.
3. Select the Organization.
4. Enter the parent organization in the Location box.
5. Click Create.
6. In the hybrid committed spend Add a cloud integration wizard, on the Project page, enter your Project ID.
7. To configure Google Cloud to filter your data before it sends the data to Red Hat, select I wish to manually customize the data set sent to hybrid committed spend, click Next.

Verification steps

1. Navigate to the Google Cloud Console Dashboard
2. Verify the project is in the menu bar.

Procedure

1. In the Google Cloud Console, click Buckets.
2. Click Create bucket.
3. Enter your bucket information. Name your bucket. In this example, use customer-data.
4. Click Create, then click Confirm in the confirmation dialog.
5. In the hybrid committed spend Add a cloud integration wizard, on the Create cloud storage bucket page, enter your Cloud storage bucket name.

Procedure

1. In the Google Cloud Console, click IAM & Admin Roles.
2. Select the hybrid committed spend project from the dropdown in the menu bar.
3. Click + Create role.
4. Enter a Title, Description and ID for the role. In this example, use customer-data-role.
5. Click + ADD PERMISSIONS.

6. Use the Enter property name or value field to search and select these four permissions for your custom role:

   • storage.objects.get
   • storage.objects.list
   • storage.buckets.get
7. Click ADD.
8. Click CREATE.
9. In the hybrid committed spend Add a cloud integration wizard, on the Create IAM role page, click Next.

Procedure

1. In the Google Cloud Console, click IAM & Admin IAM.
2. Select the hybrid committed spend project from the dropdown in the menu bar.
3. Click ADD.

4. Paste the IAM role you created into the New principals field:

   billing-export@red-hat-cost-management.iam.gserviceaccount.com

5. In the Assign roles section, assign the IAM role you created. In this example, use customer-data-role.
6. Click SAVE.
7. In the hybrid committed spend Add a cloud integration wizard, on the Assign access page, click Next.

Verification steps

1. Navigate to IAM & Admin IAM.
2. Verify the new member is present with the correct role.

Procedure

1. In Google Cloud Console, click Big Data BigQuery.
2. Select the hybrid committed spend project in the Explorer panel.
3. Click CREATE DATASET.
4. Enter a name for your dataset in the Dataset ID field. In this example, use CustomerFilteredData.
5. Click CREATE DATASET.

Procedure

1. In the Google Cloud Console, click Billing Billing export.
2. Click the Billing export tab.
3. Click EDIT SETTINGS in the Detailed usage cost section.
4. Select the hybrid committed spend Project and Billing export dataset you created in the dropdown menus.
5. Click SAVE.
6. In the hybrid committed spend Add a cloud integration wizard, on the Billing export page, click Next.
7. In the hybrid committed spend Add a cloud integration wizard, on the Review details page, click Add.

Verification steps

1. Verify a checkmark with Enabled in the Detailed usage cost section, with correct Project name and Dataset name.

Procedure

1. In the Google Cloud Console, search for secret and select the Secret manager result to set up a secret to authenticate your function with Red Hat without storing your credentials in your function.

   1. On the Secret Manager page, click Create Secret.
   2. Name your secret, add your Red Hat username, and click Create Secret.
   3. Repeat this process to save a secret for your Red Hat password.
2. In the Google Cloud Console search bar, search for functions and select the Cloud Functions result.
3. On the Cloud Functions page, click Create function.
4. Name the function. In this example, use customer-data-function.
5. In the Trigger section, click Save to accept the HTTP Trigger type.
6. In the Runtime, build, connections and security settings, click the Security and image repository, reference the secrets you created, click Done, and click Next.
7. On the Cloud Functions Code page, set the runtime to Python 3.9.

8. Open the requirements.txt file. Paste the following lines to the end of the file.

   requests
   google-cloud-bigquery
   google-cloud-storage

9. Open the main.py file.

   1. Set the Entry Point to get_filtered_data.

   2. Paste the following python script. Change the values in the section marked # Required vars to update to the values for your environment.

      import csv
      import datetime
      import uuid
      import os
      import requests
      from google.cloud import bigquery
      from google.cloud import storage
      from itertools import islice
      from dateutil.relativedelta import relativedelta
      
      query_range = 5
      now = datetime.datetime.now()
      delta = now - relativedelta(days=query_range)
      year = now.strftime("%Y")
      month = now.strftime("%m")
      day = now.strftime("%d")
      report_prefix=f"{year}/{month}/{day}/{uuid.uuid4()}"
      
      # Required vars to update
      USER = os.getenv('username')         # Cost management username
      PASS = os.getenv('password')         # Cost management password
      INTEGRATION_ID = "<integration_id>"  # Cost management integration_id
      BUCKET = "<bucket>"                  # Filtered data GCP Bucket
      PROJECT_ID = "<project_id>"          # Your project ID
      DATASET = "<dataset>"                # Your dataset name
      TABLE_ID = "<table_id>"              # Your table ID
      
      gcp_big_query_columns = [
          "billing_account_id",
          "service.id",
          "service.description",
          "sku.id",
          "sku.description",
          "usage_start_time",
          "usage_end_time",
          "project.id",
          "project.name",
          "project.labels",
          "project.ancestry_numbers",
          "labels",
          "system_labels",
          "location.location",
          "location.country",
          "location.region",
          "location.zone",
          "export_time",
          "cost",
          "currency",
          "currency_conversion_rate",
          "usage.amount",
          "usage.unit",
          "usage.amount_in_pricing_units",
          "usage.pricing_unit",
          "credits",
          "invoice.month",
          "cost_type",
          "resource.name",
          "resource.global_name",
      ]
      table_name = ".".join([PROJECT_ID, DATASET, TABLE_ID])
      
      BATCH_SIZE = 200000
      
      def batch(iterable, n):
          """Yields successive n-sized chunks from iterable"""
          it = iter(iterable)
          while chunk := tuple(islice(it, n)):
              yield chunk
      
      def build_query_select_statement():
          """Helper to build query select statement."""
          columns_list = gcp_big_query_columns.copy()
          columns_list = [
              f"TO_JSON_STRING({col})" if col in ("labels", "system_labels", "project.labels", "credits") else col
              for col in columns_list
          ]
          columns_list.append("DATE(_PARTITIONTIME) as partition_date")
          return ",".join(columns_list)
      
      def create_reports(query_date):
          query = f"SELECT {build_query_select_statement()} FROM {table_name} WHERE DATE(_PARTITIONTIME) = {query_date} AND sku.description LIKE '%RedHat%' OR sku.description LIKE '%Red Hat%' OR  service.description LIKE '%Red Hat%' ORDER BY usage_start_time"
          client = bigquery.Client()
          query_job = client.query(query).result()
          column_list = gcp_big_query_columns.copy()
          column_list.append("partition_date")
          daily_files = []
          storage_client = storage.Client()
          bucket = storage_client.bucket(BUCKET)
          for i, rows in enumerate(batch(query_job, BATCH_SIZE)):
              csv_file = f"{report_prefix}/{query_date}_part_{str(i)}.csv"
              daily_files.append(csv_file)
              blob = bucket.blob(csv_file)
              with blob.open(mode='w') as f:
                  writer = csv.writer(f)
                  writer.writerow(column_list)
                  writer.writerows(rows)
          return daily_files
      
      def post_data(files_list):
          # Post CSV's to console.redhat.com API
          url = "https://console.redhat.com/api/cost-management/v1/ingress/reports/"
          json_data = {"source": INTEGRATION_ID, "reports_list": files_list, "bill_year": year, "bill_month": month}
          resp = requests.post(url, json=json_data, auth=(USER, PASS))
          return resp
      
      def get_filtered_data(request):
          files_list = []
          query_dates = [delta + datetime.timedelta(days=x) for x in range(query_range)]
          for query_date in query_dates:
              files_list += create_reports(query_date.date())
          resp = post_data(files_list)
          return f'Files posted! {resp}'

10. Click Deploy.

Procedure

1. Copy the Trigger URL for the function you created to post the cost reports. You will need to add it to the Google Cloud Scheduler.

   1. In the Google Cloud Console, search for functions and select the Cloud Functions result.
   2. On the Cloud Functions page, select your function, and click the Trigger tab.
   3. In the HTTP section, click Copy to clipboard.
2. Create the scheduler job. In the Google Cloud Console, search for cloud scheduler and select the Cloud Scheduler result.

3. Click Create job.

   1. Name your scheduler job. In this example, use CustomerFilteredDataSchedule.
   2. In the Frequency field, set the cron expression for when you want the function to run. In this example, use 09*** to run the function daily at 9 AM.
   3. Set the timezone and click Continue.

4. Configure the execution on the next page.

   1. In the Target type field, select HTTP.
   2. In the URL field, paste the Trigger URL you copied.

   3. In the body field, paste the following code that passes into the function to trigger it.

      {"name": "Scheduler"}

   4. In the Auth header field, select Add OIDC token.
   5. Click the Service account field and click Create to create a service account and role for the scheduler job.

5. In the Service account details step, name your service account. In this example, use scheduler-service-account. Accept the default Service account ID and click Create and Continue.

   1. In the Grand this service account access to project, select two roles for your account.
   2. Click ADD ANOTHER ROLE then search for and select Cloud Scheduler Job Runner and Cloud Functions Invoker.
   3. Click Continue.
   4. Click Done to finish creating the service account.
6. On the Service accounts for your project page, select the scheduler job that you were working on. In this example, the name is scheduler-service-account.
7. In the Configure the execution page, select the Service account field and select the scheduler-service-account you just created.
8. Click Continue and then click Create.