Integrating Google Cloud data into cost management

Procedure

1. From Red Hat Hybrid Cloud Console, click Settings Menu > Integrations.
2. On the Settings page, in the Cloud tab, click Add integration.
3. In the Add a cloud integration wizard, select Google Cloud as the cloud provider type and click Next.
4. Enter a name for your integration. Click Next.
5. In the Select application step, select Cost management and click Next.

Procedure

1. In the Google Cloud Console click IAM & Admin → Create a Project.
2. Enter a Project name in the new page that appears and select your billing account.
3. Select the Organization.
4. Enter the parent organization in the Location box.
5. Click Create.

In cost management:

6. On the Project page, enter your Project ID.
7. Select I am OK with sending the default data set to cost management.
8. Click Next.

Procedure

1. In the Google Cloud Console, click IAM & Admin → Roles.
2. Select the project you created from the menu.
3. Click + Create role.
4. Enter a Title, Description and ID for the role. In this example, use customer-data-role.
5. Click + ADD PERMISSIONS.

6. Use the Enter property name or value field to search and select the following permissions for your custom role:

   • bigquery.jobs.create
   • bigquery.tables.getData
   • bigquery.tables.get
   • bigquery.tables.list
7. Click ADD.
8. Click CREATE.
9. In the Add a cloud integration wizard, on the Create IAM role page, click Next.

In the Google Cloud Console:

1. Click IAM & Admin → IAM.
2. Select the project you created from the menu.
3. Click Grant Access.

4. Paste the following principal into the New principals field:

   billing-export@red-hat-cost-management.iam.gserviceaccount.com

5. In the Assign roles section, assign the IAM role you created in Creating a Google Cloud Identity and Access Management role . In this example, use customer-data-role.
6. Click SAVE.

In the cost management:

1. On the Assign access page, click Next.

Verification steps

1. Navigate to IAM & Admin → IAM.
2. Verify the new member is present with the correct role.

Procedure

1. In Google Cloud Console, click BigQuery.
2. In the Explorer panel, select the project you created.
3. Click the action icon for your project name.
4. Click CREATE DATASET.
5. Enter a name for your dataset in the Dataset ID field. In this example, use CustomerData.
6. Click CREATE DATASET.
7. In the Add a cloud integration wizard, on the Create dataset page, enter the name of the dataset you created.
8. Click Next.

Procedure

1. In the Google Cloud Console, click Billing → Billing export.
2. Click the Billing export tab.
3. Click EDIT SETTINGS in the Detailed usage cost section.
4. Select the cost management Project and Billing export dataset you created in the dropdown menus.
5. Click SAVE.
6. In the Add a cloud integration wizard, on the Billing export page, click Next.
7. On the Review details page, review the information about your integration and click Add.

Verification steps

1. Verify a checkmark with Enabled in the Detailed usage cost section, with correct Project name and Dataset name.

Procedure

1. From Red Hat Hybrid Cloud Console, click Settings Menu > Integrations.
2. On the Settings page, in the Cloud tab, click Add integration.
3. In the Add a cloud integration wizard, select Google Cloud as the cloud provider type and click Next.
4. Enter a name for your integration. Click Next.
5. In the Select application step, select Cost management and click Next.

Procedure

1. In the Google Cloud Console click IAM & Admin → Create a Project.
2. Enter a Project name in the new page that appears and select your billing account.
3. Select the Organization.
4. Enter the parent organization in the Location box.
5. Click Create.

In cost management:

6. Pn the Project page, enter your Project ID.
7. To configure Google Cloud to filter your data before it sends the data to Red Hat, select I wish to manually customize the data set sent to cost management.
8. Click Next.

In the Google Cloud Console:

1. Go to Cloud Storage → Buckets.
2. Click Create.
3. Enter your bucket information. Name your bucket. In this example, use customer-data.
4. Click Create, then click Confirm in the confirmation dialog.

In cost management:

5. On the Create cloud storage bucket page, enter your Cloud storage bucket name.

Procedure

1. In the Google Cloud Console, click IAM & Admin → Roles.
2. Select the project you created from the menu.
3. Click + Create role.
4. Enter a Title, Description and ID for the role. In this example, use customer-data-role.
5. Click + ADD PERMISSIONS.

6. Use the Enter property name or value field to search and select the following permissions for your custom role:

   • storage.objects.get
   • storage.objects.list
   • storage.buckets.get
7. Click ADD.
8. Click CREATE.
9. In the Add a cloud integration wizard, on the Create IAM role page, click Next.

In the Google Cloud Console:

1. Click IAM & Admin → IAM.
2. Select the project you created from the menu.
3. Click Grant Access.

4. Paste the following principal into the New principals field:

   billing-export@red-hat-cost-management.iam.gserviceaccount.com

5. In the Assign roles section, assign the IAM role you created in Creating a Google Cloud Identity and Access Management role . In this example, use customer-data-role.
6. Click SAVE.

In the cost management:

1. On the Assign access page, click Next.

Verification steps

1. Navigate to IAM & Admin → IAM.
2. Verify the new member is present with the correct role.

Procedure

1. In Google Cloud Console, click BigQuery.
2. In the Explorer panel, select the project you created.
3. Click the action icon for your project name.
4. Click CREATE DATASET.
5. Enter a name for your dataset in the Dataset ID field. In this example, use CustomerFilteredData.
6. Click CREATE DATASET.
7. In the Add a cloud integration wizard, on the Create dataset page, enter the name of the dataset you created.
8. Click Next.

Procedure

1. In the Google Cloud Console, click Billing → Billing export.
2. Click the Billing export tab.
3. Click EDIT SETTINGS in the Detailed usage cost section.
4. Select the cost management Project and Billing export dataset you created in the dropdown menus.
5. Click SAVE.
6. In the Add a cloud integration wizard, on the Billing export page, click Next.
7. On the Review details page, review the information about your integration and click Add.
8. Copy your source_uuid so that you can use it in the cloud function.

Verification steps

1. Verify a checkmark with Enabled in the Detailed usage cost section, with correct Project name and Dataset name.

In the Google Cloud Console:

1. Click Security → Secret manager to set up a secret to authenticate your function with Red Hat without storing your credentials in your function. Enable the Secret Manager if it is not already enabled.

2. From Secret Manager, click Create secret.

   1. Name your secret, add your service account Client ID, and click Create Secret.
   2. Repeat this process to save a secret for your service account Client secret.
3. In the Google Cloud Console search bar, search for functions and select the Cloud Functions result.
4. On the Cloud Functions page, click Create function.
5. Name the function. In this example, use customer-data-function.
6. In the Trigger section, select HTTPS as the trigger type.

7. In Runtime, build, connections and security settings, click the Security and image repo tab.

   1. Click Add a secret reference.
   2. Select the client_id secret you created before.
   3. Set the reference method to Exposed as environment variable.
   4. Name the exposed environment variable client_id.
   5. Click Done.
8. Repeat the previous steps for your client_secret.
9. Click Next.
10. On the Cloud Functions Code page, set the runtime to the latest Python version available.

11. Open the requirements.txt file. Paste the following lines at the end of the file.

    requests
    google-cloud-bigquery
    google-cloud-storage

12. Set the Entry Point to get_filtered_data.

13. Open the main.py file.

    1. Paste the following python script. Change the values in the section marked # Required vars to update to the values for your environment. Update the values for the following lines:

       INTEGRATION_ID

       Cost management integration_id

       BUCKET

       Filtered data GCP Bucket

       PROJECT_ID

       Your project ID

       DATASET

       Your dataset name

       TABLE_ID

       Your table ID

14. Click Deploy.

Procedure

1. Copy the Trigger URL for the function you created to post the cost reports. You will need to add it to the Google Cloud Scheduler.

   1. In the Google Cloud Console, search for functions and select the Cloud Functions result.
   2. On the Cloud Functions page, select your function, and click the Trigger tab.
   3. In the HTTP section, click Copy to clipboard.
2. Create the scheduler job. In the Google Cloud Console, search for cloud scheduler and select the Cloud Scheduler result.

3. Click Create job.

   1. Name your scheduler job. In this example, use CustomerFilteredDataSchedule.
   2. In the Frequency field, set the cron expression for when you want the function to run. In this example, use 0 9 * * * to run the function daily at 9 AM.
   3. Set the time zone and click Continue.

4. Configure the execution on the next page.

   1. In the Target type field, select HTTP.
   2. In the URL field, paste the Trigger URL you copied.

   3. In the body field, paste the following code that passes into the function to trigger it.

      {"name": "Scheduler"}

   4. In the Auth header field, select Add OIDC token.
   5. Click the Service account field and click Create to create a service account and role for the scheduler job.

5. In the Service account details step, name your service account. In this example, use scheduler-service-account. Accept the default Service account ID and click Create and Continue.

   1. In the Grant this service account access to project field, search for and select Cloud Scheduler Job Runner as the first role.
   2. Click ADD ANOTHER ROLE, then search for and select Cloud Functions Invoker.
   3. Click Continue.
   4. Click Done to finish creating the service account.
6. Go back to the Cloud scheduler tab.
7. In the Configure the execution page, select the Service account field.
8. Refresh the page and select the scheduler you just created.
9. Click Continue and then click Create.

Procedure

1. Set up a function to post reports:

   1. From Cloud Functions, select Create function.
   2. Name your function.
   3. Select HTTP trigger.

2. In Runtime, build, connections, security settings, click Security.

   1. Click Reference secret.
   2. Select Exposed as environment variable.
   3. Select Secret version or Latest.
   4. Click Done.
   5. Repeat the process for your other secrets.
3. Click Save.
4. Copy your Trigger URL. Click Next.
5. Select the latest Python runtime.
6. Set Entry point to get_filtered_data.
7. Add your Google Cloud function. Update the values for INTEGRATION_ID, BUCKET, PROJECT_ID, DATASET, and TABLE_ID.

8. Remove the comments from the following lines:

    # month_end = now.replace(day=1) - timedelta(days=1) # delta = now.replace(day=1) - timedelta(days=query_range) # year = month_end.strftime("%Y") # month = month_end.strftime("%m") # day = month_end.strftime("%d")

9. Select the requirements.py file and add the requirements from the requirements.txt file.
10. Click Deploy.

11. Set up a cloud scheduler to trigger your function:

    1. Go to Cloud Scheduler.
    2. Click Schedule a job.
    3. Name your schedule
    4. Set the frequency. For example, the following cron will run the job on the fourth day of every month, 0 9 4 * *
    5. Set a Time zone.
    6. Click Continue.
    7. Paste the function Trigger URL you copied earlier.
    8. In the request body, add {"name": "Scheduler"}.
    9. Set the auth header to OIDC token.
    10. Select or create a service account with the Cloud Scheudler Job Runner and Cloud Functions Invoker roles.
    11. Click Continue.
    12. Click Save.