Integrating Oracle Cloud data into cost management

Procedure

1. From Red Hat Hybrid Cloud Console, click Settings Menu > Integrations.
2. On the Settings page, click Integrations.
3. In the Cloud tab, click Add integration.
4. In the Add a cloud integration wizard, select Oracle Cloud Infrastructure as the integration type. Click Next.
5. Enter a name for your integration and click Next.
6. In the Select application step, select Cost management. Click Next.

Procedure

1. In the Add a cloud integration wizard, on the Global compartment-id step, copy the command in step one oci iam compartment list.
2. In a new tab, log in to your Oracle Cloud account.
3. In the menu bar, click Developer tools > Cloud Shell.
4. Paste the command you copied from the Add a cloud integration wizard in the Cloud Shell window.

5. In the response, copy the value pair for the compartment-id key. In the following example, the ID starts with ocid1.tenancy.oc1.

   Example response

   {
     "data": [
       {
         "compartment-id": "ocid1.tenancy.oc1..00000000000000000000000000000000000000000000",
         "defined-tags": {
           "Oracle-Tags": {
               ...
           }
         },
         ...
       }
     ]
   }

6. Return to the Global compartment-id step in the Add a cloud integration wizard, and paste your tenant-id in the Global compartment-id field.
7. Click Next.

Procedure

1. In the Add a cloud integration wizard, on the Create new policy and compartment page, copy the oci iam policy create command.
2. Paste the command that you copied into the Cloud Shell in your Oracle Cloud tab to create a cost and usage reports policy. You can also add a policy description.
3. Return to the Create new policy and compartment step in the Add a cloud integration wizard and copy the oci iam compartment create command.
4. Paste the command that you copied into the Cloud Shell in your Oracle Cloud tab to create a cost management compartment.

5. In the response, copy the value for the id key. In the following example, copy the id that includes ocid1.compartment.oc1.

   Example response

   {
     "data": [
       {
         "compartment-id": "tenant-id",
         "defined-tags": {
           "Oracle-Tags": {
               ...
           }
         },
         "description": "Cost management compartment for cost and usage data",
         "freeform-tags": {},
         "id": "ocid1.compartment.oc1..0000000000000000000000000000000000000000000",
         ...
       },
       ...
     ]
   }

6. Return to the Create new policy and compartment step in the Add a cloud integration wizard and paste the id value you copied from the response in the last step into the New compartment-id field.
7. Click Next.

Procedure

1. In the Create bucket step, create a bucket to store cost and usage data so that cost management can access it.

2. Copy the command from the previous step and paste into the Cloud Shell in your Oracle Cloud tab to create a bucket. Refer to the example response for the next steps.

   Example response

   {
     "data": {
       ...
       "name": "cost-management",
       "namespace": "cost-management-namespace",
   
       ...
       }
   }

3. Copy the value pair for the name key. In the previous example, this value is cost-management.
4. Return to the Create bucket step in the Add a cloud integration wizard. Paste the value you copied into New data bucket name.
5. Return to your Cloud Shell and copy the value for the namespace key. In the previous example, copy cost-management-namespace.
6. Return to the Create bucket step in the Add a cloud integration wizard and check your shell prompt for your region. For example, your shell prompt might be user@cloudshell:~ (uk-london-1)$. In this example, uk-london-1 is your region. Copy your region and return to the Create bucket step in the Add a cloud integration wizard.
7. In the Create bucket step in the Add a cloud integration wizard, paste your region in New bucket region.
8. Click Next.

Procedure

1. In the Oracle Cloud console, open the Navigation menu and click Developer Services > Functions.

2. Use the following Python script to create a function application:

   #
   # Copyright 2022 Red Hat Inc.
   # SPDX-License-Identifier: Apache-2.0
   #
   ##########################################################################################
   # Script to collect cost/usage reports from OCI and replicate them to another bucket
   #
   # Pre-req's you must have a service account or other for this script to gain access to oci
   #
   # NOTE! You must update the vars below for this script to work correctly
   #
   # user: ocid of user that has correct permissions for bucket objects
   # key_file: Location of auth file for defind user
   # fingerprint: Users fingerprint
   # tenancy: Tenancy for collecting/copying cost/usage reports
   # region: Home Region of your tenancy
   # bucket: Name of Bucket reports will be replicated to
   # namespace: Object Storage Namespace
   # filename: Name of json file to store last report downloaded default hre is fine
   ##########################################################################################
   import datetime
   import io
   import json
   import logging
   
   import oci
   from fdk import response
   
   
   def connect_oci_storage_client(config):
       # Connect to OCI SDK
       try:
           object_storage = oci.object_storage.ObjectStorageClient(config)
           return object_storage
       except (Exception, ValueError) as ex:
           logging.getLogger().info("Error connecting to OCI SDK CLIENT please check credentials: " + str(ex))
   
   
   def fetch_reports_file(object_storage, namespace, bucket, filename):
       # Fetch last download report file from bucket
       last_reports_file = None
       try:
           last_reports_file = object_storage.get_object(namespace, bucket, filename)
       except (Exception, ValueError) as ex:
           logging.getLogger().info("Object file does not exist, will attempt to create it: " + str(ex))
   
       if last_reports_file:
           json_acceptable_string = last_reports_file.data.text.replace("'", '"')
           try:
               last_reports = json.loads(json_acceptable_string)
           except (Exception, ValueError) as ex:
               logging.getLogger().info(
                   "Json string file not formatted correctly and cannont be parsed, creating fresh file. " + str(ex)
               )
               last_reports = {"cost": "", "usage": ""}
       else:
           last_reports = {"cost": "", "usage": ""}
   
       return last_reports
   
   
   def get_report_list(object_storage, reporting_namespace, reporting_bucket, prefix, last_file):
       # Create a list of reports
       report_list = object_storage.list_objects(
           reporting_namespace, reporting_bucket, prefix=prefix, start_after=last_file, fields="timeCreated"
       )
       logging.getLogger().info("Fetching list of cost csv files")
       return report_list
   
   
   def copy_reports_to_bucket(
       object_storage,
       report_type,
       report_list,
       bucket,
       namespace,
       region,
       reporting_namespace,
       reporting_bucket,
       last_reports,
   ):
       # Iterate through cost reports list and copy them to new bucket
       # Start from current month
       start_from = datetime.date.today().replace(day=1)
   
       if report_list.data.objects != []:
           for report in report_list.data.objects:
               if report.time_created.date() > start_from:
                   try:
                       copy_object_details = oci.object_storage.models.CopyObjectDetails(
                           destination_bucket=bucket,
                           destination_namespace=namespace,
                           destination_object_name=report.name,
                           destination_region=region,
                           source_object_name=report.name,
                       )
                       object_storage.copy_object(
                           namespace_name=reporting_namespace,
                           bucket_name=reporting_bucket,
                           copy_object_details=copy_object_details,
                       )
                   except (Exception, ValueError) as ex:
                       logging.getLogger().info(f"Failed to copy {report.name} to bucket: {bucket}. " + str(ex))
           last_reports[report_type] = report.name
       else:
           logging.getLogger().info(f"No new {report_type} reports to copy to bucket: {bucket}.")
       return last_reports
   
   
   def handler(ctx, data: io.BytesIO = None):
       name = "OCI-cost-mgmt-report-replication-function"
       try:
           body = json.loads(data.getvalue())
           name = body.get("name")
       except (Exception, ValueError) as ex:
           logging.getLogger().info("Error parsing json payload: " + str(ex))
   
       logging.getLogger().info("Inside Python OCI reporting copy function")
   
       # PLEASE CHANGE THIS!!!! #
       user = "ocid1.user.oc1..aaaaaa"  # CHANGEME
       key_file = "auth_files/service-account.pem"  # CHANGEME
       fingerprint = "00.00.00"  # CHANGEME
       tenancy = "ocid1.tenancy.oc1..aaaaaaa"  # CHANGEME
       region = "region"  # CHANGEME
       bucket = "cost-mgmt-bucket"  # CHANGEME
       namespace = "namespace"  # CHANGEME
       filename = "last_reports.json"
   
       # Get the list of reports
       # https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/clienvironmentvariables.htm!!!
       config = {
           "user": user,
           "key_file": key_file,
           "fingerprint": fingerprint,
           "tenancy": tenancy,
           "region": region,
       }
       # The Object Storage namespace used for OCI reports is bling; the bucket name is the tenancy OCID.
       reporting_namespace = "bling"
       reporting_bucket = config["tenancy"]
       region = config["region"]
   
       # Connect to OCI
       object_storage = connect_oci_storage_client(config)
   
       # Grab reports json and set previously downloaded file values
       last_reports = fetch_reports_file(object_storage, namespace, bucket, filename)
       last_cost_file = last_reports.get("cost")
       last_usage_file = last_reports.get("usage")
   
       # Get list of cost/usage files
       cost_report_list = get_report_list(
           object_storage, reporting_namespace, reporting_bucket, "reports/cost-csv", last_cost_file
       )
       usage_report_list = get_report_list(
           object_storage, reporting_namespace, reporting_bucket, "reports/usage-csv", last_usage_file
       )
   
       # Copy cost/usage files to new bucket
       last_reports = copy_reports_to_bucket(
           object_storage,
           "cost",
           cost_report_list,
           bucket,
           namespace,
           region,
           reporting_namespace,
           reporting_bucket,
           last_reports,
       )
       last_reports = copy_reports_to_bucket(
           object_storage,
           "usage",
           usage_report_list,
           bucket,
           namespace,
           region,
           reporting_namespace,
           reporting_bucket,
           last_reports,
       )
   
       # Save updated filenames to bucket object as string
       object_storage.put_object(namespace, bucket, filename, str(last_reports))
   
       return response.Response(
           ctx,
           response_data=json.dumps(
               {
                   "message": "Last reports saved from {}, Cost: {}, Usage: {}".format(
                       name, last_reports["cost"], last_reports["usage"]
                   )
               }
           ),
           headers={"Content-Type": "application/json"},
       )

3. Change the values marked # CHANGEME to the values for your environment.

   user = "ocid1.user.oc1..aaaaaa"  # CHANGEME
   key_file = "auth_files/service-account.pem"  # CHANGEME
   fingerprint = "00.00.00"  # CHANGEME
   tenancy = "ocid1.tenancy.oc1..aaaaaaa"  # CHANGEME
   region = "region"  # CHANGEME
   bucket = "cost-mgmt-bucket"  # CHANGEME
   namespace = "namespace"  # CHANGEME
   filename = "last_reports.json"

4. Create a virtual machine or a Kubernetes CronJob to trigger your function daily.

Procedure

1. In the Populate bucket step, copy the oci iam policy create command and paste into the Cloud Shell in your Oracle Cloud tab to create a read policy.
2. Click Next.
3. Review the details of the information you provided. Click Add.