Integrating Amazon Web Services (AWS) data into cost management

Procedure

1. From Red Hat Hybrid Cloud Console, click Settings Menu > Integrations.
2. On the Settings page, in the Cloud tab, click Add integration.
3. On the Select integration type step, in the Add a cloud integration wizard, select Amazon Web Services. Click Next.
4. Enter a name for the integration and click Next.

5. On the Select configuration step, select how you want to connect to your AWS integration.

   • Select Account authorization to provide your AWS account credentials and let Red Hat configure and manage your integration for you. Click Next.
   • Select Manual configuration to customize your integration. If you are using cost management to meter your RHEL subscription, you must select Manual Configuration.
6. In the Select application step, select Cost management. Click Next.
7. If you selected the account authorization method, on the Review details step, review the details and click Add. If you selected the manual configuration method, continue to the next step in the wizard and configure your S3 bucket.

1. In the AWS S3 console, create a new S3 bucket or use an existing bucket. If you are configuring a new S3 bucket, accept the default settings.
2. On the Create storage step, in the Add a cloud source wizard, paste the name of your S3 bucket and select the region that it was created in. Click Next.

3. In the AWS Billing console, create a data export that will be delivered to your S3 bucket. Enter the following values and accept the defaults for any other values:

   • Export type: Legacy CUR export
   • Report name: koku
   • Include: resource IDs
   • Time unit: Hourly
   • Enable report data integration for: Amazon Redshift, Amazon QuickSight. Disable report data integration for Amazon Athena
   • Compression type: GZIP
   • S3 bucket: <the S3 bucket that you configured before>

   • Report path prefix: cost

     Note

     For more details on configuration, see the AWS Billing and Cost Management documentation.

4. In the Add a cloud integration wizard, on the Create cost and usage report step, click Next.

Procedure

1. In the AWS Billing console:

   1. Open the Cost Allocation Tags section.
   2. Select the tags you want to use in the cost management application, and click Activate.

2. If your organization is converting systems from CentOS 7 to RHEL and using hourly billing, activate the com_redhat_rhel tag for your systems in the Cost Allocation Tags section of the AWS console.

   1. After tagging the instances of RHEL you want to meter in AWS, select Include RHEL usage.
3. In the Red Hat Hybrid Cloud Console Integrations wizard, select Include RHEL usage.

In cost management:

1. In the Add a cloud integration wizard, select the additional data points you want to be included.
2. Click Next.
3. Copy the JSON output that is generated based on your selections.

In the AWS Identity and Access Management console:

3. From the AWS Identity and Access Management (IAM) console, create a new IAM policy for the S3 bucket that you configured before.

   1. Select the JSON tab and paste the JSON policy which you copied from the Red Hat Hybrid Cloud Console Add a cloud integration wizard:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
              "s3:Get*",
              "s3:List*"
            ],
              "Resource": [
              "arn:aws:s3:::<your_bucket_name>", 1
              "arn:aws:s3:::<your_bucket_name>/*"
            ]
          },
      
          {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
              "s3:ListBucket",
              "cur:DescribeReportDefinitions"
            ],
            "Resource": "*"
          }
        ]
      }

   2. Enter a name for the policy and create the policy. Do not close the AWS IAM console. You will use it in the following steps.

In cost management:

4. In the Red Hat Hybrid Cloud Console Add a cloud integration wizard, click Next.

In the AWS Identity and Access Management console:

5. In the AWS IAM console, create a new IAM role:

   1. Select Another AWS account as the type of trusted entity.
   2. Enter 589173575009 as the Account ID to give Red Hat Hybrid Cloud Console read access to the AWS account cost data.

In cost management:

1. Copy your external ID from the Create IAM role step in the wizard.

In the AWS Identity and Access Management console:

1. Enter your external ID in the External ID field.
2. Attach the IAM policy you just configured.
3. Enter a role name and description.

In cost management:

6. In the Red Hat Hybrid Cloud Console Add a cloud integration wizard, click Next.

In the AWS Identity and Access Management console:

7. In the AWS IAM console, in the Roles section, open the summary screen for the role you just created.

   1. Copy the Role ARN, which is a string beginning with arn:aws:.

In cost management:

8. In the Red Hat Hybrid Cloud Console Add a cloud integration wizard, paste your Role ARN and click Next.
9. Review the details of your cloud integration and click Add.

Procedure

1. From Red Hat Hybrid Cloud Console, click Settings Menu > Integrations.
2. On the Settings page, in the Cloud tab, click Add integration.
3. On the Select integration type step, in the Add a cloud integration wizard, select Amazon Web Services. Click Next.
4. Enter a name for the integration and click Next.

5. On the Select configuration step, select how you want to connect to your AWS integration.

   • Select Manual configuration to customize your integration. If you are using cost management to meter your RHEL subscription, you must select Manual Configuration. Click Next.
6. In the Select application step, select Cost management. Click Next.

Procedure

1. Log in to your AWS account.

2. In the AWS Billing console, create a data export that will be delivered to your S3 bucket. Specify the following values and accept the defaults for any other values:

   • Export type: Legacy CUR export
   • Report name: <rh_cost_report> (note this name as you will use it later)
   • Additional report details: Include resource IDs
   • S3 bucket: Select an S3 bucket you configured previously or create a bucket and accept the default settings.
   • Time granularity: Hourly
   • Enable report data integration for: Amazon Athena, which is required for lambda queries
   • Compression type: Parquet

   • Report path prefix: cost

     Note

     For more details on configuration, see the AWS Billing and Cost Management documentation.

In your AWS account:

1. Log in to your AWS account.
2. From Configure S3 Bucket, click Configure. Create a bucket and apply the default policy.
3. Click Save.

In cost management:

4. On the Create storage step, paste the name of your S3 bucket and select the region that it was created in and click Next.
5. On the Create cost and usage report step in the Add a cloud integration wizard, select I wish to manually customize the CUR sent to Cost Management.
6. Click Next.

Procedure

1. In the AWS Billing console:

   1. Open the Cost Allocation Tags section.
   2. Select the tags you want to use in the cost management application, and click Activate.

2. If your organization is converting systems from CentOS 7 to RHEL and using hourly billing, activate the com_redhat_rhel tag for your systems in the Cost Allocation Tags section of the AWS console.

   1. After tagging the instances of RHEL you want to meter in AWS, select Include RHEL usage.
3. In the Red Hat Hybrid Cloud Console Integrations wizard, select Include RHEL usage.

In cost management:

1. In the Add a cloud integration wizard, select the additional data points you want to be included.
2. Click Next.
3. Copy the JSON output that is generated based on your selections.

In the AWS Identity and Access Management console:

3. From the AWS Identity and Access Management (IAM) console, create a new IAM policy for the S3 bucket that you configured before.

   1. Select the JSON tab and paste the JSON policy which you copied from the Red Hat Hybrid Cloud Console Add a cloud integration wizard:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
              "s3:Get*",
              "s3:List*"
            ],
              "Resource": [
              "arn:aws:s3:::<your_bucket_name>", 1
              "arn:aws:s3:::<your_bucket_name>/*"
            ]
          },
      
          {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
              "s3:ListBucket",
              "cur:DescribeReportDefinitions"
            ],
            "Resource": "*"
          }
        ]
      }

   2. Enter a name for the policy and create the policy. Do not close the AWS IAM console. You will use it in the following steps.

In cost management:

4. In the Red Hat Hybrid Cloud Console Add a cloud integration wizard, click Next.

In the AWS Identity and Access Management console:

5. In the AWS IAM console, create a new IAM role:

   1. Select Another AWS account as the type of trusted entity.
   2. Enter 589173575009 as the Account ID to give Red Hat Hybrid Cloud Console read access to the AWS account cost data.

In cost management:

1. Copy your external ID from the Create IAM role step in the wizard.

In the AWS Identity and Access Management console:

1. Enter your external ID in the External ID field.
2. Attach the IAM policy you just configured.
3. Enter a role name and description.

In cost management:

6. In the Red Hat Hybrid Cloud Console Add a cloud integration wizard, click Next.

In the AWS Identity and Access Management console:

7. In the AWS IAM console, in the Roles section, open the summary screen for the role you just created.

   1. Copy the Role ARN, which is a string beginning with arn:aws:.

In cost management:

8. In the Red Hat Hybrid Cloud Console Add a cloud integration wizard, paste your Role ARN and click Next.
9. Review the details of your cloud integration and click Add.

Procedure

1. From the AWS Identity and Access Management (IAM) console, create an IAM policy for the Athena Lambda functions you will configure.

   1. Select the JSON tab and paste the following content in the JSON policy text box:

      {
      	"Version": "2012-10-17",
      	"Statement": [
          	{
              	"Effect": "Allow",
              	"Action": [
                  	"athena:*"
              	],
              	"Resource": [
                  	"*"
              	]
          	},
          	{
              	"Effect": "Allow",
              	"Action": [
                  	"glue:CreateDatabase",
                  	"glue:DeleteDatabase",
                  	"glue:GetDatabase",
                  	"glue:GetDatabases",
                  	"glue:UpdateDatabase",
                  	"glue:CreateTable",
                  	"glue:DeleteTable",
                  	"glue:BatchDeleteTable",
                  	"glue:UpdateTable",
                  	"glue:GetTable",
                  	"glue:GetTables",
                  	"glue:BatchCreatePartition",
                  	"glue:CreatePartition",
                  	"glue:DeletePartition",
                  	"glue:BatchDeletePartition",
                  	"glue:UpdatePartition",
                  	"glue:GetPartition",
                  	"glue:GetPartitions",
                  	"glue:BatchGetPartition"
              	],
              	"Resource": [
                  	"*"
              	]
          	},
          	{
              	"Effect": "Allow",
              	"Action": [
                  	"s3:GetBucketLocation",
                  	"s3:GetObject",
                  	"s3:ListBucket",
                  	"s3:ListBucketMultipartUploads",
                  	"s3:ListMultipartUploadParts",
                  	"s3:AbortMultipartUpload",
                  	"s3:CreateBucket",
                  	"s3:PutObject",
                  	"s3:PutBucketPublicAccessBlock"
              	],
              	"Resource": [
                  	"arn:aws:s3:::CHANGE-ME*"1
              	]
          	},
          	{
              	"Effect": "Allow",
              	"Action": [
                  	"s3:GetObject",
                  	"s3:ListBucket"
              	],
              	"Resource": [
                  	"arn:aws:s3:::CHANGE-ME*"2
              	]
          	},
          	{
              	"Effect": "Allow",
              	"Action": [
                  	"s3:ListBucket",
                  	"s3:GetBucketLocation",
                  	"s3:ListAllMyBuckets"
              	],
              	"Resource": [
                  	"*"
              	]
          	},
          	{
              	"Effect": "Allow",
              	"Action": [
                  	"sns:ListTopics",
                  	"sns:GetTopicAttributes"
              	],
              	"Resource": [
                  	"*"
              	]
          	},
          	{
              	"Effect": "Allow",
              	"Action": [
                  	"cloudwatch:PutMetricAlarm",
                  	"cloudwatch:DescribeAlarms",
                  	"cloudwatch:DeleteAlarms",
                  	"cloudwatch:GetMetricData"
              	],
              	"Resource": [
                  	"*"
              	]
          	},
          	{
              	"Effect": "Allow",
              	"Action": [
                  	"lakeformation:GetDataAccess"
              	],
              	"Resource": [
                  	"*"
              	]
          	},
          	{
              	"Effect": "Allow",
              	"Action": [
                  	"logs:*"
              	],
              	"Resource": "*"
          	}
      	]
      }

      1 1 1 2

      Replace CHANGE-ME* in both locations with the ARN for the S3 bucket you configured in step 2.2.

   2. Name the policy and complete the creation of the policy. Keep the AWS IAM console open because you will need it for the next step.

2. In the AWS IAM console, create a new IAM role:

   1. For the type of trusted entity, select AWS service.
   2. Select Lambda.
   3. Attach the IAM policy you just configured.
   4. Enter a role name and description and finish creating the role.

3. Store your login information in AWS Secrets Manager and add it to the role you created.

   1. Select Secret type: Other type of secret.
   2. Create a key for your Red Hat Hybrid Cloud Console client_id.
   3. Create a key for your Red Hat Hybrid Cloud Console client_secret.
   4. Add the values for your user name and password to the appropriate key.
   5. Click Continue, then name and store your secret.

   6. Update the role you created for your Lambda functions. Include the following code to reference the secret stored in AWS Secrets Manager:

      {
          "Sid": "VisualEditor3",
          "Effect": "Allow",
          "Action": [
              "secretsmanager:GetSecretValue",
              "secretsmanager:DescribeSecret"
          ],
          "Resource": "*"
      }

Procedure

1. Create two AmazonEventBridge schedules to trigger each of the functions that you created. You must trigger these functions at different cadences so that the Athena query is completed before it sends the reports:

   1. Add a name and description.
   2. In the Group field, select Default.
   3. In the Occurrence field, select Recurring schedule.
   4. In the Type field, select Chron-based.
   5. Set the cron-based schedules 12 hours apart. The following example triggers the function at 9AM and 9PM, 0 9 * * ? * and 0 21 * * ? *.
   6. Set a flexible time window.
   7. Click Next.

2. Set the Target detail to AWS Lambda invoke to associate this schedule with the Lambda function:

   1. Select the Lambda function you created before.
   2. Click Next.

3. Enable the schedule:

   1. Configure the retry logic.
   2. Ignore the encryption.
   3. Set the permissions to Create new role on the fly.
   4. Click Next.
4. Review your selections and click Create.

Procedure

1. Create Athena query for the Lambda function:

   1. Create a function for querying Athena.
   2. Select Author from scratch.
   3. Select the Python runtime.
   4. Select the x86_64 architecture.
   5. Select the role created before for permissions.
   6. Click Create.

2. Click the Code tab to add a script to collect the finalized data.

   1. Copy the Athena query function and add it to the query. Update the <integration_uuid> with the integration_uuid from the integration you created on console.redhat.com, which you can find by going to the the Integrations page and clicking your integration. Update the BUCKET and DATABASE variables with the bucket name and databases you created. Then, update export_name with the name of the data export Athena query you created before.

   2. Remove the comment from the following code:

      # last_month = now.replace(day=1) - timedelta(days=1)
      # year = last_month.strftime("%Y")
      # month = last_month.strftime("%m")
      # day = last_month.strftime("%d")
      # file_name = 'finalized-data.json'

   3. Click Deploy. Then click Test to see the execution results.

3. Create a Lambda function to post the report files to cost management:

   1. Select Author from scratch.
   2. Name your function.
   3. Select the Python runtime.
   4. Select the x86_64 architecture.
   5. Select the role created before for permissions.
   6. Click Create.

4. Click the Code tab to add a script to post the finalized data.

   1. Copy the post function and add it to the query. Update the secret_name with the name of your secret in AWS Secrets Manager. Update the bucket with the bucket name you created.

   2. Remove the comment from the following code:

      # file_name = 'finalized_data.json'

   3. Click Deploy. Then click Test to see the execution results.

5. Create an EventBridge schedule to trigger the two functions. For more information, see Section 2.7, “Creating event bridge schedules”.

   1. Set the EventBridge schedule to run one time a month on or after the 15th of the month because your AWS bill for the earlier period is final by that date. For example, (0 9 15 * ? *) and (0 21 15 * ? *).

Procedure

1. From Red Hat Hybrid Cloud Console, navigate to the cost management settings page.
2. Under Show cost as select Amortized or Blended.
3. Click Save.

Procedure

1. In AWS, tag your instances of RHEL that you want to meter. For more information about tagging your instances of RHEL in AWS, see Adding tags to an AWS resource.
2. From Red Hat Hybrid Cloud Console, click Settings .
3. Click Integrations.
4. Click the more options menu for your integration. Click Edit.
5. In Metered Product, select Red Hat Enterprise Linux from the drop-down to activate metering.