SupportConsoleDevelopersStart a trialContact

Chapter 4. Integrate your AWS account with data filtering

Configure a function script in AWS to filter and export a subset of billing data to object storage, enabling selective data sharing with cost management while maintaining privacy control.

Note
  • If you created an unfiltered AWS integration, do not complete the following steps. Your AWS integration is already complete.
  • AWS is a third-party product and its UI and documentation can change. The instructions for configuring third-party integrations are correct at the time of publishing. For the most up-to-date information, see the AWS documentation.

4.1. Initiate the AWS integration process
Copy link

Add an AWS integration to enable cost management to process your AWS Cost and Usage Reports and provide visibility into your cloud spending patterns.

Prerequisites

Procedure

  1. From Red Hat Hybrid Cloud Console, click Settings Settings icon > Integrations.
  2. On the Settings page, click Create Integration Cloud to enter the Add a cloud integration wizard.
  3. On the Select cloud provider step, select Amazon Web Services. Click Next.
  4. Enter a name for the integration and click Next.
  5. On the Select configuration step, select Manual configuration. Do not select the recommended configuration mode when you configure cost management integrations. The recommendation is for other workflows.
  6. In the Select application step, select Cost management. Click Next.

4.2. Prepare cost data for filtering with Athena
Copy link

Create an Amazon S3 bucket and data export using Athena to prepare your billing data for filtering before cost analysis.

Procedure

  1. Log in to your AWS account.

  2. In Billing and Cost Management, create a data export to deliver to your S3 bucket. Specify the following values and accept the defaults for any other values:

    • Export details: Legacy CUR export
    • Export name: <rh_cost_report> (save this name, you will use it later)
    • Additional export content: Include resource IDs
    • S3 bucket: Select an S3 bucket that you configured previously or create a new bucket and accept the default settings.
    • Time granularity: Hourly
    • Report data integration: Amazon Athena which is required for lambda queries
    • Compression type: Parquet
    • S3 path prefix: cost
    Note

    For more details about configuration, see the AWS Billing and Cost Management documentation.

Create a second S3 bucket to store filtered billing data before sharing it with Red Hat, enabling you to control which cost information is exposed.

Procedure

  1. In your AWS account, navigate to S3 and click Create Bucket.
  2. Create a bucket and apply the default policy.
  3. Click Save.
  4. In the cost management Create an integration wizard, navigate to the Create storage step.
  5. Paste the name of your S3 bucket and select the region that it was created in.
  6. Click Next.
  7. On the Create cost and usage report step, select I wish to manually customize the CUR sent to Cost Management.
  8. Click Next.

4.4. Import tags to organize cost data
Copy link

Activate your tags in AWS and then give cost management permissions to import them automatically. Tags can help you organize your AWS resources in cost management.

For more information about tagging, see Adding tags to an AWS resource.

Procedure

  1. In the AWS Billing console, click Cost Allocation Tags.

  2. Select the tags that you want to use in cost management. Click Activate.

    • If your organization is converting systems from CentOS 7 to RHEL and using hourly billing, activate the com_redhat_rhel tag for your systems.

Create an IAM policy and role in AWS to grant cost management secure, read-only access to your AWS Cost and Usage Reports for billing analysis.

Procedure

  1. In the cost management Add a cloud integration wizard, on the Tags, aliases, and organizational units step, select any additional data points that you want to include, then click Next:

    • Select Include AWS account aliases to display an AWS account alias rather than an account number. In the next step of the wizard, this selection populates iam:ListAccountAliases in your IAM JSON policy.
    • Select Include AWS organization units if you are using consolidated billing rather than the account ID. In the next step of the wizard, this selection populates _organization:List*_ and _organizations:Describe*_ in your IAM JSON policy.
  2. Copy the IAM JSON policy that is generated based on your selections.

  3. In the AWS Identity and Access Management console, create a new IAM policy:

    1. Select the JSON tab and enter the IAM JSON policy that you copied from the Red Hat Hybrid Cloud Console Add a cloud integration wizard.

      Example IAM JSON policy

      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<your_bucket_name>",
                "arn:aws:s3:::<your_bucket_name>/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "cur:DescribeReportDefinitions",
            ],
            "Resource": "*"
        }
    ]
}

    2. Enter a name and create your policy.

  4. Create a new IAM role:

    1. Select Another AWS account as the type of trusted entity.
    2. Enter 589173575009 for the Account ID to give Red Hat Hybrid Cloud Console read access to the AWS account’s cost data.
  5. In the cost management Add a cloud integration wizard, click Next, then copy your External ID from the Create IAM role step.

  6. In the AWS Identity and Access Management console, complete the IAM role configuration:

    1. Enter your External ID.
    2. Attach the IAM policy that you configured.
    3. Enter a name and description to finish creating your role.
  7. In Roles, open the summary screen for the role that you created, then copy the Role ARN (starts with arn:aws:).
  8. In the cost management Add a cloud integration wizard, click Next, enter your Role ARN, then click Next again.
  9. Review the details of your cloud integration and click Add.

Verification

  • Verify that Cost management can access Cost and Usage data from your AWS account and linked AWS accounts.

    Note

    The data can take a few days to populate before it shows on the cost management dashboard.

4.6. Configure data filtering
Copy link

Control and refine your billing reports in AWS before they are shared with Red Hat. While more complex to configure than an unfiltered connection, this method enables you to restrict data sharing to specific Red Hat products or business units.

Create an IAM policy and role to grant your Lambda function and Athena the permissions needed to filter and process billing data before sharing.

Procedure

  1. From the AWS Identity and Access Management (IAM) console, create an IAM policy for the Athena Lambda functions you will configure.

    1. Select the JSON tab and paste the following content in the JSON policy text box: 

      {
	"Version": "2012-10-17",
	"Statement": [
    	{
        	"Effect": "Allow",
        	"Action": [
            	"athena:*"
        	],
        	"Resource": [
            	"*"
        	]
    	},
    	{
        	"Effect": "Allow",
        	"Action": [
            	"glue:CreateDatabase",
            	"glue:DeleteDatabase",
            	"glue:GetDatabase",
            	"glue:GetDatabases",
            	"glue:UpdateDatabase",
            	"glue:CreateTable",
            	"glue:DeleteTable",
            	"glue:BatchDeleteTable",
            	"glue:UpdateTable",
            	"glue:GetTable",
            	"glue:GetTables",
            	"glue:BatchCreatePartition",
            	"glue:CreatePartition",
            	"glue:DeletePartition",
            	"glue:BatchDeletePartition",
            	"glue:UpdatePartition",
            	"glue:GetPartition",
            	"glue:GetPartitions",
            	"glue:BatchGetPartition"
        	],
        	"Resource": [
            	"*"
        	]
    	},
    	{
        	"Effect": "Allow",
        	"Action": [
            	"s3:GetBucketLocation",
            	"s3:GetObject",
            	"s3:ListBucket",
            	"s3:ListBucketMultipartUploads",
            	"s3:ListMultipartUploadParts",
            	"s3:AbortMultipartUpload",
            	"s3:CreateBucket",
            	"s3:PutObject",
            	"s3:PutBucketPublicAccessBlock"
        	],
        	"Resource": [
            	"arn:aws:s3:::CHANGE-ME*"
      1 
      
        	]
    	},
    	{
        	"Effect": "Allow",
        	"Action": [
            	"s3:GetObject",
            	"s3:ListBucket"
        	],
        	"Resource": [
            	"arn:aws:s3:::CHANGE-ME*"
      2 
      
        	]
    	},
    	{
        	"Effect": "Allow",
        	"Action": [
            	"s3:ListBucket",
            	"s3:GetBucketLocation",
            	"s3:ListAllMyBuckets"
        	],
        	"Resource": [
            	"*"
        	]
    	},
    	{
        	"Effect": "Allow",
        	"Action": [
            	"sns:ListTopics",
            	"sns:GetTopicAttributes"
        	],
        	"Resource": [
            	"*"
        	]
    	},
    	{
        	"Effect": "Allow",
        	"Action": [
            	"cloudwatch:PutMetricAlarm",
            	"cloudwatch:DescribeAlarms",
            	"cloudwatch:DeleteAlarms",
            	"cloudwatch:GetMetricData"
        	],
        	"Resource": [
            	"*"
        	]
    	},
    	{
        	"Effect": "Allow",
        	"Action": [
            	"lakeformation:GetDataAccess"
        	],
        	"Resource": [
            	"*"
        	]
    	},
    	{
        	"Effect": "Allow",
        	"Action": [
            	"logs:*"
        	],
        	"Resource": "*"
		},
		{
			"Sid": "VisualEditor3",
			"Effect": "Allow",
			"Action": [
				"secretsmanager:GetSecretValue",
				"secretsmanager:DescribeSecret"
			],
			"Resource": "*"
		}
	]
}

  2. Replace CHANGE-ME* in both locations with the ARN for the S3 bucket you configured in step 2.2.

    1. Enter a name and finish creating the policy.

  3. Create a new IAM role:

    1. For the type of trusted entity, select AWS service.
    2. Select Lambda.
    3. Attach the IAM policy you just configured.
    4. Enter a role name and description and finish creating the role.

  4. Store your Service Account information in AWS Secrets Manager and add it to the role you created.

    Note

    The service account used to upload data must be added to a user group that has the cost management:settings:write permission. When configuring this in the Red Hat Hybrid Cloud Console, ensure that the group’s assigned role includes the cost-management application and the settings resource type with write operations enabled. This is required to access the Ingress Reports API for data uploads. .. Select Secret type: Other type of secret to create a secret. .. Create a key for your Red Hat Hybrid Cloud Console Service Account client_id. .. Create a key for your Red Hat Hybrid Cloud Console Service Account client_secret. .. Add the values for your Service Account to the corresponding key. .. Click Continue, then enter a name and store your secret.

Configure an Athena table to query and filter your billing data before sharing it with cost management, enabling selective data exposure.

The following configuration only provides access to additional stored information. It does not provide access to anything else.

Procedure

  1. In the AWS S3 console, go to the S3 bucket you configured in step 2.2. Then, go to the crawler-cfn.yml file, which is in the path created by your data export you configured. For example: {bucket-name}/{S3_path_prefix}/{export_name}/crawler-cfn.yml. Copy the Object URL for the crawler-cfn.yml.

  2. From Cloudformation in the AWS console, create a stack with new resources:

    1. Choose an existing template.
    2. Select Specify Template.
    3. Select Template Source: Amazon S3 URL.
    4. Paste the object URL you copied before.
  3. Enter a name and click Next.
  4. Click I acknowledge that AWS Cloudformation might create IAM resources and then click Submit.

4.6.3. Define specific data to query with Athena
Copy link

Create an Athena query to extract only Red Hat-related expenses from your billing data, reducing the data volume shared with cost management.

You might need just the query included with the example script, for example, if you are filtering for Red Hat spending. If you need something more advanced, create a custom query. If you are using RHEL metering, you must adjust the query to return data that is specific to your RHEL subscriptions. The following steps guide you through creating a RHEL subscription query.

Example Athena query for Red Hat spend: 

SELECT *
    FROM <your_export_name>
    WHERE (
            bill_billing_entity = 'AWS Marketplace'
            AND line_item_legal_entity like '%Red Hat%'
        )
        OR (
            line_item_legal_entity like '%Amazon Web Services%'
            AND line_item_line_item_description like '%Red Hat%'
        )
        OR (
            line_item_legal_entity like '%Amazon Web Services%'
            AND line_item_line_item_description like '%RHEL%'
        )
        OR (
            line_item_legal_entity like '%AWS%'
            AND line_item_line_item_description like '%Red Hat%'
        )
        OR (
            line_item_legal_entity like '%AWS%'
            AND line_item_line_item_description like '%RHEL%'
        )
        OR (
            line_item_legal_entity like '%AWS%'
            AND product_product_name like '%Red Hat%'
        )
        OR (
            line_item_legal_entity like '%Amazon Web Services%'
            AND product_product_name like '%Red Hat%'
        )
        AND year = '2024'
        AND month = '07'

Procedure

  1. In your link:https://aws.amazon.com/console/AWS account, go to Amazon Athena from the editor tab.
  2. From the Data source menu, select AwsDataCatalog.
  3. From the Database menu, select your data export. Your data export name is prepended with athenacurcfn_ followed by your data export name. For example, {your_export_name}.

  4. Paste the following example query into the Query field. Replace the your_export_name value with your data export name. 

    SELECT column_name
FROM information_schema.columns
WHERE table_name = '<your_export_name>'
AND column_name LIKE 'resource_tags_%';
  5. Click Run. The results of this query returns all the tag related columns for your data set.
  6. Copy the tag column that matches the column used for your RHEL tags.

  7. Paste in the following example query. Replace the your_export_name, the tags column copied in the step before, and the year and month you want to query. The result returns EC2 instances appropriately tagged for RHEL subscriptions. Copy and save this query for use in the future Lambda function. 

    SELECT *
        FROM <your_export_name>
        WHERE (
            line_item_product_code = 'AmazonEC2'
            AND strpos(lower(<rhel_tag_column_name>), 'com_redhat_rhel') > 0
        )
        AND year = '<year>'
        AND month = '<month>'

Create a Lambda function to automate filtering of Red Hat-related expenses from your billing data, streamlining the data preparation process.

Procedure

  1. In the AWS console, go to Lambda and click Create function.
  2. Click Author from scratch.
  3. Enter a name for your function.
  4. From the Runtime menu, select the latest version of Python available.
  5. From the Architecture menu, select x86_64.
  6. Under Permissions select the Athena role you created.
  7. To add the query you built as part of the Lambda function, click Create function to save your progress.

  8. From the function Code tab, paste this script. Update the following lines:

    your_integration_external_id
    Enter the integration UUID you copied in the Configuring an IAM policy to enable account access for AWS Cost and Usage Reports step.
    bucket
    Enter the name of the S3 bucket you created to store filtered reports during the Creating a bucket for storing filtered data reporting step.
    database
    Enter the database name used in the Building your Athena query step.
    export_name
    Enter the name of your data export from when you created an AWS S3 bucket for storing your cost data.

  9. Update the default query with your custom one by replacing the where clause, for example: 

    # Athena query
query = f"SELECT * FROM {database}.{export_name} WHERE (line_item_product_code = 'AmazonEC2' AND strpos(lower(<rhel_tag_column_name>), 'com_redhat_rhel') > 0) AND year = '{year}' AND month = '{month}'"
  10. Click Deploy to test the function.

Create a second Lambda function to deliver filtered billing reports to a Red Hat-accessible bucket, completing the secure data sharing pipeline.

Procedure

  1. Go to Lambda in the AWS console and click Create function.
  2. Click Author from scratch.
  3. Enter a name for your function.
  4. From the Runtime menu, select the latest version of Python available.
  5. Select x86_64 as the architecture type for your function.
  6. Under Permissions, select the Athena role that you created.
  7. Click Create function.

  8. Paste this script into the function and replace the following lines:

    secret_name = "CHANGEME"
    Enter the secret name that you used in the Enabling account access for Athena step.
    bucket = "<your_S3_Bucket_Name>"
    Enter the name of the S3 bucket that you created to store filtered reports in the Creating a bucket for storing filtered data reporting step.
  9. Click Deploy to test the function.

Schedule automated execution of your Lambda functions using Amazon EventBridge to keep filtered billing data current without manual intervention.

Procedure

  1. Create two AmazonEventBridge schedules to trigger each of the functions that you created. You must trigger these functions at different cadences so that the Athena query is completed before it sends the reports:

    1. Add a name and description.
    2. In the Group field, select Default.
    3. In the Occurrence field, select Recurring schedule.
    4. In the Type field, select Chron-based.
    5. Set the cron-based schedules 12 hours apart. The following example triggers the function at 9AM and 9PM, 0 9 * * ? * and 0 21 * * ? *.
    6. Set a flexible time window.
    7. Click Next.

  2. Set the Target detail to AWS Lambda invoke to associate this schedule with the Lambda function:

    1. Select the Lambda function you created before.
    2. Click Next.

  3. Enable the schedule:

    1. Configure the retry logic.
    2. Ignore the encryption.
    3. Set the permissions to Create new role on the fly.
    4. Click Next.
  4. Review your selections and click Create.

AWS sends final reports for the last month at the start of the following month. Send these finalized reports to Cost management, which will analyze the extra information.

Procedure

  1. Create Athena query for the Lambda function:

    1. Create a function for querying Athena.
    2. Select Author from scratch.
    3. Select the Python runtime.
    4. Select the x86_64 architecture.
    5. Select the role created before for permissions.
    6. Click Create.

  2. Click the Code tab to add a script to collect the finalized data.

    1. Copy the Athena query function and add it to the query. Update the <integration_uuid> with the integration_uuid from the integration you created on console.redhat.com, which you can find by going to the the Integrations page and clicking your integration. Update the BUCKET and DATABASE variables with the bucket name and databases you created. Then, update export_name with the name of the data export Athena query you created before.

    2. Remove the comment from the following code: 

      # last_month = now.replace(day=1) - timedelta(days=1)
# year = last_month.strftime("%Y")
# month = last_month.strftime("%m")
# day = last_month.strftime("%d")
# file_name = 'finalized-data.json'
    3. Click Deploy. Then click Test to see the execution results.

  3. Create a Lambda function to post the report files to cost management:

    1. Select Author from scratch.
    2. Name your function.
    3. Select the Python runtime.
    4. Select the x86_64 architecture.
    5. Select the role created before for permissions.
    6. Click Create.

  4. Click the Code tab to add a script to post the finalized data.

    1. Copy the post function and add it to the query. Update the secret_name with the name of your secret in AWS Secrets Manager. Update the bucket with the bucket name you created.

    2. Remove the comment from the following code: 

      # file_name = 'finalized_data.json'
    3. Click Deploy. Then click Test to see the execution results.

  5. Create an EventBridge schedule to trigger the two functions. For more information, see Section 4.7, “Schedule automated report updates with an AmazonEventBridge”.

    1. Set the EventBridge schedule to run one time a month on or after the 15th of the month because your AWS bill for the earlier period is final by that date. For example, (0 9 15 * ? *) and (0 21 15 * ? *).

Verification

  • Verify that cost management is collecting Cost and Usage data from your AWS account and linked AWS accounts.

    Note

    The data can take a few days to populate before it shows on the cost management dashboard.

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top