Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 2. Filtering your Amazon Web Services data before sending it to cost management
To copy exports, object storage buckets, and filter your data to share only a subset of your billing information with Red Hat, configure a function script in AWS. This option is only recommended if your organization has third party data limitations.
To add data integrations to cost management, you must have a Red Hat account with Cloud Administrator permissions.
To configure your AWS account to be a cost management data integration:
- Create an AWS S3 bucket to store your cost data.
- Create an AWS S3 bucket to report your filtered cost management data.
- Configure IAM roles for your cost data bucket.
- Add your AWS integrations to Red Hat Hybrid Cloud Console.
- Configure IAM roles for AWS Athena.
- Enable Athena.
- Create Lambda tasks for Athena to export filtered data to your S3 bucket.
Because you must complete some of the following steps in the AWS Console, and some steps in the cost management user interface, keep both applications open in a web browser.
2.1. Adding an AWS account as an integration
Add an AWS integration so the cost management application can processes the Cost and Usage Reports from your AWS account. You can add an AWS integration automatically by providing your AWS account credentials, or you can configure cost management to filter the data that you send to Red Hat.
Prerequisites
- To add data integrations to cost management, you must have a Red Hat account with Cloud Administrator permissions.
Procedure
-
From Red Hat Hybrid Cloud Console, click Settings Menu
> Integrations.
- On the Settings page, in the Cloud tab, click Add integration.
- On the Select integration type step, in the Add a cloud integration wizard, select . Click .
Enter a name for the integration and click Next.
- Select to customize your integration. For example, you can filter your information before it is sent to cost management. Click .
- In the Select application step, select Cost management. Click .
2.2. Creating an AWS S3 bucket for storing your cost data
Create an Amazon S3 bucket with permissions configured to store billing reports.
Procedure
- Log in to your AWS account to begin configuring cost and usage reporting.
- In the AWS S3 console, create a new S3 bucket or use an existing bucket. If you are configuring a new S3 bucket, accept the default settings.
In the AWS Billing console, create a data export that will be delivered to your S3 bucket. Specify the following values and accept the defaults for any other values:
- Report name: <rh_cost_report> (note this name as you will use it later)
- Additional report details: Include resource IDs
- S3 bucket: <the S3 bucket you configured previously>
- Time granularity: Hourly
- Enable report data integration for: Amazon Redshift, Amazon QuickSight (do not enable report data integration for Amazon Athena)
- Compression type: GZIP
Report path prefix: cost
NoteSee the AWS Billing and Cost Management documentation for more details on configuration.
- On the Create storage step, in the Add a cloud integration wizard, paste the name of your S3 bucket and select the region that it was created in. Click .
- In the Add a cloud integration wizard, on the Create cost and usage report step, click .
2.3. Creating a data export for filtered data reporting
To create a data export in AWS, set up Athena and Lambda functions to filter the data. This process delivers your data export to your S3 bucket. Complete the following steps:
Procedure
- Log in to your AWS account.
- In the AWS S3 console, create a data export that will be delivered to your S3 bucket.
- Enter a report name. Save this name. You will use it later.
- Click include resource IDs.
- Click Next.
- From Configure S3 Bucket, click Configure. Create a bucket and apply the default policy.
- Click Save.
- On the Create storage step, in the Add a cloud integration wizard, paste the name of your S3 bucket and select the region that it was created in and click .
- On the Create cost and usage report step in the Add a cloud integration wizard, select I wish to manually customize the CUR sent to Cost Management and click .
2.4. Activating AWS tags
To use tags to organize your AWS resources in the cost management application, activate your tags in AWS to allow them to be imported automatically.
Procedure
In the AWS Billing console:
- Open the Cost Allocation Tags section.
- Select the tags you want to use in the cost management application, and click Activate.
If your organization is converting systems from CentOS 7 to RHEL and using hourly billing, activate the
com_redhat_rhel
tag for your systems in the Cost Allocation Tags section of the AWS console.- After tagging the instances of RHEL you want to meter in AWS, select .
- In the Red Hat Hybrid Cloud Console Integrations wizard, click .
Additional resources
For more information about tagging, see Adding tags to an AWS resource.
2.5. Enabling minimal account access for cost and usage consumption
For cost management to provide data, it must consume the Cost and Usage Reports produced by AWS. For cost management to obtain this data with a minimal amount of access, create an IAM policy and role for cost management to use. This configuration only provides access to the stored information.
Procedure
From the AWS Identity and Access Management (IAM) console, create a new IAM policy for the S3 bucket that you configured.
Select the JSON tab and paste the following content in JSON policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListAllMyBuckets" ], "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::<your_bucket_name>", "arn:aws:s3:::<your_bucket_name>/*"1 ] } ] }
- 1
- Replace
<your_bucket_name>
in both locations with the name of the Amazon S3 bucket you configured for storing your filtered data.
- Provide a name for the policy and complete the creation of the policy. Keep the AWS IAM console open. You will need it for the next step.
- In the Red Hat Hybrid Cloud Console Add a cloud integration wizard, click .
In the AWS IAM console, create a new IAM role:
- For the type of trusted entity, select AWS account.
- Enter 589173575009 as the Account ID to provide the cost management application with read access to the AWS account cost data.
- Attach the IAM policy you just configured.
- Enter a role name and description and finish creating the role.
- In the Red Hat Hybrid Cloud Console Add a cloud integration wizard, click .
-
In the AWS IAM console, from Roles, open the summary screen for the role you just created and copy the Role ARN. It is a string beginning with
arn:aws:
. - In the Red Hat Hybrid Cloud Console Add a cloud integration wizard, enter the ARN on the Enter ARN page and click .
- Review the details of your cloud integration and click .
Next steps
Return to AWS to customize your AWS data export by configuring Athena and Lambda to filter your reports.
2.6. Enabling account access for Athena
To provide data within the web interface and API, create an IAM policy and role for hybrid committed spend to use. This configuration provides access to the stored information and nothing else.
Procedure
From the AWS Identity and Access Management (IAM) console, create an IAM policy for the Athena Lambda functions you will configure.
Select the JSON tab and paste the following content in the JSON policy text box:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "athena:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "glue:CreateDatabase", "glue:DeleteDatabase", "glue:GetDatabase", "glue:GetDatabases", "glue:UpdateDatabase", "glue:CreateTable", "glue:DeleteTable", "glue:BatchDeleteTable", "glue:UpdateTable", "glue:GetTable", "glue:GetTables", "glue:BatchCreatePartition", "glue:CreatePartition", "glue:DeletePartition", "glue:BatchDeletePartition", "glue:UpdatePartition", "glue:GetPartition", "glue:GetPartitions", "glue:BatchGetPartition" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts", "s3:AbortMultipartUpload", "s3:CreateBucket", "s3:PutObject", "s3:PutBucketPublicAccessBlock" ], "Resource": [ "arn:aws:s3:::CHANGE-ME*"1 ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::CHANGE-ME*"2 ] }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListAllMyBuckets" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "sns:ListTopics", "sns:GetTopicAttributes" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "cloudwatch:PutMetricAlarm", "cloudwatch:DescribeAlarms", "cloudwatch:DeleteAlarms", "cloudwatch:GetMetricData" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "logs:*" ], "Resource": "*" } ] }
- Provide a name for the policy and complete the creation of the policy. Keep the AWS IAM console open because you will need it for the next step.
In the AWS IAM console, create a new IAM role:
- For the type of trusted entity, select AWS service.
- Select Lambda.
- Attach the IAM policy you just configured.
- Enter a role name and description and finish creating the role.
2.6.1. Configuring Athena for report generation
Configuring Athena to provide a filtered data export for cost management.
The following configuration only provides access to additional stored information. It does not provide access to anything else:
Procedure
-
In the AWS S3 console, navigate to the filtered bucket that you created and download the
crawler-cfn.yml
file. - From Cloudformation in the AWS console, create a new stack.
- Select Template as Ready.
-
Upload the
crawler-cfn.yml
file that you previously downloaded. This should load immediately. - Click Next.
- Enter a name and click Next.
- Click I acknowledge that AWS Cloudformation might create IAM resources and then click Submit.
2.6.2. Creating a Lambda function for Athena
You must create a Lambda function that queries the data export for your Red Hat related expenses and creates a report of your filtered expenses.
Procedure
- Navigate to Lambda in the AWS console and click Create function.
- Click Author from scratch.
- Enter a name your function.
- From the Runtime dropdown, Select python 3.7.
- Select x86_64 as the Architecture.
- Under Permissions select the Athena role you created.
- Click Create function.
Paste the following code to the function:
import boto3 import uuid import json from datetime import datetime now = datetime.now() year = now.strftime("%Y") month = now.strftime("%m") day = now.strftime("%d") # Vars to Change! integration_uuid = <your_integration_uuid> # integration_uuid bucket = <your_S3_Bucket_Name> # Bucket created for query results database = 'athenacurcfn_athena_cost_and_usage' # Database to execute athena queries output=f's3://{bucket}/{year}/{month}/{day}/{uuid.uuid4()}' # Output location for query results # Athena query query = f"SELECT * FROM {database}.koku_athena WHERE ((bill_billing_entity = 'AWS Marketplace' AND line_item_legal_entity like '%Red Hat%') OR (line_item_legal_entity like '%Amazon Web Services%' AND line_item_line_item_description like '%Red Hat%') OR (line_item_legal_entity like '%Amazon Web Services%' AND line_item_line_item_description like '%RHEL%') OR (line_item_legal_entity like '%AWS%' AND line_item_line_item_description like '%Red Hat%') OR (line_item_legal_entity like '%AWS%' AND line_item_line_item_description like '%RHEL%') OR (line_item_legal_entity like '%AWS%' AND product_product_name like '%Red Hat%') OR (line_item_legal_entity like '%Amazon Web Services%' AND product_product_name like '%Red Hat%')) AND year = '{year}' AND month = '{month}'" def lambda_handler(event, context): # Initiate Boto3 athena Client athena_client = boto3.client('athena') # Trigger athena query response = athena_client.start_query_execution( QueryString=query, QueryExecutionContext={ 'Database': database }, ResultConfiguration={ 'OutputLocation': output } ) # Save query execution to s3 object s3 = boto3.client('s3') json_object = {"integration_uuid": integration_uuid, "bill_year": year, "bill_month": month, "query_execution_id": response.get("QueryExecutionId"), "result_prefix": output} s3.put_object( Body=json.dumps(json_object), Bucket=bucket, Key='query-data.json' ) return json_object
Replace
<your_integration_uuid>
with the UUID from the integration you created on console.redhat.com. Replace<your_S3_Bucket_Name>
with the name of the S3 bucket you created to store reports.- Click Deploy to test the function.
2.6.3. Creating a Lambda function to post the report files
You must create a Lambda function to post your report files to the S3 bucket that you created.
Procedure
- Navigate to Lambda in the AWS console and click Create function.
- Click Author from scratch
- Enter a name your function
- From the Runtime dropdown, Select python 3.7.
- Select x86_64 as the Architecture.
- Under Permissions select the Athena role you created.
- Click Create function.
Paste the following code to the function:
import boto3 import json import requests from botocore.exceptions import ClientError def get_credentials(secret_name, region_name): session = boto3.session.Session() client = session.client( service_name='secretsmanager', region_name=region_name ) try: get_secret_value_response = client.get_secret_value( SecretId=secret_name ) except ClientError as e: raise e secret = get_secret_value_response['SecretString'] return secret secret_name = "CHANGEME" region_name = "us-east-1" secret = get_credentials(secret_name, region_name) json_creds = json.loads(secret) USER = json_creds.get("<your_username>") # console.redhat.com Username PASS = json_creds.get("<your_password>") # console.redhat.com Password bucket = "<your_S3_Bucket_Name>" # Bucket for athena query results def lambda_handler(event, context): # Initiate Boto3 s3 and fetch query file s3_resource = boto3.resource('s3') json_content = json.loads(s3_resource.Object(bucket, 'query-data.json').get()['Body'].read().decode('utf-8')) # Initiate Boto3 athena Client and attempt to fetch athena results athena_client = boto3.client('athena') try: athena_results = athena_client.get_query_execution(QueryExecutionId=json_content["query_execution_id"]) except Exception as e: return f"Error fetching athena query results: {e} \n Consider increasing the time between running and fetching results" reports_list = [] prefix = json_content["result_prefix"].split(f'{bucket}/')[-1] # Initiate Boto3 s3 client s3_client = boto3.client('s3') result_data = s3_client.list_objects(Bucket=bucket, Prefix=prefix) for item in result_data.get("Contents"): if item.get("Key").endswith(".csv"): reports_list.append(item.get("Key")) # Post results to console.redhat.com API url = "https://console.redhat.com/api/cost-management/v1/ingress/reports/" json_data = {"source": json_content["integration_uuid"], "reports_list": reports_list, "bill_year": json_content["bill_year"], "bill_month": json_content["bill_month"]} resp = requests.post(url, json=json_data, auth=(USER, PASS)) return resp
Replace
<your_username>
with your username for console.redhat.com. Replace<your_password>
with your password for console.redhat.com. Replace<your_S3_Bucket_Name>
with the name of the S3 bucket that you created to store reports.- Click Deploy to test the function.