Chapter 20. Configuring for Amazon Web Services (AWS)
20.1. Overview
OpenShift Container Platform can be configured to access an AWS EC2 infrastructure, including using AWS volumes as persistent storage for application data. After you configure AWS, some additional configurations must be completed on the OpenShift Container Platform hosts.
20.1.1. Configuring authorization for Amazon Web Services (AWS)
Permissions AWS instances require either IAM account with Programmatic Access using an access and secret key or IAM role assigned to instances at creation to be able to request and manage load balancers and storage in OpenShift Container Platform.
The IAM account or IAM role must have the following policy permissions to have full cloud provider functionality.
{ "Version": "2012-10-17", "Statement": [ { "Action": [ "ec2:DescribeVolume*", "ec2:CreateVolume", "ec2:CreateTags", "ec2:DescribeInstances", "ec2:AttachVolume", "ec2:DetachVolume", "ec2:DeleteVolume", "ec2:DescribeSubnets", "ec2:CreateSecurityGroup", "ec2:DescribeSecurityGroups", "ec2:DeleteSecurityGroup", "ec2:DescribeRouteTables", "ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress", "elasticloadbalancing:DescribeTags", "elasticloadbalancing:CreateLoadBalancerListeners", "elasticloadbalancing:ConfigureHealthCheck", "elasticloadbalancing:DeleteLoadBalancerListeners", "elasticloadbalancing:RegisterInstancesWithLoadBalancer", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:CreateLoadBalancer", "elasticloadbalancing:DeleteLoadBalancer", "elasticloadbalancing:ModifyLoadBalancerAttributes", "elasticloadbalancing:DescribeLoadBalancerAttributes" ], "Resource": "*", "Effect": "Allow", "Sid": "1" } ] }
aws iam put-role-policy \ --role-name openshift-role \ --policy-name openshift-admin \ --policy-document file: //openshift_iam_policy
aws iam put-user-policy \ --user-name openshift-admin \ --policy-name openshift-admin \ --policy-document file: //openshift_iam_policy
The OpenShift node instances only need the ec2:DescribeInstance
permission but the installer only allows for a single AWS access key and secret to be defined. This can be bypassed using IAM roles and assigning the permissions above to the master instances and the ec2:DescribeInstance
to nodes.
20.1.1.1. Configuring the OpenShift Container Platform cloud provider at installation
Procedure
To configure the configure the Amazon Web Services cloud provider using an IAM account with an access and secret key add the following values to the inventory:
[OSEv3:vars] openshift_cloudprovider_kind=aws openshift_clusterid=openshift 1 openshift_cloudprovider_aws_access_key=AKIAJ6VLBLISADPBUA 2 openshift_cloudprovider_aws_secret_key=g/8PmDNYHVSQn0BQE+xtsHzbaZaGYjGNzhbdgwjH 3
To configure the configure the Amazon Web Services cloud provider using an IAM role add the following values to the inventory:
[source,yaml]
----
[OSEv3:vars]
openshift_cloudprovider_kind=aws
openshift_clusterid=openshift 1
----
<1> A tag assigned to all resources (instances, load balancers, vpc, etc) used for OpenShift.
NOTE: The IAM role takes the place of needing an access and secret key.
20.1.1.2. Configuring the OpenShift Container Platform cloud provider after installation
In the event that the Amazon Web Services cloud provider values were not provided at installation time the configuration can be defined and created after the installation. Follow the steps to configure the configuration file and manually configuring the master and node Manually Configuring OpenShift Container Platform Masters for AWS.
-
Every master host, node host, and subnet must have the
kubernetes.io/cluster/<clusterid>,Value=(owned|shared)
tag. One security group, preferably the one linked to the nodes, must have the
kubernetes.io/cluster/<clusterid>,Value=(owned|shared)
tag.-
Do not tag all security groups with the
kubernetes.io/cluster/<clusterid>,Value=(owned|shared)
tag or the Elastic Load Balancing (ELB) will not be able to create a load balancer.
-
Do not tag all security groups with the
20.2. Configuring a Security Group
When installing OpenShift Container Platform on AWS, ensure that you set up the appropriate security groups.
These are some ports that you must have in your security groups, without which the installation fails. You may need more depending on the cluster configuration you want to install. For more information and to adjust your security groups accordingly, see Required Ports for more information.
All OpenShift Container Platform Hosts |
|
etcd Security Group |
|
Master Security Group |
|
Node Security Group |
|
Infrastructure Nodes (ones that can host the OpenShift Container Platform router) |
|
CRI-O |
If using CRIO, you must open tcp/10010 to allow |
If configuring external load-balancers (ELBs) for load balancing the masters and/or routers, you also need to configure Ingress and Egress security groups for the ELBs appropriately.
20.2.1. Overriding Detected IP Addresses and Host Names
In AWS, situations that require overriding the variables include:
Variable | Usage |
---|---|
|
The user is installing in a VPC that is not configured for both |
| You have multiple network interfaces configured and want to use one other than the default. |
|
|
|
|
For EC2 hosts in particular, they must be deployed in a VPC that has both DNS host names
and DNS resolution
enabled.
20.2.1.1. Configuring the OpenShift Container Platform registry for Amazon Web Services (AWS)
Amazon Web Services (AWS) provides object cloud storage that OpenShift Container Platform can use to store container images using the OpenShift Container Platform container registry.
For more information, see Amazon S3.
Prerequisites
OpenShift Container Platform uses S3 for image storage. A S3 bucket, IAM policy, and IAM user with Programmatic Access
should be created to allow for the installer to configure the registry.
The example below uses awscli to create a bucket with the name of openshift-registry-storage
in the region of us-east-1
.
# aws s3api create-bucket \ --bucket openshift-registry-storage \ --region us-east-1
The default policy
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads" ], "Resource": "arn:aws:s3:::S3_BUCKET_NAME" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListMultipartUploadParts", "s3:AbortMultipartUpload" ], "Resource": "arn:aws:s3:::S3_BUCKET_NAME/*" } ] }
20.2.1.1.1. Configuring the OpenShift Container Platform inventory to use S3
Procedure
To configure the Ansible inventory for the registry to use the S3 bucket and IAM user:
[OSEv3:vars] # AWS Registry Configuration openshift_hosted_manage_registry=true openshift_hosted_registry_storage_kind=object openshift_hosted_registry_storage_provider=s3 openshift_hosted_registry_storage_s3_accesskey=AKIAJ6VLREDHATSPBUA 1 openshift_hosted_registry_storage_s3_secretkey=g/8PmTYDQVGssFWWFvfawHpDbZyGkjGNZhbWQpjH 2 openshift_hosted_registry_storage_s3_bucket=openshift-registry-storage 3 openshift_hosted_registry_storage_s3_region=us-east-1 4 openshift_hosted_registry_storage_s3_chunksize=26214400 openshift_hosted_registry_storage_s3_rootdirectory=/registry openshift_hosted_registry_storage_s3_encrypt=false openshift_hosted_registry_storage_s3_kmskeyid=aws_kms_key_id 5 openshift_hosted_registry_pullthrough=true openshift_hosted_registry_acceptschema2=true openshift_hosted_registry_enforcequota=true openshift_hosted_registry_replicas=3
- 1 1
- The access key for the IAM user. (Not required with IAM Roles in place)
- 2
- The secret key for the IAM user. (Not required with IAM Roles in place)
- 3
- The S3 storage bucket name.
- 4
- The region in which the bucket exists.
- 5
- The AWS Key Management Service (AWS KMS) key ID of the encryption key used to encrypt data in the cluster.
20.2.1.1.2. Manually configuring OpenShift Container Platform registry to use S3
To use Amazon Web Services (AWS) S3 object storage, edit the registry’s configuration file and mount to the registry pod.
Procedure
Export the current config.yml:
$ oc get secret registry-config \ -o jsonpath='{.data.config\.yml}' -n default | base64 -d \ >> config.yml.old
Create a new configuration file from the old config.yml:
$ cp config.yml.old config.yml
Edit the file to include the S3 parameters. Specify the accountname, accountkey, container, and realm in the
storage
section of a registry’s configuration file:storage: delete: enabled: true cache: blobdescriptor: inmemory s3: accesskey: AKIAJ6VLREDHATSPBUA 1 secretkey: g/8PmTYDQVGssFWWFvfawHpDbZyGkjGNZhbWQpjH 2 region: us-east-1 3 bucket: openshift-registry-storage 4 encrypt: False secure: true v4auth: true rootdirectory: /registry 5 chunksize: "26214400"
Delete the
registry-config
secret:$ oc delete secret registry-config -n default
Recreate the secret to reference the updated configuration file:
$ oc create secret generic registry-config \ --from-file=config.yml -n default
Redeploy the registry to read the updated configuration:
$ oc rollout latest docker-registry -n default
20.2.1.1.3. Verify the registry is using S3 storage
To verify if the registry is using Amazon S3 storage:
Procedure
After a successful registry deployment, the registry
deploymentconfig
describes registry-storage asemptydir
instead of AWS S3 but the configuration for the AWS S3 bucket resides in the secretdocker-config
. Thedocker-config
secret mounts toREGISTRY_CONFIGURATION_PATH
which provides all of the paramaters when using AWS S3 for the registry object storage.$ oc describe dc docker-registry -n default ... Environment: REGISTRY_HTTP_ADDR: :5000 REGISTRY_HTTP_NET: tcp REGISTRY_HTTP_SECRET: SPLR83SDsPaGbGuwSMDfnDwrDRvGf6YXl4h9JQrToQU= REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA: false REGISTRY_HTTP_TLS_KEY: /etc/secrets/registry.key OPENSHIFT_DEFAULT_REGISTRY: docker-registry.default.svc:5000 REGISTRY_CONFIGURATION_PATH: /etc/registry/config.yml REGISTRY_OPENSHIFT_SERVER_ADDR: docker-registry.default.svc:5000 REGISTRY_HTTP_TLS_CERTIFICATE: /etc/secrets/registry.crt Mounts: /etc/registry from docker-config (rw) /etc/secrets from registry-certificates (rw) /registry from registry-storage (rw) Volumes: registry-storage: Type: EmptyDir (a temporary directory that shares a pod's lifetime) 1 Medium: registry-certificates: Type: Secret (a volume populated by a Secret) SecretName: registry-certificates Optional: false docker-config: Type: Secret (a volume populated by a Secret) SecretName: registry-config Optional: false ....
- 1
- The temporary directory that shares a pod’s lifetime.
Ensure that the /registry mountpoint is empty:
$ oc exec \ $(oc get pod -l deploymentconfig=docker-registry \ -o=jsonpath='{.items[0].metadata.name}') -i -t -- ls -l /registry total 0
If it is empty, it is because the S3 configuration is defined in the
registry-config
secret:$ oc describe secret registry-config Name: registry-config Namespace: default Labels: <none> Annotations: <none> Type: Opaque Data ==== config.yml: 398 bytes
The installer creates a config.yml file with the desired configuration using the extended registry capabilities as seen in Storage in the installation documentation. To view the configuration file, including the
storage
section where the storage bucket configuration is stored:$ oc exec \ $(oc get pod -l deploymentconfig=docker-registry \ -o=jsonpath='{.items[0].metadata.name}') \ cat /etc/registry/config.yml version: 0.1 log: level: debug http: addr: :5000 storage: delete: enabled: true cache: blobdescriptor: inmemory s3: accesskey: AKIAJ6VLREDHATSPBUA secretkey: g/8PmTYDQVGssFWWFvfawHpDbZyGkjGNZhbWQpjH region: us-east-1 bucket: openshift-registry-storage encrypt: False secure: true v4auth: true rootdirectory: /registry chunksize: "26214400" auth: openshift: realm: openshift middleware: registry: - name: openshift repository: - name: openshift options: pullthrough: true acceptschema2: true enforcequota: true storage: - name: openshift
Alternatively, you can view the secret:
$ oc get secret registry-config -o jsonpath='{.data.config\.yml}' | base64 -d version: 0.1 log: level: debug http: addr: :5000 storage: delete: enabled: true cache: blobdescriptor: inmemory s3: accesskey: AKIAJ6VLREDHATSPBUA secretkey: g/8PmTYDQVGssFWWFvfawHpDbZyGkjGNZhbWQpjH region: us-east-1 bucket: openshift-registry-storage encrypt: False secure: true v4auth: true rootdirectory: /registry chunksize: "26214400" auth: openshift: realm: openshift middleware: registry: - name: openshift repository: - name: openshift options: pullthrough: true acceptschema2: true enforcequota: true storage: - name: openshift
If using an emptyDir
volume, the /registry
mountpoint looks like the following:
$ oc exec \ $(oc get pod -l deploymentconfig=docker-registry \ -o=jsonpath='{.items[0].metadata.name}') -i -t -- df -h /registry Filesystem Size Used Avail Use% Mounted on /dev/sdc 100G 226M 30G 1% /registry $ oc exec \ $(oc get pod -l deploymentconfig=docker-registry \ -o=jsonpath='{.items[0].metadata.name}') -i -t -- ls -l /registry total 0 drwxr-sr-x. 3 1000000000 1000000000 22 Jun 19 12:24 docker
20.3. Configuring AWS Variables
To set the required AWS variables, create a /etc/origin/cloudprovider/aws.conf file with the following contents on all of your OpenShift Container Platform hosts, both masters and nodes:
[Global]
Zone = us-east-1c 1
- 1
- This is the Availability Zone of your AWS Instance and where your EBS Volume resides; this information is obtained from the AWS Management Console.
20.4. Configuring OpenShift Container Platform for AWS
You can set the AWS configuration on OpenShift Container Platform in two ways:
20.4.1. Configuring OpenShift Container Platform for AWS with Ansible
During cluster installations, AWS can be configured using the openshift_cloudprovider_aws_access_key
, openshift_cloudprovider_aws_secret_key
, openshift_cloudprovider_kind
, openshift_clusterid
parameters, which are configurable in the inventory file.
Example AWS Configuration with Ansible
# Cloud Provider Configuration # # Note: You may make use of environment variables rather than store # sensitive configuration within the ansible inventory. # For example: #openshift_cloudprovider_aws_access_key="{{ lookup('env','AWS_ACCESS_KEY_ID') }}" #openshift_cloudprovider_aws_secret_key="{{ lookup('env','AWS_SECRET_ACCESS_KEY') }}" # #openshift_clusterid=unique_identifier_per_availablility_zone # # AWS (Using API Credentials) #openshift_cloudprovider_kind=aws #openshift_cloudprovider_aws_access_key=aws_access_key_id #openshift_cloudprovider_aws_secret_key=aws_secret_access_key # # AWS (Using IAM Profiles) #openshift_cloudprovider_kind=aws # Note: IAM roles must exist before launching the instances.
When Ansible configures AWS, it automatically makes the necessary changes to the following files:
- /etc/origin/cloudprovider/aws.conf
- /etc/origin/master/master-config.yaml
- /etc/origin/node/node-config.yaml
20.4.2. Manually Configuring OpenShift Container Platform Masters for AWS
Edit or create the master configuration file on all masters (/etc/origin/master/master-config.yaml by default) and update the contents of the apiServerArguments
and controllerArguments
sections:
kubernetesMasterConfig: ... apiServerArguments: cloud-provider: - "aws" cloud-config: - "/etc/origin/cloudprovider/aws.conf" controllerArguments: cloud-provider: - "aws" cloud-config: - "/etc/origin/cloudprovider/aws.conf"
Currently, the nodeName
must match the instance name in AWS in order for the cloud provider integration to work properly. The name must also be RFC1123 compliant.
When triggering a containerized installation, only the directories of /etc/origin and /var/lib/origin are mounted to the master and node container. Therefore, aws.conf should be in /etc/origin/ instead of /etc/.
20.4.3. Manually Configuring OpenShift Container Platform Nodes for AWS
Edit the appropriate node configuration map and update the contents of the kubeletArguments
section:
kubeletArguments: cloud-provider: - "aws" cloud-config: - "/etc/origin/cloudprovider/aws.conf"
When triggering a containerized installation, only the directories of /etc/origin and /var/lib/origin are mounted to the master and node container. Therefore, aws.conf should be in /etc/origin/ instead of /etc/.
20.4.4. Manually Setting Key-Value Access Pairs
Make sure the following environment variables are set in the /etc/origin/master/master.env file on masters and the /etc/sysconfig/atomic-openshift-node file on nodes:
AWS_ACCESS_KEY_ID=<key_ID> AWS_SECRET_ACCESS_KEY=<secret_key>
Access keys are obtained when setting up your AWS IAM user.
20.5. Applying Configuration Changes
Start or restart OpenShift Container Platform services on all master and node hosts to apply your configuration changes, see Restarting OpenShift Container Platform services:
# master-restart api # master-restart controllers # systemctl restart atomic-openshift-node
Kubernetes architecture expects reliable endpoints from cloud providers. When a cloud provider is down, the kubelet prevents OpenShift Container Platform from restarting. If the underlying cloud provider endpoints are not reliable, do not install a cluster that uses the cloud provider integration. Install the cluster as if it is a bare metal environment. It is not recommended to toggle cloud provider integration on or off in an installed cluster. However, if that scenario is unavoidable, then complete the following process.
Switching from not using a cloud provider to using a cloud provider produces an error message. Adding the cloud provider tries to delete the node because the node switches from using the hostname as the externalID
(which would have been the case when no cloud provider was being used) to using the cloud provider’s instance-id
(which is what the cloud provider specifies). To resolve this issue:
- Log in to the CLI as a cluster administrator.
Check and back up existing node labels:
$ oc describe node <node_name> | grep -Poz '(?s)Labels.*\n.*(?=Taints)'
Delete the nodes:
$ oc delete node <node_name>
On each node host, restart the OpenShift Container Platform service.
# systemctl restart atomic-openshift-node
- Add back any labels on each node that you previously had.
20.6. Labeling Clusters for AWS
If you configure AWS provider credentials, you must also ensure that all hosts are labeled.
To correctly identify which resources are associated with a cluster, tag resources with the key kubernetes.io/cluster/<clusterid>
, where:
-
<clusterid>
is a unique name for the cluster.
Set the corresponding value to owned
if the node belongs exclusively to the cluster or to shared
if it is a resource shared with other systems.
Tagging all resources with the kubernetes.io/cluster/<clusterid>,Value=(owned|shared)
tag avoids potential issues with multiple zones or multiple clusters.
See Pods and Services to learn more about labeling and tagging in OpenShift Container Platform.
20.6.1. Resources That Need Tags
There are four types of resources that need to be tagged:
- Instances
- Security Groups
- Load Balancers
- EBS Volumes
20.6.2. Tagging an Existing Cluster
A cluster uses the value of the kubernetes.io/cluster/<clusterid>,Value=(owned|shared)
tag to determine which resources belong to the AWS cluster. This means that all relevant resources must be labeled with the kubernetes.io/cluster/<clusterid>,Value=(owned|shared)
tag using the same values for that key. These resources include:
- All hosts.
- All relevant load balancers to be used in the AWS instances.
All EBS volumes. The EBS Volumes that need to be tagged can found with:
$ oc get pv -o json|jq '.items[].spec.awsElasticBlockStore.volumeID'
All relevant security groups to be used with the AWS instances.
NoteDo not tag all existing security groups with the
kubernetes.io/cluster/<name>,Value=<clusterid>
tag, or the Elastic Load Balancing (ELB) will not be able to create a load balancer.
After tagging any resources, restart the master services on the master and restart the node service on all nodes. See the Applying Configuration Section.
20.6.3. About Red Hat OpenShift Container Storage
Red Hat OpenShift Container Storage (RHOCS) is a provider of agnostic persistent storage for OpenShift Container Platform either in-house or in hybrid clouds. As a Red Hat storage solution, RHOCS is completely integrated with OpenShift Container Platform for deployment, management, and monitoring regardless if it is installed on OpenShift Container Platform (converged) or with OpenShift Container Platform (independent). OpenShift Container Storage is not limited to a single availability zone or node, which makes it likely to survive an outage. You can find complete instructions for using RHOCS in the RHOCS3.11 Deployment Guide.