Chapter 4. Configuring a cluster-wide proxy
If you are using an existing Virtual Private Cloud (VPC), you can configure a cluster-wide proxy during a Red Hat OpenShift Service on AWS (ROSA) cluster installation or after the cluster is installed. When you enable a proxy, the core cluster components are denied direct access to the internet, but the proxy does not affect user workloads.
Only cluster system egress traffic is proxied, including calls to the cloud provider API.
If you use a cluster-wide proxy, you are responsible for maintaining the availability of the proxy to the cluster. If the proxy becomes unavailable, then it might impact the health and supportability of the cluster.
4.1. Prerequisites for configuring a cluster-wide proxy
To configure a cluster-wide proxy, you must meet the following requirements. These requirements are valid when you configure a proxy during installation or postinstallation.
General requirements
- You are the cluster owner.
- Your account has sufficient privileges.
- You have an existing Virtual Private Cloud (VPC) for your cluster.
- The proxy can access the VPC for the cluster and the private subnets of the VPC. The proxy is also accessible from the VPC for the cluster and from the private subnets of the VPC.
You have added the following endpoints to your VPC endpoint:
-
ec2.<aws_region>.amazonaws.com
-
elasticloadbalancing.<aws_region>.amazonaws.com
s3.<aws_region>.amazonaws.com
These endpoints are required to complete requests from the nodes to the AWS EC2 API. Because the proxy works at the container level and not at the node level, you must route these requests to the AWS EC2 API through the AWS private network. Adding the public IP address of the EC2 API to your allowlist in your proxy server is not enough.
ImportantWhen using a cluster-wide proxy, you must configure the
s3.<aws_region>.amazonaws.com
endpoint as typeGateway
.
-
Network requirements
If your proxy re-encrypts egress traffic, you must create exclusions to the domain and port combinations. The following table offers guidance into these exceptions.
Your proxy must exclude re-encrypting the following OpenShift URLs:
Address Protocol/Port Function observatorium-mst.api.openshift.com
https/443
Required. Used for Managed OpenShift-specific telemetry.
sso.redhat.com
https/443
The https://cloud.redhat.com/openshift site uses authentication from sso.redhat.com to download the cluster pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, and chargeback reporting.
Your proxy must exclude re-encrypting the following site reliability engineering (SRE) and management URLs:
Address Protocol/Port Function *.osdsecuritylogs.splunkcloud.com
OR
inputs1.osdsecuritylogs.splunkcloud.com
inputs2.osdsecuritylogs.splunkcloud.com
inputs4.osdsecuritylogs.splunkcloud.com
inputs5.osdsecuritylogs.splunkcloud.com
inputs6.osdsecuritylogs.splunkcloud.com
inputs7.osdsecuritylogs.splunkcloud.com
inputs8.osdsecuritylogs.splunkcloud.com
inputs9.osdsecuritylogs.splunkcloud.com
inputs10.osdsecuritylogs.splunkcloud.com
inputs11.osdsecuritylogs.splunkcloud.com
inputs12.osdsecuritylogs.splunkcloud.com
inputs13.osdsecuritylogs.splunkcloud.com
inputs14.osdsecuritylogs.splunkcloud.com
inputs15.osdsecuritylogs.splunkcloud.com
tcp/9997
Used by the splunk-forwarder-operator as a log forwarding endpoint to be used by Red Hat SRE for log-based alerting.
http-inputs-osdsecuritylogs.splunkcloud.com
https/443
Used by the splunk-forwarder-operator as a log forwarding endpoint to be used by Red Hat SRE for log-based alerting.
Additional resources
- For the installation prerequisites for ROSA clusters that use the AWS Security Token Service (STS), see AWS prerequisites for ROSA with STS.
- For the installation prerequisites for ROSA clusters that do not use STS, see AWS prerequisites for ROSA.
4.2. Responsibilities for additional trust bundles
If you supply an additional trust bundle, you are responsible for the following requirements:
- Ensuring that the contents of the additional trust bundle are valid
- Ensuring that the certificates, including intermediary certificates, contained in the additional trust bundle have not expired
- Tracking the expiry and performing any necessary renewals for certificates contained in the additional trust bundle
- Updating the cluster configuration with the updated additional trust bundle
4.3. Configuring a proxy during installation
You can configure an HTTP or HTTPS proxy when you install a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC). You can configure the proxy during installation by using Red Hat OpenShift Cluster Manager or the ROSA CLI (rosa
).
4.3.1. Configuring a proxy during installation using OpenShift Cluster Manager
If you are installing a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC), you can use Red Hat OpenShift Cluster Manager to enable a cluster-wide HTTP or HTTPS proxy during installation.
Prior to the installation, you must verify that the proxy is accessible from the VPC that the cluster is being installed into. The proxy must also be accessible from the private subnets of the VPC.
For detailed steps to configure a cluster-wide proxy during installation by using OpenShift Cluster Manager, see Creating a cluster with customizations by using OpenShift Cluster Manager.
4.3.2. Configuring a proxy during installation using the CLI
If you are installing a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC), you can use the ROSA CLI (rosa
) to enable a cluster-wide HTTP or HTTPS proxy during installation.
The following procedure provides details about the ROSA CLI (rosa
) arguments that are used to configure a cluster-wide proxy during installation. For general installation steps using the ROSA CLI, see Creating a cluster with customizations using the CLI.
Prerequisites
- You have verified that the proxy is accessible from the VPC that the cluster is being installed into. The proxy must also be accessible from the private subnets of the VPC.
Procedure
Specify a proxy configuration when you create your cluster:
$ rosa create cluster \ <other_arguments_here> \ --additional-trust-bundle-file <path_to_ca_bundle_file> \ 1 2 3 --http-proxy http://<username>:<password>@<ip>:<port> \ 4 5 --https-proxy https://<username>:<password>@<ip>:<port> \ 6 7 --no-proxy example.com 8
- 1 4 6
- The
additional-trust-bundle-file
,http-proxy
, andhttps-proxy
arguments are all optional. - 2
- The
additional-trust-bundle-file
argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additional-trust-bundle-file argument is required for users who use a TLS-inspecting proxy unless the identity certificate for the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle. This applies regardless of whether the proxy is transparent or requires explicit configuration using the http-proxy and https-proxy arguments. - 3 5 7
- The
http-proxy
andhttps-proxy
arguments must point to a valid URL. - 8
- A comma-separated list of destination domain names, IP addresses, or network CIDRs to exclude proxying.
Preface a domain with
.
to match subdomains only. For example,.y.com
matchesx.y.com
, but noty.com
. Use*
to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by thenetworking.machineNetwork[].cidr
field from the installation configuration, you must add them to this list to prevent connection issues.This field is ignored if neither the
httpProxy
orhttpsProxy
fields are set.
4.4. Configuring a proxy after installation
You can configure an HTTP or HTTPS proxy after you install a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC). You can configure the proxy after installation by using Red Hat OpenShift Cluster Manager or the ROSA CLI (rosa
).
4.4.1. Configuring a proxy after installation using OpenShift Cluster Manager
You can use Red Hat OpenShift Cluster Manager to add a cluster-wide proxy configuration to an existing Red Hat OpenShift Service on AWS cluster in a Virtual Private Cloud (VPC).
You can also use OpenShift Cluster Manager to update an existing cluster-wide proxy configuration. For example, you might need to update the network address for the proxy or replace the additional trust bundle if any of the certificate authorities for the proxy expire.
The cluster applies the proxy configuration to the control plane and compute nodes. While applying the configuration, each cluster node is temporarily placed in an unschedulable state and drained of its workloads. Each node is restarted as part of the process.
Prerequisites
- You have an Red Hat OpenShift Service on AWS cluster .
- Your cluster is deployed in a VPC.
Procedure
- Navigate to OpenShift Cluster Manager and select your cluster.
- Under the Virtual Private Cloud (VPC) section on the Networking page, click Edit cluster-wide proxy.
On the Edit cluster-wide proxy page, provide your proxy configuration details:
Enter a value in at least one of the following fields:
- Specify a valid HTTP proxy URL.
- Specify a valid HTTPS proxy URL.
-
In the Additional trust bundle field, provide a PEM encoded X.509 certificate bundle. If you are replacing an existing trust bundle file, select Replace file to view the field. The bundle is added to the trusted certificate store for the cluster nodes. An additional trust bundle file is required if you use a TLS-inspecting proxy unless the identity certificate for the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle. This requirement applies regardless of whether the proxy is transparent or requires explicit configuration using the
http-proxy
andhttps-proxy
arguments.
- Click Confirm.
Verification
- Under the Virtual Private Cloud (VPC) section on the Networking page, verify that the proxy configuration for your cluster is as expected.
4.4.2. Configuring a proxy after installation using the CLI
You can use the Red Hat OpenShift Service on AWS (ROSA) CLI (rosa
) to add a cluster-wide proxy configuration to an existing ROSA cluster in a Virtual Private Cloud (VPC).
You can also use rosa
to update an existing cluster-wide proxy configuration. For example, you might need to update the network address for the proxy or replace the additional trust bundle if any of the certificate authorities for the proxy expire.
The cluster applies the proxy configuration to the control plane and compute nodes. While applying the configuration, each cluster node is temporarily placed in an unschedulable state and drained of its workloads. Each node is restarted as part of the process.
Prerequisites
-
You have installed and configured the latest ROSA (
rosa
) and OpenShift (oc
) CLIs on your installation host. - You have a ROSA cluster that is deployed in a VPC.
Procedure
Edit the cluster configuration to add or update the cluster-wide proxy details:
$ rosa edit cluster \ --cluster $CLUSTER_NAME \ --additional-trust-bundle-file <path_to_ca_bundle_file> \ 1 2 3 --http-proxy http://<username>:<password>@<ip>:<port> \ 4 5 --https-proxy https://<username>:<password>@<ip>:<port> \ 6 7 --no-proxy example.com 8
- 1 4 6
- The
additional-trust-bundle-file
,http-proxy
, andhttps-proxy
arguments are all optional. - 2
- The
additional-trust-bundle-file
argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additional-trust-bundle-file argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additional-trust-bundle-file argument is required for users who use a TLS-inspecting proxy unless the identity certificate for the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle. This applies regardless of whether the proxy is transparent or requires explicit configuration using thehttp-proxy
andhttps-proxy
arguments.NoteYou should not attempt to change the proxy or additional trust bundle configuration on the cluster directly. These changes must be applied by using the ROSA CLI (
rosa
) or Red Hat OpenShift Cluster Manager. Any changes that are made directly to the cluster will be reverted automatically. - 3 5 7
- The
http-proxy
andhttps-proxy
arguments must point to a valid URL. - 8
- A comma-separated list of destination domain names, IP addresses, or network CIDRs to exclude proxying.
Preface a domain with
.
to match subdomains only. For example,.y.com
matchesx.y.com
, but noty.com
. Use*
to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by thenetworking.machineNetwork[].cidr
field from the installation configuration, you must add them to this list to prevent connection issues.This field is ignored if neither the
httpProxy
orhttpsProxy
fields are set.
Verification
List the status of the machine config pools and verify that they are updated:
$ oc get machineconfigpools
Example output
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-d9a03f612a432095dcde6dcf44597d90 True False False 3 3 3 0 31h worker rendered-worker-f6827a4efe21e155c25c21b43c46f65e True False False 6 6 6 0 31h
Display the proxy configuration for your cluster and verify that the details are as expected:
$ oc get proxy cluster -o yaml
Example output
apiVersion: config.openshift.io/v1 kind: Proxy spec: httpProxy: http://proxy.host.domain:<port> httpsProxy: https://proxy.host.domain:<port> <...more...> status: httpProxy: http://proxy.host.domain:<port> httpsProxy: https://proxy.host.domain:<port> <...more...>
4.5. Removing a cluster-wide proxy
You can remove your cluster-wide proxy by using the ROSA CLI. After removing the cluster, you should also remove any trust bundles that are added to the cluster.
4.5.1. Removing the cluster-wide proxy using CLI
You must use the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa
, to remove the proxy’s address from your cluster.
Prerequisites
- You must have cluster administrator privileges.
-
You have installed the ROSA CLI (
rosa
).
Procedure
Use the
rosa edit
command to modify the proxy. You must pass empty strings to the--http-proxy
and--https-proxy
arguments to clear the proxy from the cluster:$ rosa edit cluster -c <cluster_name> --http-proxy "" --https-proxy ""
NoteWhile your proxy might only use one of the proxy arguments, the empty fields are ignored, so passing empty strings to both the
--http-proxy
and--https-proxy
arguments do not cause any issues.Example Output
I: Updated cluster <cluster_name>
Verification
You can verify that the proxy has been removed from the cluster by using the
rosa describe
command:$ rosa describe cluster -c <cluster_name>
Before removal, the proxy IP displays in a proxy section:
Name: <cluster_name> ID: <cluster_internal_id> External ID: <cluster_external_id> OpenShift Version: 4.0 Channel Group: stable DNS: <dns> AWS Account: <aws_account_id> API URL: <api_url> Console URL: <console_url> Region: us-east-1 Multi-AZ: false Nodes: - Control plane: 3 - Infra: 2 - Compute: 2 Network: - Type: OVNKubernetes - Service CIDR: <service_cidr> - Machine CIDR: <machine_cidr> - Pod CIDR: <pod_cidr> - Host Prefix: <host_prefix> Proxy: - HTTPProxy: <proxy_url> Additional trust bundle: REDACTED
After removing the proxy, the proxy section is removed:
Name: <cluster_name> ID: <cluster_internal_id> External ID: <cluster_external_id> OpenShift Version: 4.0 Channel Group: stable DNS: <dns> AWS Account: <aws_account_id> API URL: <api_url> Console URL: <console_url> Region: us-east-1 Multi-AZ: false Nodes: - Control plane: 3 - Infra: 2 - Compute: 2 Network: - Type: OVNKubernetes - Service CIDR: <service_cidr> - Machine CIDR: <machine_cidr> - Pod CIDR: <pod_cidr> - Host Prefix: <host_prefix> Additional trust bundle: REDACTED
4.5.2. Removing certificate authorities on a Red Hat OpenShift Service on AWS cluster
You can remove certificate authorities (CA) from your cluster with the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa
.
Prerequisites
- You must have cluster administrator privileges.
-
You have installed the ROSA CLI (
rosa
). - Your cluster has certificate authorities added.
Procedure
Use the
rosa edit
command to modify the CA trust bundle. You must pass empty strings to the--additional-trust-bundle-file
argument to clear the trust bundle from the cluster:$ rosa edit cluster -c <cluster_name> --additional-trust-bundle-file ""
Example Output
I: Updated cluster <cluster_name>
Verification
You can verify that the trust bundle has been removed from the cluster by using the
rosa describe
command:$ rosa describe cluster -c <cluster_name>
Before removal, the Additional trust bundle section appears, redacting its value for security purposes:
Name: <cluster_name> ID: <cluster_internal_id> External ID: <cluster_external_id> OpenShift Version: 4.0 Channel Group: stable DNS: <dns> AWS Account: <aws_account_id> API URL: <api_url> Console URL: <console_url> Region: us-east-1 Multi-AZ: false Nodes: - Control plane: 3 - Infra: 2 - Compute: 2 Network: - Type: OVNKubernetes - Service CIDR: <service_cidr> - Machine CIDR: <machine_cidr> - Pod CIDR: <pod_cidr> - Host Prefix: <host_prefix> Proxy: - HTTPProxy: <proxy_url> Additional trust bundle: REDACTED
After removing the proxy, the Additional trust bundle section is removed:
Name: <cluster_name> ID: <cluster_internal_id> External ID: <cluster_external_id> OpenShift Version: 4.0 Channel Group: stable DNS: <dns> AWS Account: <aws_account_id> API URL: <api_url> Console URL: <console_url> Region: us-east-1 Multi-AZ: false Nodes: - Control plane: 3 - Infra: 2 - Compute: 2 Network: - Type: OVNKubernetes - Service CIDR: <service_cidr> - Machine CIDR: <machine_cidr> - Pod CIDR: <pod_cidr> - Host Prefix: <host_prefix> Proxy: - HTTPProxy: <proxy_url>