Chapter 2. Configuring your firewall

Procedure

1. Allowlist the following container registry URLs for cluster installation and upgrades:

   Expand

   URL	Port	Function
   registry.redhat.io	443	Provides core container images
   access.redhat.com	443	Hosts a signature store that a container client requires for verifying images pulled from registry.access.redhat.com. In a firewall environment, ensure that this resource is on the allowlist.
   registry.access.redhat.com	443	Hosts all the container images that are stored on the Red Hat Ecosystem Catalog, including core container images.
   quay.io	443	Provides core container images
   cdn.quay.io	443	Provides core container images
   cdn01.quay.io	443	Provides core container images
   cdn02.quay.io	443	Provides core container images
   cdn03.quay.io	443	Provides core container images
   cdn04.quay.io	443	Provides core container images
   cdn05.quay.io	443	Provides core container images
   cdn06.quay.io	443	Provides core container images
   icr.io	443	Provides IBM Cloud Pak container images. This domain is only required if you use IBM Cloud Paks.
   cp.icr.io	443	Provides IBM Cloud Pak container images. This domain is only required if you use IBM Cloud Paks.

   Show more
   • You can use the wildcard *.quay.io instead of cdn.quay.io and cdn0[1-6].quay.io in your allowlist.
   • You can use the wildcard *.access.redhat.com to simplify the configuration and ensure that all subdomains, including registry.access.redhat.com, are allowed.
   • When adding a site such as quay.io to your allowlist, do not add a wildcard entry such as *.quay.io to your denylist. In most cases, image registries use a content delivery network (CDN) to serve images. If a firewall blocks access, image downloads are denied when the initial download request redirects to a hostname such as cdn01.quay.io.

2. Allowlist the following URLs to enable cluster access, authentication, and updates:

   Expand

   URL	Port	Function
   *.apps.<cluster_name>.<base_domain>	443	Allowlist these URLs to enable cluster access, authentication, and updates.
   api.openshift.com	443	API endpoint for cluster tokens and update checks.
   console.redhat.com	443	Authentication service for cluster tokens.
   sso.redhat.com	443	The https://console.redhat.com site uses authentication from sso.redhat.com

   Show more

   For egress traffic, Operators require route access to perform health checks to establish a connection for reaching endpoints. The authentication and web console Operators connect to two routes to verify functionality. Cluster administrators who do not want to allow *.apps.<cluster_name>.<base_domain>, must allow the following routes:

   • oauth-openshift.apps.<cluster_name>.<base_domain>
   • canary-openshift-ingress-canary.apps.<cluster_name>.<base_domain>
   • console-openshift-console.apps.<cluster_name>.<base_domain>, or the hostname that is specified in the spec.route.hostname field of the consoles.operator/cluster object if the field is not empty.

3. Allowlist the following registry URLs that host related artifacts for cluster installation and upgrades, such as installation content, release images, and client tools:

   Expand

   URL	Port	Function
   mirror.openshift.com	443	Required to access mirrored installation content and images. This site is also a source of release image signatures, although the Cluster Version Operator needs only a single functioning source.
   quayio-production-s3.s3.amazonaws.com	443	Required to access Quay image content in AWS.
   rhcos.mirror.openshift.com	443	Required to download Red Hat Enterprise Linux CoreOS (RHCOS) images.
   storage.googleapis.com/openshift-release	443	A source of release image signatures, although the Cluster Version Operator needs only a single functioning source.

   Show more
4. Set your firewall’s allowlist to include any site that provides resources for a language or framework that your builds require.

5. If you do not disable Telemetry, you must grant access to the following URLs to access Telemetry and Red Hat Lightspeed:

   Expand

   URL	Port	Function
   cert-api.access.redhat.com	443	Required for Telemetry
   api.access.redhat.com	443	Required for Telemetry
   infogw.api.openshift.com	443	Required for Telemetry
   console.redhat.com	443	Required for Telemetry and for insights-operator

   Show more

6. If you use Alibaba Cloud, Amazon Web Services (AWS), Microsoft Azure, or Google Cloud to host your cluster, you must grant access to the URLs that offer the cloud provider API and DNS for that cloud:

   Expand

   Cloud	URL	Port	Function
   Alibaba	*.aliyuncs.com	443	Required to access Alibaba Cloud services and resources. Review the Alibaba endpoints_config.go file to find the exact endpoints to allow for the regions that you use.
   AWS	aws.amazon.com	443	Used to install and manage clusters in an AWS environment.
   	*.amazonaws.com Alternatively, if you choose to not use a wildcard for AWS APIs, you must include the following URLs in your allowlist:	443	Required to access AWS services and resources. Review the AWS Service Endpoints in the AWS documentation to find the exact endpoints to allow for the regions that you use.
   	ec2.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	events.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	iam.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	route53.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.dualstack.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	sts.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	sts.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	tagging.us-east-1.amazonaws.com	443	Used to install and manage clusters in an AWS environment. This endpoint is always us-east-1, regardless of the region the cluster is deployed in.
   	ec2.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	elasticloadbalancing.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	servicequotas.<aws_region>.amazonaws.com	443	Required. Used to confirm quotas for deploying the service.
   	tagging.<aws_region>.amazonaws.com	443	Allows the assignment of metadata about AWS resources in the form of tags.
   	*.cloudfront.net	443	Used to provide access to CloudFront. If you use the AWS Security Token Service (STS) and the private S3 bucket, you must provide access to CloudFront.
   GCP	*.googleapis.com	443	Required to access Google Cloud services and resources. Review Cloud Endpoints in the Google Cloud documentation to find the endpoints to allow for your APIs.
   	accounts.google.com	443	Required to access your Google Cloud account.
   Microsoft Azure	management.azure.com	443	Required to access Microsoft Azure services and resources. Review the Microsoft Azure REST API reference in the Microsoft Azure documentation to find the endpoints to allow for your APIs.
   	*.blob.core.windows.net	443	Required to download Ignition files.
   	login.microsoftonline.com	443	Required to access Microsoft Azure services and resources. Review the Azure REST API reference in the Microsoft Azure documentation to find the endpoints to allow for your APIs.

   Show more

7. Allowlist the following URL for optional third-party content:

   Expand

   URL	Port	Function
   registry.connect.redhat.com	443	Required for all third-party images and certified operators.

   Show more

8. If you use a default Red Hat Network Time Protocol (NTP) server, allow the following URLs. NTP operates on User Datagram Protocol (UDP) port 123, so this port must be opened on the firewall.

   Expand

   URL	Port	Function
   1.rhel.pool.ntp.org	123	Provides NTP services for time synchronization.
   2.rhel.pool.ntp.org	123	Provides NTP services for time synchronization.
   3.rhel.pool.ntp.org	123	Provides NTP services for time synchronization.

   Show more

Procedure

1. Log in to the Red Hat Ecosystem Catalog registry by running the following command and entering your credentials:

   $ podman login registry.redhat.io

2. Extract the commatrix binary from the plugin image by running the following commands:

   $ podman create --name oc-commatrix registry.redhat.io/openshift-kni/commatrix:v4.22
   $ podman cp oc-commatrix:/oc-commatrix .
   $ podman rm oc-commatrix

3. Move the extracted binary to a directory in your system PATH, such as /usr/local/bin/, by running the following command:

   sudo mv oc-commatrix /usr/local/bin/

Procedure

1. Generate network flow data by running the following command:

   $ oc commatrix generate

   Note

   By default, the plugin generates the network flow data in CSV format in a communication-matrix directory in your current working directory.

Procedure

1. Generate firewall rules in Butane format by running the following command:

   $ oc commatrix generate --format butane

   By default, the plugin writes the output files to the communication-matrix directory in your current working directory.

   The plugin generates one Butane config file per node pool, named butane-<pool_name>.yaml, and a node-disruption-policy.yaml patch file, for example:

   communication-matrix/
   ├── butane-master.yaml
   ├── butane-worker.yaml
   └── node-disruption-policy.yaml

2. Review the generated Butane config files by running the following command:

   $ cat communication-matrix/butane-<pool_name>.yaml

   See the following example of the butane-master.yaml file.

   Note

   You can adjust the generated firewall rules in your YAML file to suit your network environment.

   variant: openshift
   version: 4.22.0
   metadata:
     name: 98-nftables-commatrix-master
     labels:
       machineconfiguration.openshift.io/role: master
   systemd:
     units:
       - name: "nftables.service"
         enabled: true
         contents: |
           # ... systemd unit configuration ...
   storage:
     files:
       - path: /etc/sysconfig/nftables.conf
         mode: 0600
         overwrite: true
         contents:
           inline: |
             table inet openshift_filter {
                 chain OPENSHIFT {
                     type filter hook input priority 1; policy accept;
   
                     # Allow loopback traffic
                     iif lo accept
   
                     # Allow established and related traffic
                     ct state established,related accept
   
                     # Allow ICMP on ipv4
                     ip protocol icmp accept
                     ip6 nexthdr ipv6-icmp accept
   
                     # Allow specific TCP and UDP ports
                     tcp dport { 22, 6443, 9100, 10250, 30000-60999 } accept
                     udp dport { 6081, 30000-60999 } accept
   
                     # Drop broadcast traffic with rate-limited logging
                     ip daddr 255.255.255.255 jump { limit rate 1/minute log prefix "firewall "; drop; }
   
                     # Rate-limited logging and default drop
                     jump { limit rate 1/minute log prefix "firewall "; drop; }
                 }
             }

3. Review the generated NodeDisruptionPolicy patch by running the following command:

   $ cat communication-matrix/node-disruption-policy.yaml

4. Check whether your cluster already defines NodeDisruptionPolicy entries by running the following command:

   $ oc get -o yaml machineconfiguration cluster

5. Apply the NodeDisruptionPolicy patch:

   1. If the MachineConfiguration resource does not define any nodeDisruptionPolicy entries, run the following command:

      $ oc patch machineconfiguration cluster --type=merge --patch-file=communication-matrix/node-disruption-policy.yaml

   2. If the MachineConfiguration resource already contains nodeDisruptionPolicy entries, manually add the entries from node-disruption-policy.yaml to the existing .spec.nodeDisruptionPolicy.units and .spec.nodeDisruptionPolicy.files lists by running the following command:

      $ oc edit machineconfiguration cluster

6. Convert each Butane config to a MachineConfig resource by running the following command:

   $ butane --strict -o mc-<pool_name>.yaml communication-matrix/butane-<pool_name>.yaml

   • <pool_name> is the name of the target node pool.

7. Apply the MachineConfig resources by running the following command for each node pool:

   Important

   You must apply the NodeDisruptionPolicy patch before applying MachineConfig resources. If you apply MachineConfig resources without the NodeDisruptionPolicy in place, the Machine Config Operator triggers a full node reboot.

   $ oc apply -f mc-<pool_name>.yaml

   The plugin generates MachineConfig resources with the naming pattern 98-nftables-commatrix-<pool_name>.

Verification

1. Open a debug shell on a target node by running the following commands:

   $ oc debug node/<node_name>
   sh-5.1# chroot /host

   • <node_name> is the name of a cluster node.
   • chroot /host accesses the host filesystem.

2. Verify that the nftables rules are active on a node by running the following command:

   sh-5.1# nft list ruleset

   ...
   table inet openshift_filter {
   	chain OPENSHIFT {
   		type filter hook input priority filter + 1; policy accept;
   		iif "lo" accept
   		ct state established,related accept
   		ip protocol icmp accept
   		ip6 nexthdr ipv6-icmp accept
   		tcp dport { 22, 111, 2379-2380, 6080, 6443, 9001, 9099-9100, 9103-9105, 9107-9108, 9192, 9258, 9443, 9637, 9978-9980, 10250, 10256-10259, 17697, 22623-22624, 30000-60999 } accept
   		udp dport { 111, 6081, 30000-60999 } accept
   		ip daddr 255.255.255.255 jump {
   			limit rate 1/minute burst 5 packets log prefix "firewall "
   			drop
   		}
   		jump {
   			limit rate 1/minute burst 5 packets log prefix "firewall "
   			drop
   		}
   	}
   }
   ...

3. Verify that denied traffic is logged with rate limiting by checking the node journal:

   $ oc debug node/<node_name> -- chroot /host journalctl -k --grep firewall

   Denied packets are logged, but log entries are rate-limited to one per minute with an initial burst of five entries.

Procedure

1. Identify the MachineConfig resources created by the commatrix plugin by running the following command:

   $ oc get machineconfig | grep nftables

2. Delete the MachineConfig resource for each node pool by running the following command:

   $ oc delete machineconfig 98-nftables-commatrix-<pool_name>

3. Wait for all MachineConfigPool resources to return to the UPDATED state:

   $ oc get mcp

   Example output showing pools in UPDATED state

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6   True      False      False      1              1                   1                     0                      160d
   worker   rendered-worker-a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6   True      False      False      0              0                   0                     0                      160d

4. Open a debug shell on a target node by running the following commands:

   $ oc debug node/<node_name>
   sh-5.1# chroot /host

   • <node_name> is the name of a cluster node.
   • chroot /host accesses the host filesystem.

5. Verify that the custom nftables rules were removed by running the following command:

   sh-5.1# nft list ruleset 2>&1 | grep -q openshift_filter || echo "Custom rules removed"

6. Remove the related nftables rules from the NodeDisruptionPolicy entries by editing the MachineConfiguration resource:

   $ oc edit machineconfiguration cluster

   Remove the nftables.service entry from .spec.nodeDisruptionPolicy.units and the /etc/sysconfig/nftables.conf entry from .spec.nodeDisruptionPolicy.files.