Chapter 27. Configuring ingress cluster traffic

Procedure

1. Modify the default Ingress Controller by running the following command:

   $ oc edit ingresscontroller -n openshift-ingress-operator default

2. Edit the Ingress Controller to contain a namespaceSelector that excludes the routes with any of the finance, ops, and dev labels:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     namespaceSelector:
       matchExpressions:
         - key: type
           operator: NotIn
           values:
             - finance
             - ops
             - dev

Procedure

1. Edit the router-internal.yaml file:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: sharded
     namespace: openshift-ingress-operator
   spec:
     domain: <apps-sharded.basedomain.example.net> 1
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/worker: ""
     routeSelector:
       matchLabels:
         type: sharded

   1

   Specify a domain to be used by the Ingress Controller. This domain must be different from the default Ingress Controller domain.

2. Apply the Ingress Controller router-internal.yaml file:

   # oc apply -f router-internal.yaml

   The Ingress Controller selects routes in any namespace that have the label type: sharded.

3. Create a new route using the domain configured in the router-internal.yaml:

   $ oc expose svc <service-name> --hostname <route-name>.apps-sharded.basedomain.example.net

Procedure

1. Edit the router-internal.yaml file:

   $ cat router-internal.yaml

   Example output

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: sharded
     namespace: openshift-ingress-operator
   spec:
     domain: <apps-sharded.basedomain.example.net> 1
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/worker: ""
     namespaceSelector:
       matchLabels:
         type: sharded

   1

   Specify a domain to be used by the Ingress Controller. This domain must be different from the default Ingress Controller domain.

2. Apply the Ingress Controller router-internal.yaml file:

   $ oc apply -f router-internal.yaml

   The Ingress Controller selects routes in any namespace that is selected by the namespace selector that have the label type: sharded.

3. Create a new route using the domain configured in the router-internal.yaml:

   $ oc expose svc <service-name> --hostname <route-name>.apps-sharded.basedomain.example.net

Procedure

1. Create a project called hello-openshift by running the following command:

   $ oc new-project hello-openshift

2. Create a pod in the project by running the following command:

   $ oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json

3. Create a service called hello-openshift by running the following command:

   $ oc expose pod/hello-openshift

4. Create a route definition called hello-openshift-route.yaml:

   YAML definition of the created route for sharding:

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     labels:
       type: sharded 1
     name: hello-openshift-edge
     namespace: hello-openshift
   spec:
     subdomain: hello-openshift 2
     tls:
       termination: edge
     to:
       kind: Service
       name: hello-openshift

   1

   Both the label key and its corresponding label value must match the ones specified in the Ingress Controller. In this example, the Ingress Controller has the label key and value type: sharded.

   2

   The route will be exposed using the value of the subdomain field. When you specify the subdomain field, you must leave the hostname unset. If you specify both the host and subdomain fields, then the route will use the value of the host field, and ignore the subdomain field.

5. Use hello-openshift-route.yaml to create a route to the hello-openshift application by running the following command:

   $ oc -n hello-openshift create -f hello-openshift-route.yaml

Procedure

1. Create a custom resource (CR) file for the Ingress Controller:

   Example of a CR file that defines information for the IngressController object

   apiVersion: v1
   items:
   - apiVersion: operator.openshift.io/v1
     kind: IngressController
     metadata:
       name: <custom_ic_name> 1
       namespace: openshift-ingress-operator
     spec:
       replicas: 1
       domain: <custom_ic_domain_name> 2
       nodePlacement:
         nodeSelector:
           matchLabels:
             <key>: <value> 3
       namespaceSelector:
        matchLabels:
          <key>: <value> 4
       endpointPublishingStrategy:
         type: NodePortService
   # ...

   1

   Specify the a custom name for the IngressController CR.

   2

   The DNS name that the Ingress Controller services. As an example, the default ingresscontroller domain is apps.ipi-cluster.example.com, so you would specify the <custom_ic_domain_name> as nodeportsvc.ipi-cluster.example.com.

   3

   Specify the label for the nodes that include the custom Ingress Controller.

   4

   Specify the label for a set of namespaces. Substitute <key>:<value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value. For example: ingresscontroller: custom-ic.

2. Add a label to a node by using the oc label node command:

   $ oc label node <node_name> <key>=<value> 1

   1

   Where <value> must match the key-value pair specified in the nodePlacement section of your IngressController CR.

3. Create the IngressController object:

   $ oc create -f <ingress_controller_cr>.yaml

4. Find the port for the service created for the IngressController CR:

   $ oc get svc -n openshift-ingress

   Example output that shows port 80:32432/TCP for the router-nodeport-custom-ic3 service

   NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE
   router-internal-default      ClusterIP   172.30.195.74    <none>        80/TCP,443/TCP,1936/TCP                     223d
   router-nodeport-custom-ic3   NodePort    172.30.109.219   <none>        80:32432/TCP,443:31366/TCP,1936:30499/TCP   155m

5. To create a new project, enter the following command:

   $ oc new-project <project_name>

6. To label the new namespace, enter the following command:

   $ oc label namespace <project_name> <key>=<value> 1

   1

   Where <key>=<value> must match the value in the namespaceSelector section of your Ingress Controller CR.

7. Create a new application in your cluster:

   $ oc new-app --image=<image_name> 1

   1

   An example of <image_name> is quay.io/openshifttest/hello-openshift:multiarch.

8. Create a Route object for a service, so that the pod can use the service to expose the application external to the cluster.

   $ oc expose svc/<service_name> --hostname=<svc_name>-<project_name>.<custom_ic_domain_name> 1

   Note

   You must specify the domain name of your custom Ingress Controller in the --hostname argument. If you do not do this, the Ingress Operator uses the default Ingress Controller to serve all the routes for your cluster.

9. Check that the route has the Admitted status and that it includes metadata for the custom Ingress Controller:

   $ oc get route/hello-openshift -o json | jq '.status.ingress'

   Example output

   # ...
   {
     "conditions": [
       {
         "lastTransitionTime": "2024-05-17T18:25:41Z",
         "status": "True",
         "type": "Admitted"
       }
     ],
     [
       {
         "host": "hello-openshift.nodeportsvc.ipi-cluster.example.com",
         "routerCanonicalHostname": "router-nodeportsvc.nodeportsvc.ipi-cluster.example.com",
         "routerName": "nodeportsvc", "wildcardPolicy": "None"
       }
     ],
   }

10. Update the default IngressController CR to prevent the default Ingress Controller from managing the NodePort-type Service. The default Ingress Controller will continue to monitor all other cluster traffic.

    $ oc patch --type=merge -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"namespaceSelector":{"matchExpressions":[{"key":"<key>","operator":"NotIn","values":["<value>]}]}}}'

Verification

1. Verify that the DNS entry can route inside and outside of your cluster by entering the following command. The command outputs the IP address of the node that received the label from running the oc label node command earlier in the procedure.

   $ dig +short <svc_name>-<project_name>.<custom_ic_domain_name>

2. To verify that your cluster uses the IP addresses from external DNS servers for DNS resolution, check the connection of your cluster by entering the following command:

   $ curl <svc_name>-<project_name>.<custom_ic_domain_name>:<port> 1

   1 1

   Where <port> is the node port from the NodePort-type Service. Based on example output from the oc get svc -n openshift-ingress command, the 80:32432/TCP HTTP route means that 32432 is the node port.

   Output example

   Hello OpenShift!

Procedure

1. Using the oc annotate command, add the timeout to the route:

   $ oc annotate route <route_name> \
       --overwrite haproxy.router.openshift.io/timeout=<timeout><time_unit> 1

   1

   Supported time units are microseconds (us), milliseconds (ms), seconds (s), minutes (m), hours (h), or days (d).

   The following example sets a timeout of two seconds on a route named myroute:

   $ oc annotate route myroute --overwrite haproxy.router.openshift.io/timeout=2s

Procedure

1. Set an AWS connection idle timeout of five minutes for the default ingresscontroller by running the following command:

   $ oc -n openshift-ingress-operator patch ingresscontroller/default \
       --type=merge --patch='{"spec":{"endpointPublishingStrategy": \
       {"type":"LoadBalancerService", "loadBalancer": \
       {"scope":"External", "providerParameters":{"type":"AWS", "aws": \
       {"type":"Classic", "classicLoadBalancer": \
       {"connectionIdleTimeout":"5m"}}}}}}}'

2. Optional: Restore the default value of the timeout by running the following command:

   $ oc -n openshift-ingress-operator patch ingresscontroller/default \
       --type=merge --patch='{"spec":{"endpointPublishingStrategy": \
       {"loadBalancer":{"providerParameters":{"aws":{"classicLoadBalancer": \
       {"connectionIdleTimeout":null}}}}}}}'

Procedure

1. Modify the existing Ingress Controller that you want to switch to using an NLB. This example assumes that your default Ingress Controller has an External scope and no other customizations:

   Example ingresscontroller.yaml file

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: null
     name: default
     namespace: openshift-ingress-operator
   spec:
     endpointPublishingStrategy:
       loadBalancer:
         scope: External
         providerParameters:
           type: AWS
           aws:
             type: NLB
       type: LoadBalancerService

   Note

   If you do not specify a value for the spec.endpointPublishingStrategy.loadBalancer.providerParameters.aws.type field, the Ingress Controller uses the spec.loadBalancer.platform.aws.type value from the cluster Ingress configuration that was set during installation.

   Tip

   If your Ingress Controller has other customizations that you want to update, such as changing the domain, consider force replacing the Ingress Controller definition file instead.

2. Apply the changes to the Ingress Controller YAML file by running the command:

   $ oc apply -f ingresscontroller.yaml

   Expect several minutes of outages while the Ingress Controller updates.

Procedure

1. Modify the existing Ingress Controller that you want to switch to using a CLB. This example assumes that your default Ingress Controller has an External scope and no other customizations:

   Example ingresscontroller.yaml file

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: null
     name: default
     namespace: openshift-ingress-operator
   spec:
     endpointPublishingStrategy:
       loadBalancer:
         scope: External
         providerParameters:
           type: AWS
           aws:
             type: Classic
       type: LoadBalancerService

   Note

   If you do not specify a value for the spec.endpointPublishingStrategy.loadBalancer.providerParameters.aws.type field, the Ingress Controller uses the spec.loadBalancer.platform.aws.type value from the cluster Ingress configuration that was set during installation.

   Tip

   If your Ingress Controller has other customizations that you want to update, such as changing the domain, consider force replacing the Ingress Controller definition file instead.

2. Apply the changes to the Ingress Controller YAML file by running the command:

   $ oc apply -f ingresscontroller.yaml

   Expect several minutes of outages while the Ingress Controller updates.

Procedure

1. Create a file with a new default Ingress Controller. The following example assumes that your default Ingress Controller has an External scope and no other customizations:

   Example ingresscontroller.yml file

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: null
     name: default
     namespace: openshift-ingress-operator
   spec:
     endpointPublishingStrategy:
       loadBalancer:
         scope: External
         providerParameters:
           type: AWS
           aws:
             type: NLB
       type: LoadBalancerService

   If your default Ingress Controller has other customizations, ensure that you modify the file accordingly.

   Tip

   If your Ingress Controller has no other customizations and you are only updating the load balancer type, consider following the procedure detailed in "Switching the Ingress Controller from using a Classic Load Balancer to a Network Load Balancer".

2. Force replace the Ingress Controller YAML file:

   $ oc replace --force --wait -f ingresscontroller.yml

   Wait until the Ingress Controller is replaced. Expect several of minutes of outages.

1. Create the Ingress Controller manifest:

    $ cat ingresscontroller-aws-nlb.yaml

   Example output

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: $my_ingress_controller1
     namespace: openshift-ingress-operator
   spec:
     domain: $my_unique_ingress_domain2
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: External3
         providerParameters:
           type: AWS
           aws:
             type: NLB

   1

   Replace $my_ingress_controller with a unique name for the Ingress Controller.

   2

   Replace $my_unique_ingress_domain with a domain name that is unique among all Ingress Controllers in the cluster. This variable must be a subdomain of the DNS name <clustername>.<domain>.

   3

   You can replace External with Internal to use an internal NLB.

2. Create the resource in the cluster:

   $ oc create -f ingresscontroller-aws-nlb.yaml

1. Change to the directory that contains the installation program and create the manifests:

   $ ./openshift-install create manifests --dir <installation_directory> 1

   1

   For <installation_directory>, specify the name of the directory that contains the install-config.yaml file for your cluster.

2. Create a file that is named cluster-ingress-default-ingresscontroller.yaml in the <installation_directory>/manifests/ directory:

   $ touch <installation_directory>/manifests/cluster-ingress-default-ingresscontroller.yaml 1

   1

   For <installation_directory>, specify the directory name that contains the manifests/ directory for your cluster.

   After creating the file, several network configuration files are in the manifests/ directory, as shown:

   $ ls <installation_directory>/manifests/cluster-ingress-default-ingresscontroller.yaml

   Example output

   cluster-ingress-default-ingresscontroller.yaml

3. Open the cluster-ingress-default-ingresscontroller.yaml file in an editor and enter a custom resource (CR) that describes the Operator configuration you want:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: null
     name: default
     namespace: openshift-ingress-operator
   spec:
     endpointPublishingStrategy:
       loadBalancer:
         scope: External
         providerParameters:
           type: AWS
           aws:
             type: NLB
       type: LoadBalancerService

4. Save the cluster-ingress-default-ingresscontroller.yaml file and quit the text editor.
5. Optional: Back up the manifests/cluster-ingress-default-ingresscontroller.yaml file. The installation program deletes the manifests/ directory when creating the cluster.