Chapter 6. Ingress Operator in OpenShift Container Platform

1. Create a Secret resource containing the custom certificate in the

   openshift-ingress

   namespace using the

   tls.crt

   and

   tls.key

   files.

   $ oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key

2. Update the IngressController CR to reference the new certificate secret:

   $ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
     --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'

3. Verify the update was effective:

   $ echo Q |\
     openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null |\
     openssl x509 -noout -subject -issuer -enddate

   where:

   <domain>

   Specifies the base domain name for your cluster.

   Example output

   subject=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = *.apps.example.com
   issuer=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = example.com
   notAfter=May 10 08:32:45 2022 GM

   Tip

   You can alternatively apply the following YAML to set a custom default certificate:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     defaultCertificate:
       name: custom-certs-default

   The certificate secret name should match the value used to update the CR.

Procedure

1. View the current number of available replicas for the default

   IngressController

   :

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   2

2. Scale the default

   IngressController

   to the desired number of replicas using the

   oc patch

   command. The following example scales the default

   IngressController

   to 3 replicas:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"replicas": 3}}' --type=merge

   Example output

   ingresscontroller.operator.openshift.io/default patched

3. Verify that the default

   IngressController

   scaled to the number of replicas that you specified:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   3

   Tip

   You can alternatively apply the following YAML to scale an Ingress Controller to three replicas:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: 3               

   1

   1

   If you need a different amount of replicas, change the replicas value.

Procedure

1. Create an

   IngressController

   custom resource (CR) in a file named

   <name>-ingress-controller.yaml

   , such as in the following example:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: <name> 

   1

   
   spec:
     domain: <domain> 

   2

   
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal 

   3

   1

   Replace <name> with a name for the IngressController object.

   2

   Specify the domain for the application published by the controller.

   3

   Specify a value of Internal to use an internal load balancer.

2. Create the Ingress Controller defined in the previous step by running the following command:

   $ oc create -f <name>-ingress-controller.yaml 

   1

   1

   Replace <name> with the name of the IngressController object.

3. Optional: Confirm that the Ingress Controller was created by running the following command:

   $ oc --all-namespaces=true get ingresscontrollers

Procedure

1. Configure the Ingress Controller resource to allow global access.

   Note

   You can also create an Ingress Controller and specify the global access option.

   1. Configure the Ingress Controller resource:

      $ oc -n openshift-ingress-operator edit ingresscontroller/default

   2. Edit the YAML file:

      Sample clientAccess configuration to Global

        spec:
          endpointPublishingStrategy:
            loadBalancer:
              providerParameters:
                gcp:
                  clientAccess: Global 

      1

      
                type: GCP
              scope: Internal
            type: LoadBalancerService

      1

      Set gcp.clientAccess to Global.

   3. Save the file to apply the changes.

2. Run the following command to verify that the service allows global access:

   $ oc -n openshift-ingress edit svc/router-default -o yaml

   The output shows that global access is enabled for GCP with the annotation,

   networking.gke.io/internal-load-balancer-allow-global-access

   .

Procedure

1. Configure the

   default

   Ingress Controller for your cluster to be internal by deleting and recreating it.

   $ oc replace --force --wait --filename - <<EOF
   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: default
   spec:
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal
   EOF

Procedure

1. Configure the wildcard policy.

   1. Use the following command to edit the

      IngressController

      resource:

      $ oc edit IngressController

   2. Under

      spec

      , set the

      wildcardPolicy

      field to

      WildcardsDisallowed

      or

      WildcardsAllowed

      :

      spec:
        routeAdmission:
          wildcardPolicy: WildcardsDisallowed # or WildcardsAllowed

Procedure

1. Configure the

   HTTPHeaders

   field for the Ingress Controller.

   1. Use the following command to edit the

      IngressController

      resource:

      $ oc edit IngressController

   2. Under

      spec

      , set the

      HTTPHeaders

      policy field to

      Append

      ,

      Replace

      ,

      IfNone

      , or

      Never

      :

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpHeaders:
          forwardedHeaderPolicy: Append

Procedure

1. Edit the Ingress Controller resource:

   $ oc -n openshift-ingress-operator edit ingresscontroller/default

2. Set the PROXY configuration:

   • If your Ingress Controller uses the hostNetwork endpoint publishing strategy type, set the

     spec.endpointPublishingStrategy.hostNetwork.protocol

     subfield to

     PROXY

     :

     Sample hostNetwork configuration to PROXY

       spec:
         endpointPublishingStrategy:
           hostNetwork:
             protocol: PROXY
           type: HostNetwork

   • If your Ingress Controller uses the NodePortService endpoint publishing strategy type, set the

     spec.endpointPublishingStrategy.nodePort.protocol

     subfield to

     PROXY

     :

     Sample nodePort configuration to PROXY

       spec:
         endpointPublishingStrategy:
           nodePort:
             protocol: PROXY
           type: NodePortService

Procedure

1. Configure the

   appsDomain

   field by specifying an alternative default domain for user-created routes.

   1. Edit the ingress

      cluster

      resource:

      $ oc edit ingresses.config/cluster -o yaml

   2. Edit the YAML file:

      Sample appsDomain configuration to test.example.com

      apiVersion: config.openshift.io/v1
      kind: Ingress
      metadata:
        name: cluster
      spec:
        domain: apps.example.com            

      1

      
        appsDomain: <test.example.com>      

      2

      1

      Specifies the default domain. You cannot modify the default domain after installation.

      2

      Optional: Domain for OpenShift Container Platform infrastructure to use for application routes. Instead of the default prefix, apps, you can use an alternative prefix like test.

2. Verify that an existing route contains the domain name specified in the

   appsDomain

   field by exposing the route and verifying the route domain change:
   Note

   Wait for the

   openshift-apiserver

   finish rolling updates before exposing the route.

   1. Expose the route:

      $ oc expose service hello-openshift
      route.route.openshift.io/hello-openshift exposed

      Example output:

      $ oc get routes
      NAME              HOST/PORT                                   PATH   SERVICES          PORT       TERMINATION   WILDCARD
      hello-openshift   hello_openshift-<my_project>.test.example.com
      hello-openshift   8080-tcp                 None

Procedure

1. Configure the

   httpCompression

   field for the Ingress Controller.

   1. Use the following command to edit the

      IngressController

      resource:

      $ oc edit -n openshift-ingress-operator ingresscontrollers/default

   2. Under

      spec

      , set the

      httpCompression

      policy field to

      mimeTypes

      and specify a list of MIME types that should have compression applied:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpCompression:
          mimeTypes:
          - "text/html"
          - "text/css; charset=utf-8"
          - "application/json"
         ...

Procedure

1. Get the router pod name by running the following command:

   $ oc get pods -n openshift-ingress

   Example output

   NAME                              READY   STATUS    RESTARTS   AGE
   router-default-76bfffb66c-46qwp   1/1     Running   0          11h

2. Get the router’s username and password, which the router pod stores in the

   /var/lib/haproxy/conf/metrics-auth/statsUsername

   and

   /var/lib/haproxy/conf/metrics-auth/statsPassword

   files:

   1. Get the username by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsUsername

   2. Get the password by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsPassword

3. Get the router IP and metrics certificates by running the following command:

   $ oc describe pod <router_pod>

4. Get the raw statistics in Prometheus format by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

5. Access the metrics securely by running the following command:

   $ curl -u user:password https://<router_IP>:<stats_port>/metrics -k

6. Access the default stats port, 1936, by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

   Example 6.1. Example output

   …​ # HELP haproxy_backend_connections_total Total number of connections. # TYPE haproxy_backend_connections_total gauge haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route"} 0 haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route-alt"} 0 haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route01"} 0 …​ # HELP haproxy_exporter_server_threshold Number of servers tracked and the current threshold value. # TYPE haproxy_exporter_server_threshold gauge haproxy_exporter_server_threshold{type="current"} 11 haproxy_exporter_server_threshold{type="limit"} 500 …​ # HELP haproxy_frontend_bytes_in_total Current total of incoming bytes. # TYPE haproxy_frontend_bytes_in_total gauge haproxy_frontend_bytes_in_total{frontend="fe_no_sni"} 0 haproxy_frontend_bytes_in_total{frontend="fe_sni"} 0 haproxy_frontend_bytes_in_total{frontend="public"} 119070 …​ # HELP haproxy_server_bytes_in_total Current total of incoming bytes. # TYPE haproxy_server_bytes_in_total gauge haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_no_sni",service=""} 0 haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_sni",service=""} 0 haproxy_server_bytes_in_total{namespace="default",pod="docker-registry-5-nk5fz",route="docker-registry",server="10.130.0.89:5000",service="docker-registry"} 0 haproxy_server_bytes_in_total{namespace="default",pod="hello-rc-vkjqx",route="hello-route",server="10.130.0.90:8080",service="hello-svc-1"} 0 …​

7. Launch the stats window by entering the following URL in a browser:

   http://<user>:<password>@<router_IP>:<stats_port>

8. Optional: Get the stats in CSV format by entering the following URL in a browser:

   http://<user>:<password>@<router_ip>:1936/metrics;csv

Procedure

1. Create a config map named

   my-custom-error-code-pages

   in the

   openshift-config

   namespace:

   $ oc -n openshift-config create configmap my-custom-error-code-pages \
   --from-file=error-page-503.http \
   --from-file=error-page-404.http

   Important

   If you do not specify the correct format for the custom error code response page, a router pod outage occurs. To resolve this outage, you must delete or correct the config map and delete the affected router pods so they can be recreated with the correct information.

2. Patch the Ingress Controller to reference the

   my-custom-error-code-pages

   config map by name:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"httpErrorCodePages":{"name":"my-custom-error-code-pages"}}}' --type=merge

   The Ingress Operator copies the

   my-custom-error-code-pages

   config map from the

   openshift-config

   namespace to the

   openshift-ingress

   namespace. The Operator names the config map according to the pattern,

   <your_ingresscontroller_name>-errorpages

   , in the

   openshift-ingress

   namespace.

3. Display the copy:

   $ oc get cm default-errorpages -n openshift-ingress

   Example output

   NAME                       DATA   AGE
   default-errorpages         2      25s  

   1

   1

   The example config map name is default-errorpages because the default Ingress Controller custom resource (CR) was patched.

4. Confirm that the config map containing the custom error response page mounts on the router volume where the config map key is the filename that has the custom HTTP error code response:

   • For 503 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-503.http

   • For 404 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-404.http

1. Create a test project and application:

    $ oc new-project test-ingress

   $ oc new-app django-psql-example

2. For 503 custom http error code response:

   1. Stop all the pods for the application.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

3. For 404 custom http error code response:

   1. Visit a non-existent route or an incorrect route.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

4. Check if the

   errorfile

   attribute is properly in the

   haproxy.config

   file:

   $ oc -n openshift-ingress rsh <router> cat /var/lib/haproxy/conf/haproxy.config | grep errorfile