Chapter 8. Ingress Operator in OpenShift Container Platform

1. Create a Secret resource containing the custom certificate in the openshift-ingress namespace using the tls.crt and tls.key files.

   $ oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key

   Copy to Clipboard Toggle word wrap

2. Update the IngressController CR to reference the new certificate secret:

   $ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
     --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'

   Copy to Clipboard Toggle word wrap

3. Verify the update was effective:

   $ echo Q |\
     openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null |\
     openssl x509 -noout -subject -issuer -enddate

   Copy to Clipboard Toggle word wrap

   where:

   <domain>

   Specifies the base domain name for your cluster.

   Example output

   subject=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = *.apps.example.com
   issuer=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = example.com
   notAfter=May 10 08:32:45 2022 GM

   Copy to Clipboard Toggle word wrap

   Tip

   You can alternatively apply the following YAML to set a custom default certificate:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     defaultCertificate:
       name: custom-certs-default

   Copy to Clipboard Toggle word wrap

   The certificate secret name should match the value used to update the CR.

Procedure

1. Create a service account to authenticate with Thanos by running the following command:

   $ oc create -n openshift-ingress-operator serviceaccount thanos && oc describe -n openshift-ingress-operator serviceaccount thanos

   Copy to Clipboard Toggle word wrap

   Example output

   Name:                thanos
   Namespace:           openshift-ingress-operator
   Labels:              <none>
   Annotations:         <none>
   Image pull secrets:  thanos-dockercfg-kfvf2
   Mountable secrets:   thanos-dockercfg-kfvf2
   Tokens:              <none>
   Events:              <none>

   Copy to Clipboard Toggle word wrap

2. Manually create the service account secret token with the following command:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: thanos-token
     namespace: openshift-ingress-operator
     annotations:
       kubernetes.io/service-account.name: thanos
   type: kubernetes.io/service-account-token
   EOF

   Copy to Clipboard Toggle word wrap

3. Define a TriggerAuthentication object within the openshift-ingress-operator namespace by using the service account’s token.

   1. Create the TriggerAuthentication object and pass the value of the secret variable to the TOKEN parameter:

      $ oc apply -f - <<EOF
      apiVersion: keda.sh/v1alpha1
      kind: TriggerAuthentication
      metadata:
        name: keda-trigger-auth-prometheus
        namespace: openshift-ingress-operator
      spec:
        secretTargetRef:
        - parameter: bearerToken
          name: thanos-token
          key: token
        - parameter: ca
          name: thanos-token
          key: ca.crt
      EOF

      Copy to Clipboard Toggle word wrap

4. Create and apply a role for reading metrics from Thanos:

   1. Create a new role, thanos-metrics-reader.yaml, that reads metrics from pods and nodes:

      thanos-metrics-reader.yaml

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: thanos-metrics-reader
        namespace: openshift-ingress-operator
      rules:
      - apiGroups:
        - ""
        resources:
        - pods
        - nodes
        verbs:
        - get
      - apiGroups:
        - metrics.k8s.io
        resources:
        - pods
        - nodes
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - ""
        resources:
        - namespaces
        verbs:
        - get

      Copy to Clipboard Toggle word wrap

   2. Apply the new role by running the following command:

      $ oc apply -f thanos-metrics-reader.yaml

      Copy to Clipboard Toggle word wrap

5. Add the new role to the service account by entering the following commands:

   $ oc adm policy -n openshift-ingress-operator add-role-to-user thanos-metrics-reader -z thanos --role-namespace=openshift-ingress-operator

   Copy to Clipboard Toggle word wrap

   $ oc adm policy -n openshift-ingress-operator add-cluster-role-to-user cluster-monitoring-view -z thanos

   Copy to Clipboard Toggle word wrap
   Note

   The argument add-cluster-role-to-user is only required if you use cross-namespace queries. The following step uses a query from the kube-metrics namespace which requires this argument.

6. Create a new ScaledObject YAML file, ingress-autoscaler.yaml, that targets the default Ingress Controller deployment:

   Example ScaledObject definition

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: ingress-scaler
     namespace: openshift-ingress-operator
   spec:
     scaleTargetRef: 

   1

   
       apiVersion: operator.openshift.io/v1
       kind: IngressController
       name: default
       envSourceContainerName: ingress-operator
     minReplicaCount: 1
     maxReplicaCount: 20 

   2

   
     cooldownPeriod: 1
     pollingInterval: 1
     triggers:
     - type: prometheus
       metricType: AverageValue
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 

   3

   
         namespace: openshift-ingress-operator 

   4

   
         metricName: 'kube-node-role'
         threshold: '1'
         query: 'sum(kube_node_role{role="worker",service="kube-state-metrics"})' 

   5

   
         authModes: "bearer"
       authenticationRef:
         name: keda-trigger-auth-prometheus

   Copy to Clipboard Toggle word wrap

   1

   The custom resource that you are targeting. In this case, the Ingress Controller.

   2

   Optional: The maximum number of replicas. If you omit this field, the default maximum is set to 100 replicas.

   3

   The Thanos service endpoint in the openshift-monitoring namespace.

   4

   The Ingress Operator namespace.

   5

   This expression evaluates to however many worker nodes are present in the deployed cluster.

   Important

   If you are using cross-namespace queries, you must target port 9091 and not port 9092 in the serverAddress field. You also must have elevated privileges to read metrics from this port.

7. Apply the custom resource definition by running the following command:

   $ oc apply -f ingress-autoscaler.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. View the current number of available replicas for the default IngressController:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Copy to Clipboard Toggle word wrap

2. Scale the default IngressController to the desired number of replicas by using the oc patch command. The following example scales the default IngressController to 3 replicas.

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"replicas": 3}}' --type=merge

   Copy to Clipboard Toggle word wrap

3. Verify that the default IngressController scaled to the number of replicas that you specified:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Copy to Clipboard Toggle word wrap
   Tip

   You can alternatively apply the following YAML to scale an Ingress Controller to three replicas:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: 3               

   1

   Copy to Clipboard Toggle word wrap

   1

   If you need a different amount of replicas, change the replicas value.

Procedure

1. Create an IngressController custom resource (CR) in a file named <name>-ingress-controller.yaml, such as in the following example:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: <name> 

   1

   
   spec:
     domain: <domain> 

   2

   
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal 

   3

   Copy to Clipboard Toggle word wrap

   1

   Replace <name> with a name for the IngressController object.

   2

   Specify the domain for the application published by the controller.

   3

   Specify a value of Internal to use an internal load balancer.

2. Create the Ingress Controller defined in the previous step by running the following command:

   $ oc create -f <name>-ingress-controller.yaml 

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace <name> with the name of the IngressController object.

3. Optional: Confirm that the Ingress Controller was created by running the following command:

   $ oc --all-namespaces=true get ingresscontrollers

   Copy to Clipboard Toggle word wrap

Procedure

1. Configure the Ingress Controller resource to allow global access.

   Note

   You can also create an Ingress Controller and specify the global access option.

   1. Configure the Ingress Controller resource:

      $ oc -n openshift-ingress-operator edit ingresscontroller/default

      Copy to Clipboard Toggle word wrap

   2. Edit the YAML file:

      Sample clientAccess configuration to Global

        spec:
          endpointPublishingStrategy:
            loadBalancer:
              providerParameters:
                gcp:
                  clientAccess: Global 

      1

      
                type: GCP
              scope: Internal
            type: LoadBalancerService

      Copy to Clipboard Toggle word wrap

      1

      Set gcp.clientAccess to Global.

   3. Save the file to apply the changes.

2. Run the following command to verify that the service allows global access:

   $ oc -n openshift-ingress edit svc/router-default -o yaml

   Copy to Clipboard Toggle word wrap

   The output shows that global access is enabled for GCP with the annotation, networking.gke.io/internal-load-balancer-allow-global-access.

Procedure

1. Configure the default Ingress Controller for your cluster to be internal by deleting and recreating it.

   $ oc replace --force --wait --filename - <<EOF
   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: default
   spec:
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal
   EOF

   Copy to Clipboard Toggle word wrap

Procedure

1. Configure the wildcard policy.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

      Copy to Clipboard Toggle word wrap

   2. Under spec, set the wildcardPolicy field to WildcardsDisallowed or WildcardsAllowed:

      spec:
        routeAdmission:
          wildcardPolicy: WildcardsDisallowed # or WildcardsAllowed

      Copy to Clipboard Toggle word wrap

Procedure

1. Edit the Ingress Controller resource:

   $ oc -n openshift-ingress-operator edit ingresscontroller/default

   Copy to Clipboard Toggle word wrap

2. Replace the X-SSL-Client-Der HTTP request header with the X-Forwarded-Client-Cert HTTP request header:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     httpHeaders:
       actions: 

   1

   
         request: 

   2

   
         - name: X-Forwarded-Client-Cert 

   3

   
           action:
             type: Set 

   4

   
             set:
              value: "%{+Q}[ssl_c_der,base64]" 

   5

   
         - name: X-SSL-Client-Der
           action:
             type: Delete

   Copy to Clipboard Toggle word wrap

   1

   The list of actions you want to perform on the HTTP headers.

   2

   The type of header you want to change. In this case, a request header.

   3

   The name of the header you want to change. For a list of available headers you can set or delete, see HTTP header configuration.

   4

   The type of action being taken on the header. This field can have the value Set or Delete.

   5

   When setting HTTP headers, you must provide a value. The value can be a string from a list of available directives for that header, for example DENY, or it can be a dynamic value that will be interpreted using HAProxy’s dynamic value syntax. In this case, a dynamic value is added.

   Note

   For setting dynamic header values for HTTP responses, allowed sample fetchers are res.hdr and ssl_c_der. For setting dynamic header values for HTTP requests, allowed sample fetchers are req.hdr and ssl_c_der. Both request and response dynamic values can use the lower and base64 converters.

3. Save the file to apply the changes.

Procedure

1. Configure the HTTPHeaders field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

      Copy to Clipboard Toggle word wrap

   2. Under spec, set the HTTPHeaders policy field to Append, Replace, IfNone, or Never:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpHeaders:
          forwardedHeaderPolicy: Append

      Copy to Clipboard Toggle word wrap

Procedure

1. Edit the Ingress Controller resource by entering the following command in your CLI:

   $ oc -n openshift-ingress-operator edit ingresscontroller/default

   Copy to Clipboard Toggle word wrap

2. Set the PROXY configuration:

   • If your Ingress Controller uses the HostNetwork endpoint publishing strategy type, set the spec.endpointPublishingStrategy.hostNetwork.protocol subfield to PROXY:

     Sample hostNetwork configuration to PROXY

     # ...
       spec:
         endpointPublishingStrategy:
           hostNetwork:
             protocol: PROXY
           type: HostNetwork
     # ...

     Copy to Clipboard Toggle word wrap

   • If your Ingress Controller uses the NodePortService endpoint publishing strategy type, set the spec.endpointPublishingStrategy.nodePort.protocol subfield to PROXY:

     Sample nodePort configuration to PROXY

     # ...
       spec:
         endpointPublishingStrategy:
           nodePort:
             protocol: PROXY
           type: NodePortService
     # ...

     Copy to Clipboard Toggle word wrap

   • If your Ingress Controller uses the Private endpoint publishing strategy type, set the spec.endpointPublishingStrategy.private.protocol subfield to PROXY:

     Sample private configuration to PROXY

     # ...
       spec:
         endpointPublishingStrategy:
           private:
             protocol: PROXY
         type: Private
     # ...

     Copy to Clipboard Toggle word wrap

Procedure

1. Configure the appsDomain field by specifying an alternative default domain for user-created routes.

   1. Edit the ingress cluster resource:

      $ oc edit ingresses.config/cluster -o yaml

      Copy to Clipboard Toggle word wrap

   2. Edit the YAML file:

      Sample appsDomain configuration to test.example.com

      apiVersion: config.openshift.io/v1
      kind: Ingress
      metadata:
        name: cluster
      spec:
        domain: apps.example.com            

      1

      
        appsDomain: <test.example.com>      

      2

      Copy to Clipboard Toggle word wrap

      1

      Specifies the default domain. You cannot modify the default domain after installation.

      2

      Optional: Domain for OpenShift Container Platform infrastructure to use for application routes. Instead of the default prefix, apps, you can use an alternative prefix like test.

2. Verify that an existing route contains the domain name specified in the appsDomain field by exposing the route and verifying the route domain change:

   Note

   Wait for the openshift-apiserver finish rolling updates before exposing the route.

   1. Expose the route by entering the following command. The command outputs route.route.openshift.io/hello-openshift exposed to designate exposure of the route.

      $ oc expose service hello-openshift

      Copy to Clipboard Toggle word wrap

   2. Get a list of routes by running the following command:

      $ oc get routes

      Copy to Clipboard Toggle word wrap

      Example output

      NAME              HOST/PORT                                   PATH   SERVICES          PORT       TERMINATION   WILDCARD
      hello-openshift   hello_openshift-<my_project>.test.example.com
      hello-openshift   8080-tcp                 None

      Copy to Clipboard Toggle word wrap

Procedure

1. Configure the httpCompression field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit -n openshift-ingress-operator ingresscontrollers/default

      Copy to Clipboard Toggle word wrap

   2. Under spec, set the httpCompression policy field to mimeTypes and specify a list of MIME types that should have compression applied:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpCompression:
          mimeTypes:
          - "text/html"
          - "text/css; charset=utf-8"
          - "application/json"
         ...

      Copy to Clipboard Toggle word wrap

Procedure

1. Get the router pod name by running the following command:

   $ oc get pods -n openshift-ingress

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                              READY   STATUS    RESTARTS   AGE
   router-default-76bfffb66c-46qwp   1/1     Running   0          11h

   Copy to Clipboard Toggle word wrap

2. Get the router’s username and password, which the router pod stores in the /var/lib/haproxy/conf/metrics-auth/statsUsername and /var/lib/haproxy/conf/metrics-auth/statsPassword files:

   1. Get the username by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsUsername

      Copy to Clipboard Toggle word wrap

   2. Get the password by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsPassword

      Copy to Clipboard Toggle word wrap

3. Get the router IP and metrics certificates by running the following command:

   $ oc describe pod <router_pod>

   Copy to Clipboard Toggle word wrap

4. Get the raw statistics in Prometheus format by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

   Copy to Clipboard Toggle word wrap

5. Access the metrics securely by running the following command:

   $ curl -u user:password https://<router_IP>:<stats_port>/metrics -k

   Copy to Clipboard Toggle word wrap

6. Access the default stats port, 1936, by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

   Copy to Clipboard Toggle word wrap

   Example 8.1. Example output

   ...
   # HELP haproxy_backend_connections_total Total number of connections.
   # TYPE haproxy_backend_connections_total gauge
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route"} 0
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route-alt"} 0
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route01"} 0
   ...
   # HELP haproxy_exporter_server_threshold Number of servers tracked and the current threshold value.
   # TYPE haproxy_exporter_server_threshold gauge
   haproxy_exporter_server_threshold{type="current"} 11
   haproxy_exporter_server_threshold{type="limit"} 500
   ...
   # HELP haproxy_frontend_bytes_in_total Current total of incoming bytes.
   # TYPE haproxy_frontend_bytes_in_total gauge
   haproxy_frontend_bytes_in_total{frontend="fe_no_sni"} 0
   haproxy_frontend_bytes_in_total{frontend="fe_sni"} 0
   haproxy_frontend_bytes_in_total{frontend="public"} 119070
   ...
   # HELP haproxy_server_bytes_in_total Current total of incoming bytes.
   # TYPE haproxy_server_bytes_in_total gauge
   haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_no_sni",service=""} 0
   haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_sni",service=""} 0
   haproxy_server_bytes_in_total{namespace="default",pod="docker-registry-5-nk5fz",route="docker-registry",server="10.130.0.89:5000",service="docker-registry"} 0
   haproxy_server_bytes_in_total{namespace="default",pod="hello-rc-vkjqx",route="hello-route",server="10.130.0.90:8080",service="hello-svc-1"} 0
   ...

   Copy to Clipboard Toggle word wrap

7. Launch the stats window by entering the following URL in a browser:

   http://<user>:<password>@<router_IP>:<stats_port>

   Copy to Clipboard Toggle word wrap

8. Optional: Get the stats in CSV format by entering the following URL in a browser:

   http://<user>:<password>@<router_ip>:1936/metrics;csv

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a config map named my-custom-error-code-pages in the openshift-config namespace:

   $ oc -n openshift-config create configmap my-custom-error-code-pages \
     --from-file=error-page-503.http \
     --from-file=error-page-404.http

   Copy to Clipboard Toggle word wrap
   Important

   If you do not specify the correct format for the custom error code response page, a router pod outage occurs. To resolve this outage, you must delete or correct the config map and delete the affected router pods so they can be recreated with the correct information.

2. Patch the Ingress Controller to reference the my-custom-error-code-pages config map by name:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"httpErrorCodePages":{"name":"my-custom-error-code-pages"}}}' --type=merge

   Copy to Clipboard Toggle word wrap

   The Ingress Operator copies the my-custom-error-code-pages config map from the openshift-config namespace to the openshift-ingress namespace. The Operator names the config map according to the pattern, <your_ingresscontroller_name>-errorpages, in the openshift-ingress namespace.

3. Display the copy:

   $ oc get cm default-errorpages -n openshift-ingress

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                       DATA   AGE
   default-errorpages         2      25s  

   1

   Copy to Clipboard Toggle word wrap

   1

   The example config map name is default-errorpages because the default Ingress Controller custom resource (CR) was patched.

4. Confirm that the config map containing the custom error response page mounts on the router volume where the config map key is the filename that has the custom HTTP error code response:

   • For 503 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-503.http

     Copy to Clipboard Toggle word wrap

   • For 404 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-404.http

     Copy to Clipboard Toggle word wrap

1. Create a test project and application:

   $ oc new-project test-ingress

   Copy to Clipboard Toggle word wrap

   $ oc new-app django-psql-example

   Copy to Clipboard Toggle word wrap

2. For 503 custom http error code response:

   1. Stop all the pods for the application.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

      Copy to Clipboard Toggle word wrap

3. For 404 custom http error code response:

   1. Visit a non-existent route or an incorrect route.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

      Copy to Clipboard Toggle word wrap

4. Check if the errorfile attribute is properly in the haproxy.config file:

   $ oc -n openshift-ingress rsh <router> cat /var/lib/haproxy/conf/haproxy.config | grep errorfile

   Copy to Clipboard Toggle word wrap