Networking

Verification

1. To create a namespace with network policies complete the following steps:

   1. Create a namespace for verification:

      $ cat <<EOF| oc create -f -
      kind: Namespace
      apiVersion: v1
      metadata:
        name: verify-audit-logging
        annotations:
          k8s.ovn.org/acl-logging: '{ "deny": "alert", "allow": "alert" }'
      EOF

      Example output

      namespace/verify-audit-logging created

   2. Create network policies for the namespace:

      $ cat <<EOF| oc create -n verify-audit-logging -f -
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: deny-all
      spec:
        podSelector:
          matchLabels:
        policyTypes:
        - Ingress
        - Egress
      ---
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-from-same-namespace
        namespace: verify-audit-logging
      spec:
        podSelector: {}
        policyTypes:
         - Ingress
         - Egress
        ingress:
          - from:
              - podSelector: {}
        egress:
          - to:
             - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: verify-audit-logging
      EOF

      Example output

      networkpolicy.networking.k8s.io/deny-all created
      networkpolicy.networking.k8s.io/allow-from-same-namespace created

2. Create a pod for source traffic in the default namespace:

   $ cat <<EOF| oc create -n default -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: client
   spec:
     containers:
       - name: client
         image: registry.access.redhat.com/rhel7/rhel-tools
         command: ["/bin/sh", "-c"]
         args:
           ["sleep inf"]
   EOF

3. Create two pods in the verify-audit-logging namespace:

   $ for name in client server; do
   cat <<EOF| oc create -n verify-audit-logging -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: ${name}
   spec:
     containers:
       - name: ${name}
         image: registry.access.redhat.com/rhel7/rhel-tools
         command: ["/bin/sh", "-c"]
         args:
           ["sleep inf"]
   EOF
   done

   Example output

   pod/client created
   pod/server created

4. To generate traffic and produce network policy audit log entries, complete the following steps:

   1. Obtain the IP address for pod named server in the verify-audit-logging namespace:

      $ POD_IP=$(oc get pods server -n verify-audit-logging -o jsonpath='{.status.podIP}')

   2. Ping the IP address from the previous command from the pod named client in the default namespace and confirm that all packets are dropped:

      $ oc exec -it client -n default -- /bin/ping -c 2 $POD_IP

      Example output

      PING 10.128.2.55 (10.128.2.55) 56(84) bytes of data.
      
      --- 10.128.2.55 ping statistics ---
      2 packets transmitted, 0 received, 100% packet loss, time 2041ms

   3. Ping the IP address saved in the POD_IP shell environment variable from the pod named client in the verify-audit-logging namespace and confirm that all packets are allowed:

      $ oc exec -it client -n verify-audit-logging -- /bin/ping -c 2 $POD_IP

      Example output

      PING 10.128.0.86 (10.128.0.86) 56(84) bytes of data.
      64 bytes from 10.128.0.86: icmp_seq=1 ttl=64 time=2.21 ms
      64 bytes from 10.128.0.86: icmp_seq=2 ttl=64 time=0.440 ms
      
      --- 10.128.0.86 ping statistics ---
      2 packets transmitted, 2 received, 0% packet loss, time 1001ms
      rtt min/avg/max/mdev = 0.440/1.329/2.219/0.890 ms

5. Display the latest entries in the network policy audit log:

   $ for pod in $(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node --no-headers=true | awk '{ print $1 }') ; do
       oc exec -it $pod -n openshift-ovn-kubernetes -- tail -4 /var/log/ovn/acl-audit-log.log
     done

   Example output

   2023-11-02T16:28:54.139Z|00004|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
   2023-11-02T16:28:55.187Z|00005|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
   2023-11-02T16:28:57.235Z|00006|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
   2023-11-02T16:49:57.909Z|00028|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Egress:0", verdict=allow, severity=alert, direction=from-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
   2023-11-02T16:49:57.909Z|00029|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Ingress:0", verdict=allow, severity=alert, direction=to-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
   2023-11-02T16:49:58.932Z|00030|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Egress:0", verdict=allow, severity=alert, direction=from-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
   2023-11-02T16:49:58.932Z|00031|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Ingress:0", verdict=allow, severity=alert, direction=to-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0

1. Create the IngressNodeFirewallConfig inside the openshift-ingress-node-firewall namespace named ingressnodefirewallconfig.

2. Run the following command to deploy Ingress Node Firewall Operator rules:

   $ oc apply -f rule.yaml

Procedure

1. Run the following command to view all current rules :

   $ oc get ingressnodefirewall

2. Choose one of the returned <resource> names and run the following command to view the rules or configs:

   $ oc get <resource> <name> -o yaml

Procedure

1. To enable IPsec encryption, enter the following command:

   $ oc patch networks.operator.openshift.io cluster --type=merge \
   -p '{
     "spec":{
       "defaultNetwork":{
         "ovnKubernetesConfig":{
           "ipsecConfig":{
             "mode":<mode>
           }}}}}'

   where:

   mode

   Specify External to encrypt only traffic to external hosts or specify Full to encrypt pod to pod traffic and optionally traffic to external hosts. By default, IPsec is disabled.

2. Optional: If you need to encrypt traffic to external hosts, complete the "Configuring IPsec encryption for external traffic" procedure.

Verification

1. To find the names of the OVN-Kubernetes data plane pods, enter the following command:

   $ oc get pods -n openshift-ovn-kubernetes -l=app=ovnkube-node

   Example output

   ovnkube-node-5xqbf                       8/8     Running   0              28m
   ovnkube-node-6mwcx                       8/8     Running   0              29m
   ovnkube-node-ck5fr                       8/8     Running   0              31m
   ovnkube-node-fr4ld                       8/8     Running   0              26m
   ovnkube-node-wgs4l                       8/8     Running   0              33m
   ovnkube-node-zfvcl                       8/8     Running   0              34m

2. Verify that IPsec is enabled on your cluster by running the following command:

   Note

   As a cluster administrator, you can verify that IPsec is enabled between pods on your cluster when IPsec is configured in Full mode. This step does not verify whether IPsec is working between your cluster and external hosts.

   $ oc -n openshift-ovn-kubernetes rsh ovnkube-node-<XXXXX> ovn-nbctl --no-leader-only get nb_global . ipsec

   where:

   <XXXXX>

   Specifies the random sequence of letters for a pod from the previous step.

   Example output

   true

Procedure

1. Create an IPsec configuration with an NMState Operator node network configuration policy. For more information, see Libreswan as an IPsec VPN implementation.

   1. To identify the IP address of the cluster node that is the IPsec endpoint, enter the following command:

      $ oc get nodes

   2. Create a file named ipsec-config.yaml that contains a node network configuration policy for the NMState Operator, such as in the following examples. For an overview about NodeNetworkConfigurationPolicy objects, see The Kubernetes NMState project.

      Example NMState IPsec transport configuration

      apiVersion: nmstate.io/v1
      kind: NodeNetworkConfigurationPolicy
      metadata:
        name: ipsec-config
      spec:
        nodeSelector:
          kubernetes.io/hostname: "<hostname>" 1
        desiredState:
          interfaces:
          - name: <interface_name> 2
            type: ipsec
            libreswan:
              left: <cluster_node> 3
              leftid: '%fromcert'
              leftrsasigkey: '%cert'
              leftcert: left_server
              leftmodecfgclient: false
              right: <external_host> 4
              rightid: '%fromcert'
              rightrsasigkey: '%cert'
              rightsubnet: <external_address>/32 5
              ikev2: insist
              type: transport

      1

      Specifies the host name to apply the policy to. This host serves as the left side host in the IPsec configuration.

      2

      Specifies the name of the interface to create on the host.

      3

      Specifies the host name of the cluster node that terminates the IPsec tunnel on the cluster side. The name should match SAN [Subject Alternate Name] from your supplied PKCS#12 certificates.

      4

      Specifies the external host name, such as host.example.com. The name should match the SAN [Subject Alternate Name] from your supplied PKCS#12 certificates.

      5

      Specifies the IP address of the external host, such as 10.1.2.3/32.

      Example NMState IPsec tunnel configuration

      apiVersion: nmstate.io/v1
      kind: NodeNetworkConfigurationPolicy
      metadata:
        name: ipsec-config
      spec:
        nodeSelector:
          kubernetes.io/hostname: "<hostname>" 1
        desiredState:
          interfaces:
          - name: <interface_name> 2
            type: ipsec
            libreswan:
              left: <cluster_node> 3
              leftid: '%fromcert'
              leftmodecfgclient: false
              leftrsasigkey: '%cert'
              leftcert: left_server
              right: <external_host> 4
              rightid: '%fromcert'
              rightrsasigkey: '%cert'
              rightsubnet: <external_address>/32 5
              ikev2: insist
              type: tunnel

      1

      Specifies the host name to apply the policy to. This host serves as the left side host in the IPsec configuration.

      2

      Specifies the name of the interface to create on the host.

      3

      Specifies the host name of the cluster node that terminates the IPsec tunnel on the cluster side. The name should match SAN [Subject Alternate Name] from your supplied PKCS#12 certificates.

      4

      Specifies the external host name, such as host.example.com. The name should match the SAN [Subject Alternate Name] from your supplied PKCS#12 certificates.

      5

      Specifies the IP address of the external host, such as 10.1.2.3/32.

   3. To configure the IPsec interface, enter the following command:

      $ oc create -f ipsec-config.yaml

2. Provide the following certificate files to add to the Network Security Services (NSS) database on each host. These files are imported as part of the Butane configuration in subsequent steps.

   • left_server.p12: The certificate bundle for the IPsec endpoints
   • ca.pem: The certificate authority that you signed your certificates with

3. Create a machine config to add your certificates to the cluster:

   1. To create Butane config files for the control plane and worker nodes, enter the following command:

      $ for role in master worker; do
        cat >> "99-ipsec-${role}-endpoint-config.bu" <<-EOF
        variant: openshift
        version: 4.16.0
        metadata:
          name: 99-${role}-import-certs
          labels:
            machineconfiguration.openshift.io/role: $role
        systemd:
          units:
          - name: ipsec-import.service
            enabled: true
            contents: |
              [Unit]
              Description=Import external certs into ipsec NSS
              Before=ipsec.service
      
              [Service]
              Type=oneshot
              ExecStart=/usr/local/bin/ipsec-addcert.sh
              RemainAfterExit=false
              StandardOutput=journal
      
              [Install]
              WantedBy=multi-user.target
        storage:
          files:
          - path: /etc/pki/certs/ca.pem
            mode: 0400
            overwrite: true
            contents:
              local: ca.pem
          - path: /etc/pki/certs/left_server.p12
            mode: 0400
            overwrite: true
            contents:
              local: left_server.p12
          - path: /usr/local/bin/ipsec-addcert.sh
            mode: 0740
            overwrite: true
            contents:
              inline: |
                #!/bin/bash -e
                echo "importing cert to NSS"
                certutil -A -n "CA" -t "CT,C,C" -d /var/lib/ipsec/nss/ -i /etc/pki/certs/ca.pem
                pk12util -W "" -i /etc/pki/certs/left_server.p12 -d /var/lib/ipsec/nss/
                certutil -M -n "left_server" -t "u,u,u" -d /var/lib/ipsec/nss/
      EOF
      done

   2. To transform the Butane files that you created in the previous step into machine configs, enter the following command:

      $ for role in master worker; do
        butane -d . 99-ipsec-${role}-endpoint-config.bu -o ./99-ipsec-$role-endpoint-config.yaml
      done

4. To apply the machine configs to your cluster, enter the following command:

   $ for role in master worker; do
     oc apply -f 99-ipsec-${role}-endpoint-config.yaml
   done

   Important

   As the Machine Config Operator (MCO) updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated before external IPsec connectivity is available.

5. Check the machine config pool status by entering the following command:

   $ oc get mcp

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

6. To confirm that IPsec machine configs rolled out successfully, enter the following commands:

   1. Confirm that the IPsec machine configs were created:

      $ oc get mc | grep ipsec

      Example output

      80-ipsec-master-extensions        3.2.0        6d15h
      80-ipsec-worker-extensions        3.2.0        6d15h

   2. Confirm that the that the IPsec extension are applied to control plane nodes:

      $ oc get mcp master -o yaml | grep 80-ipsec-master-extensions -c

      Expected output

      2

   3. Confirm that the that the IPsec extension are applied to worker nodes:

      $ oc get mcp worker -o yaml | grep 80-ipsec-worker-extensions -c

      Expected output

      2

Procedure

1. Create a file named remove-ipsec-tunnel.yaml with the following YAML:

   kind: NodeNetworkConfigurationPolicy
   apiVersion: nmstate.io/v1
   metadata:
     name: <name>
   spec:
     nodeSelector:
       kubernetes.io/hostname: <node_name>
     desiredState:
       interfaces:
       - name: <tunnel_name>
         type: ipsec
         state: absent

   where:

   name

   Specifies a name for the node network configuration policy.

   node_name

   Specifies the name of the node where the IPsec tunnel that you want to remove exists.

   tunnel_name

   Specifies the interface name for the existing IPsec tunnel.

2. To remove the IPsec tunnel, enter the following command:

   $ oc apply -f remove-ipsec-tunnel.yaml

Procedure

1. To disable IPsec encryption, enter the following command:

   $ oc patch networks.operator.openshift.io cluster --type=merge \
   -p '{
     "spec":{
       "defaultNetwork":{
         "ovnKubernetesConfig":{
           "ipsecConfig":{
             "mode":"Disabled"
           }}}}}'

2. Optional: You can increase the size of your cluster MTU by 46 bytes because there is no longer any overhead from the IPsec ESP header in IP packets.

Procedure

1. Change managementState to Unmanaged in the DNS Operator:

   oc patch dns.operator.openshift.io default --type merge --patch '{"spec":{"managementState":"Unmanaged"}}'

2. Review managementState of the DNS Operator using the jsonpath command line JSON parser:

   $ oc get dns.operator.openshift.io default -ojsonpath='{.spec.managementState}'

   Example output

   "Unmanaged"

Procedure

1. Modify the DNS Operator object named default:

   $ oc edit dns.operator/default

   Cluster administrators can configure transport layer security (TLS) for forwarded DNS queries.

   Configuring DNS forwarding with TLS

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     name: default
   spec:
     servers:
     - name: example-server 1
       zones:
       - example.com 2
       forwardPlugin:
         transportConfig:
           transport: TLS 3
           tls:
             caBundle:
               name: mycacert
             serverName: dnstls.example.com  4
         policy: Random 5
         upstreams: 6
         - 1.1.1.1
         - 2.2.2.2:5353
     upstreamResolvers: 7
       transportConfig:
         transport: TLS
         tls:
           caBundle:
             name: mycacert
           serverName: dnstls.example.com
       upstreams:
       - type: Network 8
         address: 1.2.3.4 9
         port: 53 10

   1

   Must comply with the rfc6335 service name syntax.

   2

   Must conform to the definition of a subdomain in the rfc1123 service name syntax. The cluster domain, cluster.local, is an invalid subdomain for the zones field. The cluster domain, cluster.local, is an invalid subdomain for zones.

   3

   When configuring TLS for forwarded DNS queries, set the transport field to have the value TLS.

   4

   When configuring TLS for forwarded DNS queries, this is a mandatory server name used as part of the server name indication (SNI) to validate the upstream TLS server certificate.

   5

   Defines the policy to select upstream resolvers. Default value is Random. You can also use the values RoundRobin, and Sequential.

   6

   Required. Use it to provide upstream resolvers. A maximum of 15 upstreams entries are allowed per forwardPlugin entry.

   7

   Optional. You can use it to override the default policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. If you do not provide any upstream resolvers, the DNS name queries go to the servers in /etc/resolv.conf.

   8

   Only the Network type is allowed when using TLS and you must provide an IP address. Network type indicates that this upstream resolver should handle forwarded requests separately from the upstream resolvers listed in /etc/resolv.conf.

   9

   The address field must be a valid IPv4 or IPv6 address.

   10

   You can optionally provide a port. The port must have a value between 1 and 65535. If you do not specify a port for the upstream, the default port is 853.

   Note

   If servers is undefined or invalid, the config map only contains the default server.

Verification

1. View the config map:

   $ oc get configmap/dns-default -n openshift-dns -o yaml

   Sample DNS ConfigMap based on TLS forwarding example

   apiVersion: v1
   data:
     Corefile: |
       example.com:5353 {
           forward . 1.1.1.1 2.2.2.2:5353
       }
       bar.com:5353 example.com:5353 {
           forward . 3.3.3.3 4.4.4.4:5454 1
       }
       .:5353 {
           errors
           health
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods insecure
               upstream
               fallthrough in-addr.arpa ip6.arpa
           }
           prometheus :9153
           forward . /etc/resolv.conf 1.2.3.4:53 {
               policy Random
           }
           cache 30
           reload
       }
   kind: ConfigMap
   metadata:
     labels:
       dns.operator.openshift.io/owning-dns: default
     name: dns-default
     namespace: openshift-dns

   1

   Changes to the forwardPlugin triggers a rolling update of the CoreDNS daemon set.

1. Create a Secret resource containing the custom certificate in the openshift-ingress namespace using the tls.crt and tls.key files.

   $ oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key

2. Update the IngressController CR to reference the new certificate secret:

   $ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
     --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'

3. Verify the update was effective:

   $ echo Q |\
     openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null |\
     openssl x509 -noout -subject -issuer -enddate

   where:

   <domain>

   Specifies the base domain name for your cluster.

   Example output

   subject=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = *.apps.example.com
   issuer=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = example.com
   notAfter=May 10 08:32:45 2022 GM

   Tip

   You can alternatively apply the following YAML to set a custom default certificate:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     defaultCertificate:
       name: custom-certs-default

   The certificate secret name should match the value used to update the CR.

Procedure

1. Create a service account to authenticate with Thanos by running the following command:

   $ oc create -n openshift-ingress-operator serviceaccount thanos && oc describe -n openshift-ingress-operator serviceaccount thanos

   Example output

   Name:                thanos
   Namespace:           openshift-ingress-operator
   Labels:              <none>
   Annotations:         <none>
   Image pull secrets:  thanos-dockercfg-kfvf2
   Mountable secrets:   thanos-dockercfg-kfvf2
   Tokens:              <none>
   Events:              <none>

2. Manually create the service account secret token with the following command:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: thanos-token
     namespace: openshift-ingress-operator
     annotations:
       kubernetes.io/service-account.name: thanos
   type: kubernetes.io/service-account-token
   EOF

3. Define a TriggerAuthentication object within the openshift-ingress-operator namespace by using the service account’s token.

   1. Create the TriggerAuthentication object and pass the value of the secret variable to the TOKEN parameter:

      $ oc apply -f - <<EOF
      apiVersion: keda.sh/v1alpha1
      kind: TriggerAuthentication
      metadata:
        name: keda-trigger-auth-prometheus
        namespace: openshift-ingress-operator
      spec:
        secretTargetRef:
        - parameter: bearerToken
          name: thanos-token
          key: token
        - parameter: ca
          name: thanos-token
          key: ca.crt
      EOF

4. Create and apply a role for reading metrics from Thanos:

   1. Create a new role, thanos-metrics-reader.yaml, that reads metrics from pods and nodes:

      thanos-metrics-reader.yaml

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: thanos-metrics-reader
        namespace: openshift-ingress-operator
      rules:
      - apiGroups:
        - ""
        resources:
        - pods
        - nodes
        verbs:
        - get
      - apiGroups:
        - metrics.k8s.io
        resources:
        - pods
        - nodes
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - ""
        resources:
        - namespaces
        verbs:
        - get

   2. Apply the new role by running the following command:

      $ oc apply -f thanos-metrics-reader.yaml

5. Add the new role to the service account by entering the following commands:

   $ oc adm policy -n openshift-ingress-operator add-role-to-user thanos-metrics-reader -z thanos --role-namespace=openshift-ingress-operator

   $ oc adm policy -n openshift-ingress-operator add-cluster-role-to-user cluster-monitoring-view -z thanos

   Note

   The argument add-cluster-role-to-user is only required if you use cross-namespace queries. The following step uses a query from the kube-metrics namespace which requires this argument.

6. Create a new ScaledObject YAML file, ingress-autoscaler.yaml, that targets the default Ingress Controller deployment:

   Example ScaledObject definition

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: ingress-scaler
     namespace: openshift-ingress-operator
   spec:
     scaleTargetRef: 1
       apiVersion: operator.openshift.io/v1
       kind: IngressController
       name: default
       envSourceContainerName: ingress-operator
     minReplicaCount: 1
     maxReplicaCount: 20 2
     cooldownPeriod: 1
     pollingInterval: 1
     triggers:
     - type: prometheus
       metricType: AverageValue
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 3
         namespace: openshift-ingress-operator 4
         metricName: 'kube-node-role'
         threshold: '1'
         query: 'sum(kube_node_role{role="worker",service="kube-state-metrics"})' 5
         authModes: "bearer"
       authenticationRef:
         name: keda-trigger-auth-prometheus

   1

   The custom resource that you are targeting. In this case, the Ingress Controller.

   2

   Optional: The maximum number of replicas. If you omit this field, the default maximum is set to 100 replicas.

   3

   The Thanos service endpoint in the openshift-monitoring namespace.

   4

   The Ingress Operator namespace.

   5

   This expression evaluates to however many worker nodes are present in the deployed cluster.

   Important

   If you are using cross-namespace queries, you must target port 9091 and not port 9092 in the serverAddress field. You also must have elevated privileges to read metrics from this port.

7. Apply the custom resource definition by running the following command:

   $ oc apply -f ingress-autoscaler.yaml

Procedure

1. View the current number of available replicas for the default IngressController:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   2

2. Scale the default IngressController to the desired number of replicas using the oc patch command. The following example scales the default IngressController to 3 replicas:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"replicas": 3}}' --type=merge

   Example output

   ingresscontroller.operator.openshift.io/default patched

3. Verify that the default IngressController scaled to the number of replicas that you specified:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   3

   Tip

   You can alternatively apply the following YAML to scale an Ingress Controller to three replicas:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: 3               1

   1

   If you need a different amount of replicas, change the replicas value.