Chapter 6. Networking Operators

Procedure

1. Select Operators OperatorHub.
2. In the search field below All Items, enter nmstate and click Enter to search for the Kubernetes NMState Operator.
3. Click on the Kubernetes NMState Operator search result.
4. Click on Install to open the Install Operator window.
5. Click Install to install the Operator.
6. After the Operator finishes installing, click View Operator.
7. Under Provided APIs, click Create Instance to open the dialog box for creating an instance of kubernetes-nmstate.

8. In the Name field of the dialog box, ensure the name of the instance is nmstate.

   Note

   The name restriction is a known issue. The instance is a singleton for the entire cluster.

9. Accept the default settings and click Create to create the instance.

Procedure

1. Create the nmstate Operator namespace:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-nmstate
   spec:
     finalizers:
     - kubernetes
   EOF

2. Create the OperatorGroup:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-nmstate
     namespace: openshift-nmstate
   spec:
     targetNamespaces:
     - openshift-nmstate
   EOF

3. Subscribe to the nmstate Operator:

   $ cat << EOF| oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: kubernetes-nmstate-operator
     namespace: openshift-nmstate
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: kubernetes-nmstate-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

4. Confirm the ClusterServiceVersion (CSV) status for the nmstate Operator deployment equals Succeeded:

   $ oc get clusterserviceversion -n openshift-nmstate \
    -o custom-columns=Name:.metadata.name,Phase:.status.phase

   Example output

   Name                                             Phase
   kubernetes-nmstate-operator.4.16.0-202210210157   Succeeded

5. Create an instance of the nmstate Operator:

   $ cat << EOF | oc apply -f -
   apiVersion: nmstate.io/v1
   kind: NMState
   metadata:
     name: nmstate
   EOF

6. Verify the pods for NMState Operator are running:

   $ oc get pod -n openshift-nmstate

   Example output

   Name                                      Ready   Status  Restarts  Age
   pod/nmstate-cert-manager-5b47d8dddf-5wnb5   1/1   Running  0         77s
   pod/nmstate-console-plugin-d6b76c6b9-4dcwm  1/1   Running  0         77s
   pod/nmstate-handler-6v7rm                   1/1   Running  0         77s
   pod/nmstate-handler-bjcxw                   1/1   Running  0         77s
   pod/nmstate-handler-fv6m2                   1/1   Running  0         77s
   pod/nmstate-handler-kb8j6                   1/1   Running  0         77s
   pod/nmstate-handler-wn55p                   1/1   Running  0         77s
   pod/nmstate-operator-f6bb869b6-v5m92        1/1   Running  0        4m51s
   pod/nmstate-webhook-66d6bbd84b-6n674        1/1   Running  0         77s
   pod/nmstate-webhook-66d6bbd84b-vlzrd        1/1   Running  0         77s

Procedure

1. You can deploy the AWS Load Balancer Operator on demand from OperatorHub, by creating a Subscription object by running the following command:

   $ oc -n aws-load-balancer-operator get sub aws-load-balancer-operator --template='{{.status.installplan.name}}{{"\n"}}'

   Example output

   install-zlfbt

2. Check if the status of an install plan is Complete by running the following command:

   $ oc -n aws-load-balancer-operator get ip <install_plan_name> --template='{{.status.phase}}{{"\n"}}'

   Example output

   Complete

3. View the status of the aws-load-balancer-operator-controller-manager deployment by running the following command:

   $ oc get -n aws-load-balancer-operator deployment/aws-load-balancer-operator-controller-manager

   Example output

   NAME                                           READY     UP-TO-DATE   AVAILABLE   AGE
   aws-load-balancer-operator-controller-manager  1/1       1            1           23h

Procedure

1. Navigate to Operators OperatorHub in the OpenShift Container Platform web console.
2. Select the AWS Load Balancer Operator. You can use the Filter by keyword text box or use the filter list to search for the AWS Load Balancer Operator from the list of Operators.
3. Select the aws-load-balancer-operator namespace.

4. On the Install Operator page, select the following options:

   1. Update the channel as stable-v1.
   2. Installation mode as All namespaces on the cluster (default).
   3. Installed Namespace as aws-load-balancer-operator. If the aws-load-balancer-operator namespace does not exist, it gets created during the Operator installation.
   4. Select Update approval as Automatic or Manual. By default, the Update approval is set to Automatic. If you select automatic updates, the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention. If you select manual updates, the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator updated to the new version.
5. Click Install.

Procedure

1. Create a Namespace object:

   1. Create a YAML file that defines the Namespace object:

      Example namespace.yaml file

      apiVersion: v1
      kind: Namespace
      metadata:
        name: aws-load-balancer-operator

   2. Create the Namespace object by running the following command:

      $ oc apply -f namespace.yaml

2. Create an OperatorGroup object:

   1. Create a YAML file that defines the OperatorGroup object:

      Example operatorgroup.yaml file

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: aws-lb-operatorgroup
        namespace: aws-load-balancer-operator
      spec:
        upgradeStrategy: Default

   2. Create the OperatorGroup object by running the following command:

      $ oc apply -f operatorgroup.yaml

3. Create a Subscription object:

   1. Create a YAML file that defines the Subscription object:

      Example subscription.yaml file

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: aws-load-balancer-operator
        namespace: aws-load-balancer-operator
      spec:
        channel: stable-v1
        installPlanApproval: Automatic
        name: aws-load-balancer-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the Subscription object by running the following command:

      $ oc apply -f subscription.yaml

Verification

1. Get the name of the install plan from the subscription:

   $ oc -n aws-load-balancer-operator \
       get subscription aws-load-balancer-operator \
       --template='{{.status.installplan.name}}{{"\n"}}'

2. Check the status of the install plan:

   $ oc -n aws-load-balancer-operator \
       get ip <install_plan_name> \
       --template='{{.status.phase}}{{"\n"}}'

   The output must be Complete.

Procedure

1. Create the aws-load-balancer-operator project by running the following command:

   $ oc new-project aws-load-balancer-operator

2. Create the OperatorGroup object by running the following command:

   $ cat <<EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     targetNamespaces: []
   EOF

3. Create the Subscription object by running the following command:

   $ cat <<EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     channel: stable-v1
     name: aws-load-balancer-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     config:
       env:
       - name: ROLEARN
         value: "<albo_role_arn>" 1
   EOF

   1

   Specifies the ARN role to be used in the CredentialsRequest to provision the AWS credentials for the AWS Load Balancer Operator. An example for <albo_role_arn> is arn:aws:iam::<aws_account_number>:role/albo-operator.

   Note

   The AWS Load Balancer Operator waits until the secret is created before moving to the Available status.

Procedure

1. Create a YAML file that defines the AWSLoadBalancerController object:

   Example sample-aws-lb.yaml file

   apiVersion: networking.olm.openshift.io/v1
   kind: AWSLoadBalancerController 1
   metadata:
     name: cluster 2
   spec:
     subnetTagging: Auto 3
     additionalResourceTags: 4
     - key: example.org/security-scope
       value: staging
     ingressClass: alb 5
     config:
       replicas: 2 6
     enabledAddons: 7
       - AWSWAFv2 8

   1

   Defines the AWSLoadBalancerController object.

   2

   Defines the AWS Load Balancer Controller name. This instance name gets added as a suffix to all related resources.

   3

   Configures the subnet tagging method for the AWS Load Balancer Controller. The following values are valid:

   • Auto: The AWS Load Balancer Operator determines the subnets that belong to the cluster and tags them appropriately. The Operator cannot determine the role correctly if the internal subnet tags are not present on internal subnet.
   • Manual: You manually tag the subnets that belong to the cluster with the appropriate role tags. Use this option if you installed your cluster on user-provided infrastructure.

   4

   Defines the tags used by the AWS Load Balancer Controller when it provisions AWS resources.

   5

   Defines the ingress class name. The default value is alb.

   6

   Specifies the number of replicas of the AWS Load Balancer Controller.

   7

   Specifies annotations as an add-on for the AWS Load Balancer Controller.

   8

   Enables the alb.ingress.kubernetes.io/wafv2-acl-arn annotation.

2. Create the AWSLoadBalancerController object by running the following command:

   $ oc create -f sample-aws-lb.yaml

3. Create a YAML file that defines the Deployment resource:

   Example sample-aws-lb.yaml file

   apiVersion: apps/v1
   kind: Deployment 1
   metadata:
     name: <echoserver> 2
     namespace: echoserver
   spec:
     selector:
       matchLabels:
         app: echoserver
     replicas: 3 3
     template:
       metadata:
         labels:
           app: echoserver
       spec:
         containers:
           - image: openshift/origin-node
             command:
              - "/bin/socat"
             args:
               - TCP4-LISTEN:8080,reuseaddr,fork
               - EXEC:'/bin/bash -c \"printf \\\"HTTP/1.0 200 OK\r\n\r\n\\\"; sed -e \\\"/^\r/q\\\"\"'
             imagePullPolicy: Always
             name: echoserver
             ports:
               - containerPort: 8080

   1

   Defines the deployment resource.

   2

   Specifies the deployment name.

   3

   Specifies the number of replicas of the deployment.

4. Create a YAML file that defines the Service resource:

   Example service-albo.yaml file

   apiVersion: v1
   kind: Service 1
   metadata:
     name: <echoserver> 2
     namespace: echoserver
   spec:
     ports:
       - port: 80
         targetPort: 8080
         protocol: TCP
     type: NodePort
     selector:
       app: echoserver

   1

   Defines the service resource.

   2

   Specifies the service name.

5. Create a YAML file that defines the Ingress resource:

   Example ingress-albo.yaml file

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: <name> 1
     namespace: echoserver
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: instance
   spec:
     ingressClassName: alb
     rules:
       - http:
           paths:
             - path: /
               pathType: Exact
               backend:
                 service:
                   name: <echoserver> 2
                   port:
                     number: 80

   1

   Specify a name for the Ingress resource.

   2

   Specifies the service name.

Procedure

1. Create an IngressClassParams resource YAML file, for example, sample-single-lb-params.yaml, as follows:

   apiVersion: elbv2.k8s.aws/v1beta1 1
   kind: IngressClassParams
   metadata:
     name: single-lb-params 2
   spec:
     group:
       name: single-lb 3

   1

   Defines the API group and version of the IngressClassParams resource.

   2

   Specifies the IngressClassParams resource name.

   3

   Specifies the IngressGroup resource name. All of the Ingress resources of this class belong to this IngressGroup.

2. Create the IngressClassParams resource by running the following command:

   $ oc create -f sample-single-lb-params.yaml

3. Create the IngressClass resource YAML file, for example, sample-single-lb-class.yaml, as follows:

   apiVersion: networking.k8s.io/v1 1
   kind: IngressClass
   metadata:
     name: single-lb 2
   spec:
     controller: ingress.k8s.aws/alb 3
     parameters:
       apiGroup: elbv2.k8s.aws 4
       kind: IngressClassParams 5
       name: single-lb-params 6

   1

   Defines the API group and version of the IngressClass resource.

   2

   Specifies the ingress class name.

   3

   Defines the controller name. The ingress.k8s.aws/alb value denotes that all ingress resources of this class should be managed by the AWS Load Balancer Controller.

   4

   Defines the API group of the IngressClassParams resource.

   5

   Defines the resource type of the IngressClassParams resource.

   6

   Defines the IngressClassParams resource name.

4. Create the IngressClass resource by running the following command:

   $ oc create -f sample-single-lb-class.yaml

5. Create the AWSLoadBalancerController resource YAML file, for example, sample-single-lb.yaml, as follows:

   apiVersion: networking.olm.openshift.io/v1
   kind: AWSLoadBalancerController
   metadata:
     name: cluster
   spec:
     subnetTagging: Auto
     ingressClass: single-lb 1

   1

   Defines the name of the IngressClass resource.

6. Create the AWSLoadBalancerController resource by running the following command:

   $ oc create -f sample-single-lb.yaml

7. Create the Ingress resource YAML file, for example, sample-multiple-ingress.yaml, as follows:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: example-1 1
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing 2
       alb.ingress.kubernetes.io/group.order: "1" 3
       alb.ingress.kubernetes.io/target-type: instance 4
   spec:
     ingressClassName: single-lb 5
     rules:
     - host: example.com 6
       http:
           paths:
           - path: /blog 7
             pathType: Prefix
             backend:
               service:
                 name: example-1 8
                 port:
                   number: 80 9
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: example-2
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/group.order: "2"
       alb.ingress.kubernetes.io/target-type: instance
   spec:
     ingressClassName: single-lb
     rules:
     - host: example.com
       http:
           paths:
           - path: /store
             pathType: Prefix
             backend:
               service:
                 name: example-2
                 port:
                   number: 80
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: example-3
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/group.order: "3"
       alb.ingress.kubernetes.io/target-type: instance
   spec:
     ingressClassName: single-lb
     rules:
     - host: example.com
       http:
           paths:
           - path: /
             pathType: Prefix
             backend:
               service:
                 name: example-3
                 port:
                   number: 80

   1

   Specifies the ingress name.

   2

   Indicates the load balancer to provision in the public subnet to access the internet.

   3

   Specifies the order in which the rules from the multiple ingress resources are matched when the request is received at the load balancer.

   4

   Indicates that the load balancer will target OpenShift Container Platform nodes to reach the service.

   5

   Specifies the ingress class that belongs to this ingress.

   6

   Defines a domain name used for request routing.

   7

   Defines the path that must route to the service.

   8

   Defines the service name that serves the endpoint configured in the Ingress resource.

   9

   Defines the port on the service that serves the endpoint.

8. Create the Ingress resource by running the following command:

   $ oc create -f sample-multiple-ingress.yaml

Procedure

1. Create a YAML file that defines the AWSLoadBalancerController resource:

   Example add-tls-termination-albc.yaml file

   apiVersion: networking.olm.openshift.io/v1
   kind: AWSLoadBalancerController
   metadata:
     name: cluster
   spec:
     subnetTagging: Auto
     ingressClass: tls-termination 1

   1

   Defines the ingress class name. If the ingress class is not present in your cluster the AWS Load Balancer Controller creates one. The AWS Load Balancer Controller reconciles the additional ingress class values if spec.controller is set to ingress.k8s.aws/alb.

2. Create a YAML file that defines the Ingress resource:

   Example add-tls-termination-ingress.yaml file

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: <example> 1
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing 2
       alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:xxxxx 3
   spec:
     ingressClassName: tls-termination 4
     rules:
     - host: <example.com> 5
       http:
           paths:
             - path: /
               pathType: Exact
               backend:
                 service:
                   name: <example-service> 6
                   port:
                     number: 80

   1

   Specifies the ingress name.

   2

   The controller provisions the load balancer for ingress in a public subnet to access the load balancer over the internet.

   3

   The Amazon Resource Name (ARN) of the certificate that you attach to the load balancer.

   4

   Defines the ingress class name.

   5

   Defines the domain for traffic routing.

   6

   Defines the service for traffic routing.

1. Create the config map to contain the certificate authority (CA) bundle in the aws-load-balancer-operator namespace by running the following command:

   $ oc -n aws-load-balancer-operator create configmap trusted-ca

2. To inject the trusted CA bundle into the config map, add the config.openshift.io/inject-trusted-cabundle=true label to the config map by running the following command:

   $ oc -n aws-load-balancer-operator label cm trusted-ca config.openshift.io/inject-trusted-cabundle=true

3. Update the AWS Load Balancer Operator subscription to access the config map in the AWS Load Balancer Operator deployment by running the following command:

   $ oc -n aws-load-balancer-operator patch subscription aws-load-balancer-operator --type='merge' -p '{"spec":{"config":{"env":[{"name":"TRUSTED_CA_CONFIGMAP_NAME","value":"trusted-ca"}],"volumes":[{"name":"trusted-ca","configMap":{"name":"trusted-ca"}}],"volumeMounts":[{"name":"trusted-ca","mountPath":"/etc/pki/tls/certs/albo-tls-ca-bundle.crt","subPath":"ca-bundle.crt"}]}}}'

4. After the AWS Load Balancer Operator is deployed, verify that the CA bundle is added to the aws-load-balancer-operator-controller-manager deployment by running the following command:

   $ oc -n aws-load-balancer-operator exec deploy/aws-load-balancer-operator-controller-manager -c manager -- bash -c "ls -l /etc/pki/tls/certs/albo-tls-ca-bundle.crt; printenv TRUSTED_CA_CONFIGMAP_NAME"

   Example output

   -rw-r--r--. 1 root 1000690000 5875 Jan 11 12:25 /etc/pki/tls/certs/albo-tls-ca-bundle.crt
   trusted-ca

5. Optional: Restart deployment of the AWS Load Balancer Operator every time the config map changes by running the following command:

   $ oc -n aws-load-balancer-operator rollout restart deployment/aws-load-balancer-operator-controller-manager

1. Check the name of an install plan by running the following command:

   $ oc -n external-dns-operator get sub external-dns-operator -o yaml | yq '.status.installplan.name'

   Example output

   install-zcvlr

2. Check if the status of an install plan is Complete by running the following command:

   $ oc -n external-dns-operator get ip <install_plan_name> -o yaml | yq '.status.phase'

   Example output

   Complete

3. View the status of the external-dns-operator deployment by running the following command:

   $ oc get -n external-dns-operator deployment/external-dns-operator

   Example output

   NAME                    READY     UP-TO-DATE   AVAILABLE   AGE
   external-dns-operator   1/1       1            1           23h

Procedure

1. View the logs of the External DNS Operator by running the following command:

   $ oc logs -n external-dns-operator deployment/external-dns-operator -c external-dns-operator

Procedure

1. Click Operators OperatorHub in the OpenShift Container Platform web console.
2. Click External DNS Operator. You can use the Filter by keyword text box or the filter list to search for External DNS Operator from the list of Operators.
3. Select the external-dns-operator namespace.
4. On the External DNS Operator page, click Install.

5. On the Install Operator page, ensure that you selected the following options:

   1. Update the channel as stable-v1.
   2. Installation mode as A specific name on the cluster.
   3. Installed namespace as external-dns-operator. If namespace external-dns-operator does not exist, it gets created during the Operator installation.
   4. Select Approval Strategy as Automatic or Manual. Approval Strategy is set to Automatic by default.
   5. Click Install.

Procedure

1. Create a Namespace object:

   1. Create a YAML file that defines the Namespace object:

      Example namespace.yaml file

      apiVersion: v1
      kind: Namespace
      metadata:
        name: external-dns-operator

   2. Create the Namespace object by running the following command:

      $ oc apply -f namespace.yaml

2. Create an OperatorGroup object:

   1. Create a YAML file that defines the OperatorGroup object:

      Example operatorgroup.yaml file

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: external-dns-operator
        namespace: external-dns-operator
      spec:
        upgradeStrategy: Default
        targetNamespaces:
        - external-dns-operator

   2. Create the OperatorGroup object by running the following command:

      $ oc apply -f operatorgroup.yaml

3. Create a Subscription object:

   1. Create a YAML file that defines the Subscription object:

      Example subscription.yaml file

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: external-dns-operator
        namespace: external-dns-operator
      spec:
        channel: stable-v1
        installPlanApproval: Automatic
        name: external-dns-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the Subscription object by running the following command:

      $ oc apply -f subscription.yaml

Verification

1. Get the name of the install plan from the subscription by running the following command:

   $ oc -n external-dns-operator \
       get subscription external-dns-operator \
       --template='{{.status.installplan.name}}{{"\n"}}'

2. Verify that the status of the install plan is Complete by running the following command:

   $ oc -n external-dns-operator \
       get ip <install_plan_name> \
       --template='{{.status.phase}}{{"\n"}}'

3. Verify that the status of the external-dns-operator pod is Running by running the following command:

   $ oc -n external-dns-operator get pod

   Example output

   NAME                                     READY   STATUS    RESTARTS   AGE
   external-dns-operator-5584585fd7-5lwqm   2/2     Running   0          11m

4. Verify that the catalog source of the subscription is redhat-operators by running the following command:

   $ oc -n external-dns-operator get subscription

   Example output

   NAME                    PACKAGE                 SOURCE             CHANNEL
   external-dns-operator   external-dns-operator   redhat-operators   stable-v1

5. Check the external-dns-operator version by running the following command:

   $ oc -n external-dns-operator get csv

   Example output

   NAME                           DISPLAY                VERSION   REPLACES   PHASE
   external-dns-operator.v<1.y.z>   ExternalDNS Operator   <1.y.z>                Succeeded

Procedure

1. Check the user. The user must have access to the kube-system namespace. If you don’t have the credentials, as you can fetch the credentials from the kube-system namespace to use the cloud provider client:

   $ oc whoami

   Example output

   system:admin

2. Fetch the values from aws-creds secret present in kube-system namespace.

   $ export AWS_ACCESS_KEY_ID=$(oc get secrets aws-creds -n kube-system  --template={{.data.aws_access_key_id}} | base64 -d)
   $ export AWS_SECRET_ACCESS_KEY=$(oc get secrets aws-creds -n kube-system  --template={{.data.aws_secret_access_key}} | base64 -d)

3. Get the routes to check the domain:

   $ oc get routes --all-namespaces | grep console

   Example output

   openshift-console          console             console-openshift-console.apps.testextdnsoperator.apacshift.support                       console             https   reencrypt/Redirect     None
   openshift-console          downloads           downloads-openshift-console.apps.testextdnsoperator.apacshift.support                     downloads           http    edge/Redirect          None

4. Get the list of dns zones to find the one which corresponds to the previously found route’s domain:

   $ aws route53 list-hosted-zones | grep testextdnsoperator.apacshift.support

   Example output

   HOSTEDZONES	terraform	/hostedzone/Z02355203TNN1XXXX1J6O	testextdnsoperator.apacshift.support.	5

5. Create ExternalDNS resource for route source:

   $ cat <<EOF | oc create -f -
   apiVersion: externaldns.olm.openshift.io/v1beta1
   kind: ExternalDNS
   metadata:
     name: sample-aws 1
   spec:
     domains:
     - filterType: Include   2
       matchType: Exact   3
       name: testextdnsoperator.apacshift.support 4
     provider:
       type: AWS 5
     source:  6
       type: OpenShiftRoute 7
       openshiftRouteOptions:
         routerName: default 8
   EOF

   1

   Defines the name of external DNS resource.

   2

   By default all hosted zones are selected as potential targets. You can include a hosted zone that you need.

   3

   The matching of the target zone’s domain has to be exact (as opposed to regular expression match).

   4

   Specify the exact domain of the zone you want to update. The hostname of the routes must be subdomains of the specified domain.

   5

   Defines the AWS Route53 DNS provider.

   6

   Defines options for the source of DNS records.

   7

   Defines OpenShift route resource as the source for the DNS records which gets created in the previously specified DNS provider.

   8

   If the source is OpenShiftRoute, then you can pass the OpenShift Ingress Controller name. External DNS Operator selects the canonical hostname of that router as the target while creating CNAME record.

6. Check the records created for OCP routes using the following command:

   $ aws route53 list-resource-record-sets --hosted-zone-id Z02355203TNN1XXXX1J6O --query "ResourceRecordSets[?Type == 'CNAME']" | grep console

Procedure

1. Get the Role ARN of the IAM Role that you created to allow Account B to access Account A’s Route 53 hosted zone by running the following command:

   $ aws --profile account-a iam get-role --role-name user-rol1 | head -1

   Example output

   ROLE	arn:aws:iam::1234567890123:role/user-rol1	2023-09-14T17:21:54+00:00	3600	/	AROA3SGB2ZRKRT5NISNJN	user-rol1

2. Locate the private hosted zone to use with Account A’s credentials by running the following command:

   $ aws --profile account-a route53 list-hosted-zones | grep testextdnsoperator.apacshift.support

   Example output

   HOSTEDZONES	terraform	/hostedzone/Z02355203TNN1XXXX1J6O	testextdnsoperator.apacshift.support. 5

3. Create the ExternalDNS object by running the following command:

   $ cat <<EOF | oc create -f -
   apiVersion: externaldns.olm.openshift.io/v1beta1
   kind: ExternalDNS
   metadata:
     name: sample-aws
   spec:
     domains:
     - filterType: Include
       matchType: Exact
       name: testextdnsoperator.apacshift.support
     provider:
       type: AWS
       aws:
         assumeRole:
           arn: arn:aws:iam::12345678901234:role/user-rol1 1
     source:
       type: OpenShiftRoute
       openshiftRouteOptions:
         routerName: default
   EOF

   1

   Specify the Role ARN to have DNS records created in Account A.

4. Check the records created for OpenShift Container Platform (OCP) routes by using the following command:

   $ aws --profile account-a route53 list-resource-record-sets --hosted-zone-id Z02355203TNN1XXXX1J6O --query "ResourceRecordSets[?Type == 'CNAME']" | grep console-openshift-console

Procedure

1. Fetch the credentials from the kube-system namespace to use the cloud provider client by running the following command:

   $ CLIENT_ID=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_client_id}} | base64 -d)
   $ CLIENT_SECRET=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_client_secret}} | base64 -d)
   $ RESOURCE_GROUP=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_resourcegroup}} | base64 -d)
   $ SUBSCRIPTION_ID=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_subscription_id}} | base64 -d)
   $ TENANT_ID=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_tenant_id}} | base64 -d)

2. Log in to Azure by running the following command:

   $ az login --service-principal -u "${CLIENT_ID}" -p "${CLIENT_SECRET}" --tenant "${TENANT_ID}"

3. Get a list of routes by running the following command:

   $ oc get routes --all-namespaces | grep console

   Example output

   openshift-console          console             console-openshift-console.apps.test.azure.example.com                       console             https   reencrypt/Redirect     None
   openshift-console          downloads           downloads-openshift-console.apps.test.azure.example.com                     downloads           http    edge/Redirect          None

4. Get a list of DNS zones by running the following command:

   $ az network dns zone list --resource-group "${RESOURCE_GROUP}"

5. Create a YAML file, for example, external-dns-sample-azure.yaml, that defines the ExternalDNS object:

   Example external-dns-sample-azure.yaml file

   apiVersion: externaldns.olm.openshift.io/v1beta1
   kind: ExternalDNS
   metadata:
     name: sample-azure 1
   spec:
     zones:
     - "/subscriptions/1234567890/resourceGroups/test-azure-xxxxx-rg/providers/Microsoft.Network/dnszones/test.azure.example.com" 2
     provider:
       type: Azure 3
     source:
       openshiftRouteOptions: 4
         routerName: default 5
       type: OpenShiftRoute 6

   1

   Specifies the External DNS name.

   2

   Defines the zone ID.

   3

   Defines the provider type.

   4

   You can define options for the source of DNS records.

   5

   If the source type is OpenShiftRoute, you can pass the OpenShift Ingress Controller name. External DNS selects the canonical hostname of that router as the target while creating CNAME record.

   6

   Defines the route resource as the source for the Azure DNS records.

6. Check the DNS records created for OpenShift Container Platform routes by running the following command:

   $ az network dns record-set list -g "${RESOURCE_GROUP}"  -z test.azure.example.com | grep console

   Note

   To create records on private hosted zones on private Azure DNS, you need to specify the private zone under the zones field which populates the provider type to azure-private-dns in the ExternalDNS container arguments.

Procedure

1. Copy the gcp-credentials secret in the encoded-gcloud.json file by running the following command:

   $ oc get secret gcp-credentials -n kube-system --template='{{$v := index .data "service_account.json"}}{{$v}}' | base64 -d - > decoded-gcloud.json

2. Export your Google credentials by running the following command:

   $ export GOOGLE_CREDENTIALS=decoded-gcloud.json

3. Activate your account by using the following command:

   $ gcloud auth activate-service-account  <client_email as per decoded-gcloud.json> --key-file=decoded-gcloud.json

4. Set your project by running the following command:

   $ gcloud config set project <project_id as per decoded-gcloud.json>

5. Get a list of routes by running the following command:

   $ oc get routes --all-namespaces | grep console

   Example output

   openshift-console          console             console-openshift-console.apps.test.gcp.example.com                       console             https   reencrypt/Redirect     None
   openshift-console          downloads           downloads-openshift-console.apps.test.gcp.example.com                     downloads           http    edge/Redirect          None

6. Get a list of managed zones by running the following command:

   $ gcloud dns managed-zones list | grep test.gcp.example.com

   Example output

   qe-cvs4g-private-zone test.gcp.example.com

7. Create a YAML file, for example, external-dns-sample-gcp.yaml, that defines the ExternalDNS object:

   Example external-dns-sample-gcp.yaml file

   apiVersion: externaldns.olm.openshift.io/v1beta1
   kind: ExternalDNS
   metadata:
     name: sample-gcp 1
   spec:
     domains:
       - filterType: Include 2
         matchType: Exact 3
         name: test.gcp.example.com 4
     provider:
       type: GCP 5
     source:
       openshiftRouteOptions: 6
         routerName: default 7
       type: OpenShiftRoute 8

   1

   Specifies the External DNS name.

   2

   By default, all hosted zones are selected as potential targets. You can include your hosted zone.

   3

   The domain of the target must match the string defined by the name key.

   4

   Specify the exact domain of the zone you want to update. The hostname of the routes must be subdomains of the specified domain.

   5

   Defines the provider type.

   6

   You can define options for the source of DNS records.

   7

   If the source type is OpenShiftRoute, you can pass the OpenShift Ingress Controller name. External DNS selects the canonical hostname of that router as the target while creating CNAME record.

   8

   Defines the route resource as the source for GCP DNS records.

8. Check the DNS records created for OpenShift Container Platform routes by running the following command:

   $ gcloud dns record-sets list --zone=qe-cvs4g-private-zone | grep console

Procedure

1. Create a secret object with Infoblox credentials by running the following command:

   $ oc -n external-dns-operator create secret generic infoblox-credentials --from-literal=EXTERNAL_DNS_INFOBLOX_WAPI_USERNAME=<infoblox_username> --from-literal=EXTERNAL_DNS_INFOBLOX_WAPI_PASSWORD=<infoblox_password>

2. Get a list of routes by running the following command:

   $ oc get routes --all-namespaces | grep console

   Example Output

   openshift-console          console             console-openshift-console.apps.test.example.com                       console             https   reencrypt/Redirect     None
   openshift-console          downloads           downloads-openshift-console.apps.test.example.com                     downloads           http    edge/Redirect          None

3. Create a YAML file, for example, external-dns-sample-infoblox.yaml, that defines the ExternalDNS object:

   Example external-dns-sample-infoblox.yaml file

   apiVersion: externaldns.olm.openshift.io/v1beta1
   kind: ExternalDNS
   metadata:
     name: sample-infoblox 1
   spec:
     provider:
       type: Infoblox 2
       infoblox:
         credentials:
           name: infoblox-credentials
         gridHost: ${INFOBLOX_GRID_PUBLIC_IP}
         wapiPort: 443
         wapiVersion: "2.3.1"
     domains:
     - filterType: Include
       matchType: Exact
       name: test.example.com
     source:
       type: OpenShiftRoute 3
       openshiftRouteOptions:
         routerName: default 4

   1

   Specifies the External DNS name.

   2

   Defines the provider type.

   3

   You can define options for the source of DNS records.

   4

   If the source type is OpenShiftRoute, you can pass the OpenShift Ingress Controller name. External DNS selects the canonical hostname of that router as the target while creating CNAME record.

4. Create the ExternalDNS resource on Infoblox by running the following command:

   $ oc create -f external-dns-sample-infoblox.yaml

5. From the Infoblox UI, check the DNS records created for console routes:

   1. Click Data Management DNS Zones.
   2. Select the zone name.

Procedure

1. Create the config map to contain the CA bundle in the external-dns-operator namespace by running the following command:

   $ oc -n external-dns-operator create configmap trusted-ca

2. To inject the trusted CA bundle into the config map, add the config.openshift.io/inject-trusted-cabundle=true label to the config map by running the following command:

   $ oc -n external-dns-operator label cm trusted-ca config.openshift.io/inject-trusted-cabundle=true

3. Update the subscription of the External DNS Operator by running the following command:

   $ oc -n external-dns-operator patch subscription external-dns-operator --type='json' -p='[{"op": "add", "path": "/spec/config", "value":{"env":[{"name":"TRUSTED_CA_CONFIGMAP_NAME","value":"trusted-ca"}]}}]'

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators OperatorHub.

2. Type a keyword into the Filter by keyword box or scroll to find the Operator you want. For example, type metallb to find the MetalLB Operator.

   You can also filter options by Infrastructure Features. For example, select Disconnected if you want to see Operators that work in disconnected environments, also known as restricted network environments.

3. On the Install Operator page, accept the defaults and click Install.

Verification

1. To confirm that the installation is successful:

   1. Navigate to the Operators Installed Operators page.
   2. Check that the Operator is installed in the openshift-operators namespace and that its status is Succeeded.

2. If the Operator is not installed successfully, check the status of the Operator and review the logs:

   1. Navigate to the Operators Installed Operators page and inspect the Status column for any errors or failures.
   2. Navigate to the Workloads Pods page and check the logs in any pods in the openshift-operators project that are reporting issues.

Procedure

1. Create a namespace for the MetalLB Operator by entering the following command:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: metallb-system
   EOF

2. Create an Operator group custom resource (CR) in the namespace:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: metallb-operator
     namespace: metallb-system
   EOF

3. Confirm the Operator group is installed in the namespace:

   $ oc get operatorgroup -n metallb-system

   Example output

   NAME               AGE
   metallb-operator   14m

4. Create a Subscription CR:

   1. Define the Subscription CR and save the YAML file, for example, metallb-sub.yaml:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: metallb-operator-sub
        namespace: metallb-system
      spec:
        channel: stable
        name: metallb-operator
        source: redhat-operators 1
        sourceNamespace: openshift-marketplace

      1

      You must specify the redhat-operators value.

   2. To create the Subscription CR, run the following command:

      $ oc create -f metallb-sub.yaml

5. Optional: To ensure BGP and BFD metrics appear in Prometheus, you can label the namespace as in the following command:

   $ oc label ns metallb-system "openshift.io/cluster-monitoring=true"

1. Confirm the install plan is in the namespace:

   $ oc get installplan -n metallb-system

   Example output

   NAME            CSV                                   APPROVAL    APPROVED
   install-wzg94   metallb-operator.4.16.0-nnnnnnnnnnnn   Automatic   true

   Note

   Installation of the Operator might take a few seconds.

2. To verify that the Operator is installed, enter the following command:

   $ oc get clusterserviceversion -n metallb-system \
     -o custom-columns=Name:.metadata.name,Phase:.status.phase

   Example output

   Name                                  Phase
   metallb-operator.4.16.0-nnnnnnnnnnnn   Succeeded

1. Create a single instance of a MetalLB custom resource:

   $ cat << EOF | oc apply -f -
   apiVersion: metallb.io/v1beta1
   kind: MetalLB
   metadata:
     name: metallb
     namespace: metallb-system
   EOF

1. Verify that the deployment for the controller is running:

   $ oc get deployment -n metallb-system controller

   Example output

   NAME         READY   UP-TO-DATE   AVAILABLE   AGE
   controller   1/1     1            1           11m

2. Verify that the daemon set for the speaker is running:

   $ oc get daemonset -n metallb-system speaker

   Example output

   NAME      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
   speaker   6         6         6       6            6           kubernetes.io/os=linux   18m

   The example output indicates 6 speaker pods. The number of speaker pods in your cluster might differ from the example output. Make sure the output indicates one pod for each node in your cluster.

Procedure

1. Navigate to the Operators Installed Operators page.
2. Search for the MetalLB Operator. Then, click on it.

3. On the right side of the Operator Details page, select Uninstall Operator from the Actions drop-down menu.

   An Uninstall Operator? dialog box is displayed.

4. Select Uninstall to remove the Operator, Operator deployments, and pods. Following this action, the Operator stops running and no longer receives updates.

   Note

   This action does not remove resources managed by the Operator, including custom resource definitions (CRDs) and custom resources (CRs). Dashboards and navigation items enabled by the web console and off-cluster resources that continue to run might need manual clean up. To remove these after uninstalling the Operator, you might need to manually delete the Operator CRDs.

Procedure

1. Check the current version of the subscribed MetalLB Operator in the currentCSV field:

   $ oc get subscription metallb-operator -n metallb-system -o yaml | grep currentCSV

   Example output

     currentCSV: metallb-operator.4.10.0-202207051316

2. Delete the subscription:

   $ oc delete subscription metallb-operator -n metallb-system

   Example output

   subscription.operators.coreos.com "metallb-operator" deleted

3. Delete the CSV for the Operator in the target namespace using the currentCSV value from the previous step:

   $ oc delete clusterserviceversion metallb-operator.4.10.0-202207051316 -n metallb-system

   Example output

   clusterserviceversion.operators.coreos.com "metallb-operator.4.10.0-202207051316" deleted

Procedure

1. List the Operator groups in the metallb-system namespace by running the following command:

   $ oc get operatorgroup -n metallb-system

   Example output

   NAME                   AGE
   metallb-system-7jc66   85m

2. Verify that the spec.targetNamespaces is present in the Operator group CR associated with the metallb-system namespace by running the following command:

   $ oc get operatorgroup metallb-system-7jc66 -n metallb-system -o yaml

   Example output

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     annotations:
       olm.providedAPIs: ""
     creationTimestamp: "2023-10-25T09:42:49Z"
     generateName: metallb-system-
     generation: 1
     name: metallb-system-7jc66
     namespace: metallb-system
     resourceVersion: "25027"
     uid: f5f644a0-eef8-4e31-a306-e2bbcfaffab3
   spec:
     targetNamespaces:
     - metallb-system
     upgradeStrategy: Default
   status:
     lastUpdated: "2023-10-25T09:42:49Z"
     namespaces:
     - metallb-system

3. Edit the Operator group and remove the targetNamespaces and metallb-system present under the spec section by running the following command:

   $ oc edit n metallb-system

   Example output

   operatorgroup.operators.coreos.com/metallb-system-7jc66 edited

4. Verify the spec.targetNamespaces is removed from the Operator group custom resource associated with the metallb-system namespace by running the following command:

   $ oc get operatorgroup metallb-system-7jc66 -n metallb-system -o yaml

   Example output

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     annotations:
       olm.providedAPIs: ""
     creationTimestamp: "2023-10-25T09:42:49Z"
     generateName: metallb-system-
     generation: 2
     name: metallb-system-7jc66
     namespace: metallb-system
     resourceVersion: "61658"
     uid: f5f644a0-eef8-4e31-a306-e2bbcfaffab3
   spec:
     upgradeStrategy: Default
   status:
     lastUpdated: "2023-10-25T14:31:30Z"
     namespaces:
     - ""

Procedure

1. Verify that the metallb-system namespace still exists:

   $ oc get namespaces | grep metallb-system

   Example output

   metallb-system                                     Active   31m

2. Verify the metallb custom resource still exists:

   $ oc get metallb -n metallb-system

   Example output

   NAME      AGE
   metallb   33m

3. Follow the guidance in "Installing from OperatorHub using the CLI" to install the latest 4.16 version of the MetalLB Operator.

   Note

   When installing the latest 4.16 version of the MetalLB Operator, you must install the Operator to the same namespace it was previously installed to.

4. Verify the upgraded version of the Operator is now the 4.16 version.

   $ oc get csv -n metallb-system

   Example output

   NAME                                   DISPLAY            VERSION               REPLACES   PHASE
   metallb-operator.4.16.0-202207051316   MetalLB Operator   4.16.0-202207051316              Succeeded

Procedure

1. Change managementState to Unmanaged in the DNS Operator:

   oc patch dns.operator.openshift.io default --type merge --patch '{"spec":{"managementState":"Unmanaged"}}'

2. Review managementState of the DNS Operator using the jsonpath command line JSON parser:

   $ oc get dns.operator.openshift.io default -ojsonpath='{.spec.managementState}'

   Example output

   "Unmanaged"

Procedure

1. Modify the DNS Operator object named default:

   $ oc edit dns.operator/default

   Cluster administrators can configure transport layer security (TLS) for forwarded DNS queries.

   Configuring DNS forwarding with TLS

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     name: default
   spec:
     servers:
     - name: example-server 1
       zones:
       - example.com 2
       forwardPlugin:
         transportConfig:
           transport: TLS 3
           tls:
             caBundle:
               name: mycacert
             serverName: dnstls.example.com  4
         policy: Random 5
         upstreams: 6
         - 1.1.1.1
         - 2.2.2.2:5353
     upstreamResolvers: 7
       transportConfig:
         transport: TLS
         tls:
           caBundle:
             name: mycacert
           serverName: dnstls.example.com
       upstreams:
       - type: Network 8
         address: 1.2.3.4 9
         port: 53 10

   1

   Must comply with the rfc6335 service name syntax.

   2

   Must conform to the definition of a subdomain in the rfc1123 service name syntax. The cluster domain, cluster.local, is an invalid subdomain for the zones field. The cluster domain, cluster.local, is an invalid subdomain for zones.

   3

   When configuring TLS for forwarded DNS queries, set the transport field to have the value TLS.

   4

   When configuring TLS for forwarded DNS queries, this is a mandatory server name used as part of the server name indication (SNI) to validate the upstream TLS server certificate.

   5

   Defines the policy to select upstream resolvers. Default value is Random. You can also use the values RoundRobin, and Sequential.

   6

   Required. Use it to provide upstream resolvers. A maximum of 15 upstreams entries are allowed per forwardPlugin entry.

   7

   Optional. You can use it to override the default policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. If you do not provide any upstream resolvers, the DNS name queries go to the servers in /etc/resolv.conf.

   8

   Only the Network type is allowed when using TLS and you must provide an IP address. Network type indicates that this upstream resolver should handle forwarded requests separately from the upstream resolvers listed in /etc/resolv.conf.

   9

   The address field must be a valid IPv4 or IPv6 address.

   10

   You can optionally provide a port. The port must have a value between 1 and 65535. If you do not specify a port for the upstream, the default port is 853.

   Note

   If servers is undefined or invalid, the config map only contains the default server.

Verification

1. View the config map:

   $ oc get configmap/dns-default -n openshift-dns -o yaml

   Sample DNS ConfigMap based on TLS forwarding example

   apiVersion: v1
   data:
     Corefile: |
       example.com:5353 {
           forward . 1.1.1.1 2.2.2.2:5353
       }
       bar.com:5353 example.com:5353 {
           forward . 3.3.3.3 4.4.4.4:5454 1
       }
       .:5353 {
           errors
           health
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods insecure
               upstream
               fallthrough in-addr.arpa ip6.arpa
           }
           prometheus :9153
           forward . /etc/resolv.conf 1.2.3.4:53 {
               policy Random
           }
           cache 30
           reload
       }
   kind: ConfigMap
   metadata:
     labels:
       dns.operator.openshift.io/owning-dns: default
     name: dns-default
     namespace: openshift-dns

   1

   Changes to the forwardPlugin triggers a rolling update of the CoreDNS daemon set.

1. Create a Secret resource containing the custom certificate in the openshift-ingress namespace using the tls.crt and tls.key files.

   $ oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key

2. Update the IngressController CR to reference the new certificate secret:

   $ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
     --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'

3. Verify the update was effective:

   $ echo Q |\
     openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null |\
     openssl x509 -noout -subject -issuer -enddate

   where:

   <domain>

   Specifies the base domain name for your cluster.

   Example output

   subject=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = *.apps.example.com
   issuer=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = example.com
   notAfter=May 10 08:32:45 2022 GM

   Tip

   You can alternatively apply the following YAML to set a custom default certificate:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     defaultCertificate:
       name: custom-certs-default

   The certificate secret name should match the value used to update the CR.

Procedure

1. Create a service account to authenticate with Thanos by running the following command:

   $ oc create -n openshift-ingress-operator serviceaccount thanos && oc describe -n openshift-ingress-operator serviceaccount thanos

   Example output

   Name:                thanos
   Namespace:           openshift-ingress-operator
   Labels:              <none>
   Annotations:         <none>
   Image pull secrets:  thanos-dockercfg-kfvf2
   Mountable secrets:   thanos-dockercfg-kfvf2
   Tokens:              <none>
   Events:              <none>

2. Manually create the service account secret token with the following command:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: thanos-token
     namespace: openshift-ingress-operator
     annotations:
       kubernetes.io/service-account.name: thanos
   type: kubernetes.io/service-account-token
   EOF

3. Define a TriggerAuthentication object within the openshift-ingress-operator namespace by using the service account’s token.

   1. Create the TriggerAuthentication object and pass the value of the secret variable to the TOKEN parameter:

      $ oc apply -f - <<EOF
      apiVersion: keda.sh/v1alpha1
      kind: TriggerAuthentication
      metadata:
        name: keda-trigger-auth-prometheus
        namespace: openshift-ingress-operator
      spec:
        secretTargetRef:
        - parameter: bearerToken
          name: thanos-token
          key: token
        - parameter: ca
          name: thanos-token
          key: ca.crt
      EOF

4. Create and apply a role for reading metrics from Thanos:

   1. Create a new role, thanos-metrics-reader.yaml, that reads metrics from pods and nodes:

      thanos-metrics-reader.yaml

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: thanos-metrics-reader
        namespace: openshift-ingress-operator
      rules:
      - apiGroups:
        - ""
        resources:
        - pods
        - nodes
        verbs:
        - get
      - apiGroups:
        - metrics.k8s.io
        resources:
        - pods
        - nodes
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - ""
        resources:
        - namespaces
        verbs:
        - get

   2. Apply the new role by running the following command:

      $ oc apply -f thanos-metrics-reader.yaml

5. Add the new role to the service account by entering the following commands:

   $ oc adm policy -n openshift-ingress-operator add-role-to-user thanos-metrics-reader -z thanos --role-namespace=openshift-ingress-operator

   $ oc adm policy -n openshift-ingress-operator add-cluster-role-to-user cluster-monitoring-view -z thanos

   Note

   The argument add-cluster-role-to-user is only required if you use cross-namespace queries. The following step uses a query from the kube-metrics namespace which requires this argument.

6. Create a new ScaledObject YAML file, ingress-autoscaler.yaml, that targets the default Ingress Controller deployment:

   Example ScaledObject definition

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: ingress-scaler
     namespace: openshift-ingress-operator
   spec:
     scaleTargetRef: 1
       apiVersion: operator.openshift.io/v1
       kind: IngressController
       name: default
       envSourceContainerName: ingress-operator
     minReplicaCount: 1
     maxReplicaCount: 20 2
     cooldownPeriod: 1
     pollingInterval: 1
     triggers:
     - type: prometheus
       metricType: AverageValue
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 3
         namespace: openshift-ingress-operator 4
         metricName: 'kube-node-role'
         threshold: '1'
         query: 'sum(kube_node_role{role="worker",service="kube-state-metrics"})' 5
         authModes: "bearer"
       authenticationRef:
         name: keda-trigger-auth-prometheus

   1

   The custom resource that you are targeting. In this case, the Ingress Controller.

   2

   Optional: The maximum number of replicas. If you omit this field, the default maximum is set to 100 replicas.

   3

   The Thanos service endpoint in the openshift-monitoring namespace.

   4

   The Ingress Operator namespace.

   5

   This expression evaluates to however many worker nodes are present in the deployed cluster.

   Important

   If you are using cross-namespace queries, you must target port 9091 and not port 9092 in the serverAddress field. You also must have elevated privileges to read metrics from this port.

7. Apply the custom resource definition by running the following command:

   $ oc apply -f ingress-autoscaler.yaml

Procedure

1. View the current number of available replicas for the default IngressController:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   2

2. Scale the default IngressController to the desired number of replicas using the oc patch command. The following example scales the default IngressController to 3 replicas:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"replicas": 3}}' --type=merge

   Example output

   ingresscontroller.operator.openshift.io/default patched

3. Verify that the default IngressController scaled to the number of replicas that you specified:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   3

   Tip

   You can alternatively apply the following YAML to scale an Ingress Controller to three replicas:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: 3               1

   1

   If you need a different amount of replicas, change the replicas value.

Procedure

1. Create an IngressController custom resource (CR) in a file named <name>-ingress-controller.yaml, such as in the following example:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: <name> 1
   spec:
     domain: <domain> 2
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal 3

   1

   Replace <name> with a name for the IngressController object.

   2

   Specify the domain for the application published by the controller.

   3

   Specify a value of Internal to use an internal load balancer.

2. Create the Ingress Controller defined in the previous step by running the following command:

   $ oc create -f <name>-ingress-controller.yaml 1

   1

   Replace <name> with the name of the IngressController object.

3. Optional: Confirm that the Ingress Controller was created by running the following command:

   $ oc --all-namespaces=true get ingresscontrollers

Procedure

1. Configure the Ingress Controller resource to allow global access.

   Note

   You can also create an Ingress Controller and specify the global access option.

   1. Configure the Ingress Controller resource:

      $ oc -n openshift-ingress-operator edit ingresscontroller/default

   2. Edit the YAML file:

      Sample clientAccess configuration to Global

        spec:
          endpointPublishingStrategy:
            loadBalancer:
              providerParameters:
                gcp:
                  clientAccess: Global 1
                type: GCP
              scope: Internal
            type: LoadBalancerService

      1

      Set gcp.clientAccess to Global.

   3. Save the file to apply the changes.

2. Run the following command to verify that the service allows global access:

   $ oc -n openshift-ingress edit svc/router-default -o yaml

   The output shows that global access is enabled for GCP with the annotation, networking.gke.io/internal-load-balancer-allow-global-access.

Procedure

1. Configure the default Ingress Controller for your cluster to be internal by deleting and recreating it.

   $ oc replace --force --wait --filename - <<EOF
   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: default
   spec:
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal
   EOF

Procedure

1. Configure the wildcard policy.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

   2. Under spec, set the wildcardPolicy field to WildcardsDisallowed or WildcardsAllowed:

      spec:
        routeAdmission:
          wildcardPolicy: WildcardsDisallowed # or WildcardsAllowed

Procedure

1. Edit the Ingress Controller resource:

   $ oc -n openshift-ingress-operator edit ingresscontroller/default

2. Replace the X-SSL-Client-Der HTTP request header with the X-Forwarded-Client-Cert HTTP request header:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     httpHeaders:
       actions: 1
         request: 2
         - name: X-Forwarded-Client-Cert 3
           action:
             type: Set 4
             set:
              value: "%{+Q}[ssl_c_der,base64]" 5
         - name: X-SSL-Client-Der
           action:
             type: Delete

   1

   The list of actions you want to perform on the HTTP headers.

   2

   The type of header you want to change. In this case, a request header.

   3

   The name of the header you want to change. For a list of available headers you can set or delete, see HTTP header configuration.

   4

   The type of action being taken on the header. This field can have the value Set or Delete.

   5

   When setting HTTP headers, you must provide a value. The value can be a string from a list of available directives for that header, for example DENY, or it can be a dynamic value that will be interpreted using HAProxy’s dynamic value syntax. In this case, a dynamic value is added.

   Note

   For setting dynamic header values for HTTP responses, allowed sample fetchers are res.hdr and ssl_c_der. For setting dynamic header values for HTTP requests, allowed sample fetchers are req.hdr and ssl_c_der. Both request and response dynamic values can use the lower and base64 converters.

3. Save the file to apply the changes.

Procedure

1. Configure the HTTPHeaders field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

   2. Under spec, set the HTTPHeaders policy field to Append, Replace, IfNone, or Never:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpHeaders:
          forwardedHeaderPolicy: Append

Procedure

1. Edit the Ingress Controller resource by entering the following command in your CLI:

   $ oc -n openshift-ingress-operator edit ingresscontroller/default

2. Set the PROXY configuration:

   • If your Ingress Controller uses the HostNetwork endpoint publishing strategy type, set the spec.endpointPublishingStrategy.hostNetwork.protocol subfield to PROXY:

     Sample hostNetwork configuration to PROXY

     # ...
       spec:
         endpointPublishingStrategy:
           hostNetwork:
             protocol: PROXY
           type: HostNetwork
     # ...

   • If your Ingress Controller uses the NodePortService endpoint publishing strategy type, set the spec.endpointPublishingStrategy.nodePort.protocol subfield to PROXY:

     Sample nodePort configuration to PROXY

     # ...
       spec:
         endpointPublishingStrategy:
           nodePort:
             protocol: PROXY
           type: NodePortService
     # ...

   • If your Ingress Controller uses the Private endpoint publishing strategy type, set the spec.endpointPublishingStrategy.private.protocol subfield to PROXY:

     Sample private configuration to PROXY

     # ...
       spec:
         endpointPublishingStrategy:
           private:
             protocol: PROXY
         type: Private
     # ...

Procedure

1. Configure the appsDomain field by specifying an alternative default domain for user-created routes.

   1. Edit the ingress cluster resource:

      $ oc edit ingresses.config/cluster -o yaml

   2. Edit the YAML file:

      Sample appsDomain configuration to test.example.com

      apiVersion: config.openshift.io/v1
      kind: Ingress
      metadata:
        name: cluster
      spec:
        domain: apps.example.com            1
        appsDomain: <test.example.com>      2

      1

      Specifies the default domain. You cannot modify the default domain after installation.

      2

      Optional: Domain for OpenShift Container Platform infrastructure to use for application routes. Instead of the default prefix, apps, you can use an alternative prefix like test.

2. Verify that an existing route contains the domain name specified in the appsDomain field by exposing the route and verifying the route domain change:

   Note

   Wait for the openshift-apiserver finish rolling updates before exposing the route.

   1. Expose the route:

      $ oc expose service hello-openshift
      route.route.openshift.io/hello-openshift exposed

      Example output

      $ oc get routes
      NAME              HOST/PORT                                   PATH   SERVICES          PORT       TERMINATION   WILDCARD
      hello-openshift   hello_openshift-<my_project>.test.example.com
      hello-openshift   8080-tcp                 None

Procedure

1. Configure the httpCompression field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit -n openshift-ingress-operator ingresscontrollers/default

   2. Under spec, set the httpCompression policy field to mimeTypes and specify a list of MIME types that should have compression applied:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpCompression:
          mimeTypes:
          - "text/html"
          - "text/css; charset=utf-8"
          - "application/json"
         ...

Procedure

1. Get the router pod name by running the following command:

   $ oc get pods -n openshift-ingress

   Example output

   NAME                              READY   STATUS    RESTARTS   AGE
   router-default-76bfffb66c-46qwp   1/1     Running   0          11h

2. Get the router’s username and password, which the router pod stores in the /var/lib/haproxy/conf/metrics-auth/statsUsername and /var/lib/haproxy/conf/metrics-auth/statsPassword files:

   1. Get the username by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsUsername

   2. Get the password by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsPassword

3. Get the router IP and metrics certificates by running the following command:

   $ oc describe pod <router_pod>

4. Get the raw statistics in Prometheus format by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

5. Access the metrics securely by running the following command:

   $ curl -u user:password https://<router_IP>:<stats_port>/metrics -k

6. Access the default stats port, 1936, by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

   Example 6.1. Example output

   ...
   # HELP haproxy_backend_connections_total Total number of connections.
   # TYPE haproxy_backend_connections_total gauge
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route"} 0
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route-alt"} 0
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route01"} 0
   ...
   # HELP haproxy_exporter_server_threshold Number of servers tracked and the current threshold value.
   # TYPE haproxy_exporter_server_threshold gauge
   haproxy_exporter_server_threshold{type="current"} 11
   haproxy_exporter_server_threshold{type="limit"} 500
   ...
   # HELP haproxy_frontend_bytes_in_total Current total of incoming bytes.
   # TYPE haproxy_frontend_bytes_in_total gauge
   haproxy_frontend_bytes_in_total{frontend="fe_no_sni"} 0
   haproxy_frontend_bytes_in_total{frontend="fe_sni"} 0
   haproxy_frontend_bytes_in_total{frontend="public"} 119070
   ...
   # HELP haproxy_server_bytes_in_total Current total of incoming bytes.
   # TYPE haproxy_server_bytes_in_total gauge
   haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_no_sni",service=""} 0
   haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_sni",service=""} 0
   haproxy_server_bytes_in_total{namespace="default",pod="docker-registry-5-nk5fz",route="docker-registry",server="10.130.0.89:5000",service="docker-registry"} 0
   haproxy_server_bytes_in_total{namespace="default",pod="hello-rc-vkjqx",route="hello-route",server="10.130.0.90:8080",service="hello-svc-1"} 0
   ...

7. Launch the stats window by entering the following URL in a browser:

   http://<user>:<password>@<router_IP>:<stats_port>

8. Optional: Get the stats in CSV format by entering the following URL in a browser:

   http://<user>:<password>@<router_ip>:1936/metrics;csv

Procedure

1. Create a config map named my-custom-error-code-pages in the openshift-config namespace:

   $ oc -n openshift-config create configmap my-custom-error-code-pages \
   --from-file=error-page-503.http \
   --from-file=error-page-404.http

   Important

   If you do not specify the correct format for the custom error code response page, a router pod outage occurs. To resolve this outage, you must delete or correct the config map and delete the affected router pods so they can be recreated with the correct information.

2. Patch the Ingress Controller to reference the my-custom-error-code-pages config map by name:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"httpErrorCodePages":{"name":"my-custom-error-code-pages"}}}' --type=merge

   The Ingress Operator copies the my-custom-error-code-pages config map from the openshift-config namespace to the openshift-ingress namespace. The Operator names the config map according to the pattern, <your_ingresscontroller_name>-errorpages, in the openshift-ingress namespace.

3. Display the copy:

   $ oc get cm default-errorpages -n openshift-ingress

   Example output

   NAME                       DATA   AGE
   default-errorpages         2      25s  1

   1

   The example config map name is default-errorpages because the default Ingress Controller custom resource (CR) was patched.

4. Confirm that the config map containing the custom error response page mounts on the router volume where the config map key is the filename that has the custom HTTP error code response:

   • For 503 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-503.http

   • For 404 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-404.http

1. Create a test project and application:

    $ oc new-project test-ingress

   $ oc new-app django-psql-example

2. For 503 custom http error code response:

   1. Stop all the pods for the application.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

3. For 404 custom http error code response:

   1. Visit a non-existent route or an incorrect route.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

4. Check if the errorfile attribute is properly in the haproxy.config file:

   $ oc -n openshift-ingress rsh <router> cat /var/lib/haproxy/conf/haproxy.config | grep errorfile

Procedure

1. To create the openshift-ingress-node-firewall namespace, enter the following command:

   $ cat << EOF| oc create -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     labels:
       pod-security.kubernetes.io/enforce: privileged
       pod-security.kubernetes.io/enforce-version: v1.24
     name: openshift-ingress-node-firewall
   EOF

2. To create an OperatorGroup CR, enter the following command:

   $ cat << EOF| oc create -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: ingress-node-firewall-operators
     namespace: openshift-ingress-node-firewall
   EOF

3. Subscribe to the Ingress Node Firewall Operator.

   1. To create a Subscription CR for the Ingress Node Firewall Operator, enter the following command:

      $ cat << EOF| oc create -f -
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: ingress-node-firewall-sub
        namespace: openshift-ingress-node-firewall
      spec:
        name: ingress-node-firewall
        channel: stable
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      EOF

4. To verify that the Operator is installed, enter the following command:

   $ oc get ip -n openshift-ingress-node-firewall

   Example output

   NAME            CSV                                         APPROVAL    APPROVED
   install-5cvnz   ingress-node-firewall.4.16.0-202211122336   Automatic   true

5. To verify the version of the Operator, enter the following command:

   $ oc get csv -n openshift-ingress-node-firewall

   Example output

   NAME                                        DISPLAY                          VERSION               REPLACES                                    PHASE
   ingress-node-firewall.4.16.0-202211122336   Ingress Node Firewall Operator   4.16.0-202211122336   ingress-node-firewall.4.16.0-202211102047   Succeeded

Procedure

1. Install the Ingress Node Firewall Operator:

   1. In the OpenShift Container Platform web console, click Operators OperatorHub.
   2. Select Ingress Node Firewall Operator from the list of available Operators, and then click Install.
   3. On the Install Operator page, under Installed Namespace, select Operator recommended Namespace.
   4. Click Install.

2. Verify that the Ingress Node Firewall Operator is installed successfully:

   1. Navigate to the Operators Installed Operators page.

   2. Ensure that Ingress Node Firewall Operator is listed in the openshift-ingress-node-firewall project with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the Operator does not have a Status of InstallSucceeded, troubleshoot using the following steps:

      • Inspect the Operator Subscriptions and Install Plans tabs for any failures or errors under Status.
      • Navigate to the Workloads Pods page and check the logs for pods in the openshift-ingress-node-firewall project.

      • Check the namespace of the YAML file. If the annotation is missing, you can add the annotation workload.openshift.io/allowed=management to the Operator namespace with the following command:

        $ oc annotate ns/openshift-ingress-node-firewall workload.openshift.io/allowed=management

        Note

        For single-node OpenShift clusters, the openshift-ingress-node-firewall namespace requires the workload.openshift.io/allowed=management annotation.