Networking

Procedure

1. Download the CredentialsRequest custom resource (CR) and store it in a directory by running the following command:

   $ curl --create-dirs -o <credentials_requests_dir>/operator.yaml https://raw.githubusercontent.com/openshift/aws-load-balancer-operator/main/hack/operator-credentials-request.yaml

2. Use the ccoctl utility to create an AWS IAM role by running the following command:

   $ ccoctl aws create-iam-roles \
       --name <name> \
       --region=<aws_region> \
       --credentials-requests-dir=<credentials_requests_dir> \
       --identity-provider-arn <oidc_arn>

   Example output

   2023/09/12 11:38:57 Role arn:aws:iam::777777777777:role/<name>-aws-load-balancer-operator-aws-load-balancer-operator created 1
   2023/09/12 11:38:57 Saved credentials configuration to: /home/user/<credentials_requests_dir>/manifests/aws-load-balancer-operator-aws-load-balancer-operator-credentials.yaml
   2023/09/12 11:38:58 Updated Role policy for Role <name>-aws-load-balancer-operator-aws-load-balancer-operator created

   1

   Note the Amazon Resource Name (ARN) of an AWS IAM role that was created for the AWS Load Balancer Operator, such as arn:aws:iam::777777777777:role/<name>-aws-load-balancer-operator-aws-load-balancer-operator.

   Note

   The length of an AWS IAM role name must be less than or equal to 12 characters.

Procedure

1. Download the CredentialsRequest custom resource (CR) and store it in a directory by running the following command:

   $ curl --create-dirs -o <credentials_requests_dir>/controller.yaml https://raw.githubusercontent.com/openshift/aws-load-balancer-operator/main/hack/controller/controller-credentials-request.yaml

2. Use the ccoctl utility to create an AWS IAM role by running the following command:

   $ ccoctl aws create-iam-roles \
       --name <name> \
       --region=<aws_region> \
       --credentials-requests-dir=<credentials_requests_dir> \
       --identity-provider-arn <oidc_arn>

   Example output

   2023/09/12 11:38:57 Role arn:aws:iam::777777777777:role/<name>-aws-load-balancer-operator-aws-load-balancer-controller created 1
   2023/09/12 11:38:57 Saved credentials configuration to: /home/user/<credentials_requests_dir>/manifests/aws-load-balancer-operator-aws-load-balancer-controller-credentials.yaml
   2023/09/12 11:38:58 Updated Role policy for Role <name>-aws-load-balancer-operator-aws-load-balancer-controller created

   1

   Note the Amazon Resource Name (ARN) of an AWS IAM role that was created for the AWS Load Balancer Controller, such as arn:aws:iam::777777777777:role/<name>-aws-load-balancer-operator-aws-load-balancer-controller.

   Note

   The length of an AWS IAM role name must be less than or equal to 12 characters.

Procedure

1. Identify the cluster infrastructure ID and the cluster OpenID Connect (OIDC) DNS by running the following commands:

   1. Identify the ROSA cluster ID:

      $ rosa describe cluster --cluster=<cluster_name> | grep -i 'Infra ID'

      or

      $ oc get infrastructure cluster -o json | jq -r '.status.infrastructureName'

   2. Identify the ROSA cluster OIDC DNS by using the following rosa CLI command:

      $ rosa describe cluster --cluster=<cluster_name> | grep -i OIDC 1

      1

      An OIDC DNS example is oidc.op1.openshiftapps.com/28q7fsn54m2jjts3kd556aij4mu9omah.

      or

      $ oc get authentication.config cluster -o=jsonpath="{.spec.serviceAccountIssuer}"

   3. Locate the OIDC Amazon Resource Name (ARN) information on the AWS Web Console by navigating to IAM Access management Identity providers. An OIDC ARN example is arn:aws:iam::777777777777:oidc-provider/<oidc_dns_url>.
   4. Save the output from the commands. You will use this information in future steps within this procedure.

2. Create the AWS IAM policy required for the AWS Load Balancer Operator by using the AWS CLI.

   1. Log in to the ROSA cluster as a user with the dedicated-admin role and create a new project using the following command:

      $ oc new-project aws-load-balancer-operator

   2. Assign the following trust policy to the newly-created AWS IAM role:

      $ IDP='{Cluster_OIDC_Endpoint}'

      $ IDP_ARN="arn:aws:iam::{AWS_AccountNo}:oidc-provider/${IDP}" 1

      1

      Replace {AWS_AccountNo} with your AWS account number and {Cluster_OIDC_Endpoint} with the OIDC DNS identified earlier in this procedure.

   3. Verify that the trsut policy was assigned to the AWS IAM role.

      Example output

      $ cat EOF albo-operator-trusted-policy.json
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "Federated": "${IDP_ARN}"
                  },
                  "Action": "sts:AssumeRoleWithWebIdentity",
                  "Condition": {
                      "StringEquals": {
                          "${IDP}:sub": "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-operator-controller-manager"
                      }
                  }
              }
          ]
      }
      EOF

      Important

      Do not include the https portion of the OIDC DNS URL when replacing {Cluster_OIDC_Endpoint} with the OIDC DNS you identified earlier. Only the alphanumeric information that follows the / within the URL is needed.

   4. Create and verify the role by using the generated trust policy:

      $ aws iam create-role --role-name albo-operator --assume-role-policy-document file://albo-operator-trusted-policy.json

      $ OPERATOR_ROLE_ARN=$(aws iam get-role --role-name albo-operator --output json | jq -r '.Role.Arn')

      $ echo $OPERATOR_ROLE_ARN

      Example output

      ROLE arn:aws:iam::<aws_account_number>:role/albo-operator	2023-08-02T12:13:22Z
      ASSUMEROLEPOLICYDOCUMENT	2012-10-17
      STATEMENT	sts:AssumeRoleWithWebIdentity	Allow
      STRINGEQUALS system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-manager
      PRINCIPAL arn:aws:iam:<aws_account_number>:oidc-provider/<oidc_provider_id>

      Note

      Where arn of the AWS IAM role was created for the AWS Load Balancer Operator, such as arn:aws:iam::777777777777:role/albo-operator.

   5. Attach the operator’s permission policy to the role:

      $ curl -o albo-operator-permission-policy.json https://raw.githubusercontent.com/openshift/aws-load-balancer-operator/release-1.1/hack/operator-permission-policy.json
      $ aws iam put-role-policy --role-name albo-operator --policy-name perms-policy-albo-operator --policy-document file://albo-operator-permission-policy.json

3. Create the AWS IAM policy required for the AWS Load Balancer Controller by using the AWS CLI:

   1. Generate a trust policy file for your identity provider. The following example uses OpenID Connect:

      $ IDP='{Cluster_OIDC_Endpoint}'
      $ IDP_ARN="arn:aws:iam::{AWS_AccountNo}:oidc-provider/${IDP}"
      $ cat <EOF> albo-controller-trusted-policy.json
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "Federated": "${IDP_ARN}"
                  },
                  "Action": "sts:AssumeRoleWithWebIdentity",
                  "Condition": {
                      "StringEquals": {
                          "${IDP}:sub": "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster"
                      }
                  }
              }
          ]
      }
      EOF

   2. Create and verify the role by using the generated trust policy:

      $ aws iam create-role --role-name albo-controller --assume-role-policy-document file://albo-controller-trusted-policy.json
      $ CONTROLLER_ROLE_ARN=$(aws iam get-role --role-name albo-controller --output json | jq -r '.Role.Arn')
      $ echo $CONTROLLER_ROLE_ARN

      Example output

      ROLE	arn:aws:iam::<aws_account_number>:role/albo-controller	2023-08-02T12:13:22Z
      ASSUMEROLEPOLICYDOCUMENT	2012-10-17
      STATEMENT	    sts:AssumeRoleWithWebIdentity	Allow
      STRINGEQUALS	system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster
      PRINCIPAL	    arn:aws:iam:<aws_account_number>:oidc-provider/<oidc_provider_id>

      Note

      Where arn of the AWS IAM role that was created for the AWS Load Balancer Controller, such as arn:aws:iam::777777777777:role/albo-controller.

   3. Attach the controller’s permission policy to the role:

      $ curl -o albo-controller-permission-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.7/docs/install/iam_policy.json
      $ aws iam put-role-policy --role-name albo-controller --policy-name perms-policy-albo-controller --policy-document file://albo-controller-permission-policy.json

4. For a ROSA with HCP cluster, add the tags necessary for subnet discovery:

   1. Add the following {Key: Value} tag to the VPC hosting the ROSA cluster and to all its subnets. Replace {Cluster Infra ID} with the Infra ID specified previously:

      kubernetes.io/cluster/${Cluster Infra ID}:owned

   2. Add the following ELBv2 {Key: Value} tags to the private subnets and, optionally, to the public subnets:

      • Private subnets: kubernetes.io/role/internal-elb:1

      • Public subnets: kubernetes.io/role/elb:1

        Note

        Internet-facing and internal load balancers will be created within the AWS Availability Zone to which these subnets belong.

        Important

        ELBv2 resources (such as ALBs and NLBs) created by AWS Load Balancer Operator do not inherit custom tags set for ROSA clusters. You must set tags separately for these resources.

5. Create the AWS Load Balancer Operator by completing the following steps:

   1. Create an OperatorGroup object by running the following command:

      $ cat EOF | oc apply -f -
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: aws-load-balancer-operator
        namespace: aws-load-balancer-operator
      spec:
        targetNamespaces: []
      EOF

   2. Create a Subscription object by running the following command:

      $ cat EOF | oc apply -f -
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: aws-load-balancer-operator
        namespace: aws-load-balancer-operator
      spec:
        channel: stable-v1
        name: aws-load-balancer-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        config:
          env:
          - name: ROLEARN
            value: "<operator_role_arn>" 1
      EOF

      1

      Specifies the ARN role for the AWS Load Balancer Operator. The CredentialsRequest object uses this ARN role to provision the AWS credentials. An example of <operator_role_arn> is arn:aws:iam::<aws_account_number>:role/albo-operator.

6. Create the AWS Load Balancer Controller:

   apiVersion: networking.olm.openshift.io/v1
   kind: AWSLoadBalancerController
   metadata:
     name: cluster
   spec:
     subnetTagging: Manual
     credentialsRequestConfig:
       stsIAMRoleARN: <controller_role_arn>

   Important

   Because AWS Load Balancer Controllers do not support creating AWS Load Balancers (ALBs) associated with both AWS Availability Zones and AWS Local Zones, ROSA clusters can have ALBs associated exclusively with either AWS Local Zones or AWS Availability Zones but not both simultaneously.

Verification

1. Confirm a successful installation by running the following commands:

   1. Gather information about pods within the project:

      $ oc get pods -n aws-load-balancer-operator

   2. View the logs within the project:

      $ oc logs -n aws-load-balancer-operator deployment/aws-load-balancer-operator-controller-manager -c manager

Procedure

1. Clean up the sample application by deleting the Load Balancers created and managed by the ALBO. For more information about deleting Load Balancers, see Delete an Application Load Balancer.
2. Clean up the AWS VPC tags by removing the VPC tags that were added to the subnets for discovering subnets and for creating Application Load Balancers (ALBs). For more information, see Tag basics.
3. Clean up AWS Load Balancer Operator components by deleting both the AWS Load Balancer Operator and the Application Load Balancer Controller. For more information, see Deleting Operators from a cluster.

1. Create a Secret resource containing the custom certificate in the openshift-ingress namespace using the tls.crt and tls.key files.

   $ oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key

2. Update the IngressController CR to reference the new certificate secret:

   $ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
     --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'

3. Verify the update was effective:

   $ echo Q |\
     openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null |\
     openssl x509 -noout -subject -issuer -enddate

   where:

   <domain>

   Specifies the base domain name for your cluster.

   Example output

   subject=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = *.apps.example.com
   issuer=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = example.com
   notAfter=May 10 08:32:45 2022 GM

   Tip

   You can alternatively apply the following YAML to set a custom default certificate:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     defaultCertificate:
       name: custom-certs-default

   The certificate secret name should match the value used to update the CR.

Procedure

1. Create a service account to authenticate with Thanos by running the following command:

   $ oc create -n openshift-ingress-operator serviceaccount thanos && oc describe -n openshift-ingress-operator serviceaccount thanos

   Example output

   Name:                thanos
   Namespace:           openshift-ingress-operator
   Labels:              <none>
   Annotations:         <none>
   Image pull secrets:  thanos-dockercfg-kfvf2
   Mountable secrets:   thanos-dockercfg-kfvf2
   Tokens:              <none>
   Events:              <none>

2. Manually create the service account secret token with the following command:

   $ oc apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: thanos-token
     namespace: openshift-ingress-operator
     annotations:
       kubernetes.io/service-account.name: thanos
   type: kubernetes.io/service-account-token
   EOF

3. Define a TriggerAuthentication object within the openshift-ingress-operator namespace by using the service account’s token.

   1. Create the TriggerAuthentication object and pass the value of the secret variable to the TOKEN parameter:

      $ oc apply -f - <<EOF
      apiVersion: keda.sh/v1alpha1
      kind: TriggerAuthentication
      metadata:
        name: keda-trigger-auth-prometheus
        namespace: openshift-ingress-operator
      spec:
        secretTargetRef:
        - parameter: bearerToken
          name: thanos-token
          key: token
        - parameter: ca
          name: thanos-token
          key: ca.crt
      EOF

4. Create and apply a role for reading metrics from Thanos:

   1. Create a new role, thanos-metrics-reader.yaml, that reads metrics from pods and nodes:

      thanos-metrics-reader.yaml

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: thanos-metrics-reader
        namespace: openshift-ingress-operator
      rules:
      - apiGroups:
        - ""
        resources:
        - pods
        - nodes
        verbs:
        - get
      - apiGroups:
        - metrics.k8s.io
        resources:
        - pods
        - nodes
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - ""
        resources:
        - namespaces
        verbs:
        - get

   2. Apply the new role by running the following command:

      $ oc apply -f thanos-metrics-reader.yaml

5. Add the new role to the service account by entering the following commands:

   $ oc adm policy -n openshift-ingress-operator add-role-to-user thanos-metrics-reader -z thanos --role-namespace=openshift-ingress-operator

   $ oc adm policy -n openshift-ingress-operator add-cluster-role-to-user cluster-monitoring-view -z thanos

   Note

   The argument add-cluster-role-to-user is only required if you use cross-namespace queries. The following step uses a query from the kube-metrics namespace which requires this argument.

6. Create a new ScaledObject YAML file, ingress-autoscaler.yaml, that targets the default Ingress Controller deployment:

   Example ScaledObject definition

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: ingress-scaler
     namespace: openshift-ingress-operator
   spec:
     scaleTargetRef: 1
       apiVersion: operator.openshift.io/v1
       kind: IngressController
       name: default
       envSourceContainerName: ingress-operator
     minReplicaCount: 1
     maxReplicaCount: 20 2
     cooldownPeriod: 1
     pollingInterval: 1
     triggers:
     - type: prometheus
       metricType: AverageValue
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 3
         namespace: openshift-ingress-operator 4
         metricName: 'kube-node-role'
         threshold: '1'
         query: 'sum(kube_node_role{role="worker",service="kube-state-metrics"})' 5
         authModes: "bearer"
       authenticationRef:
         name: keda-trigger-auth-prometheus

   1

   The custom resource that you are targeting. In this case, the Ingress Controller.

   2

   Optional: The maximum number of replicas. If you omit this field, the default maximum is set to 100 replicas.

   3

   The Thanos service endpoint in the openshift-monitoring namespace.

   4

   The Ingress Operator namespace.

   5

   This expression evaluates to however many worker nodes are present in the deployed cluster.

   Important

   If you are using cross-namespace queries, you must target port 9091 and not port 9092 in the serverAddress field. You also must have elevated privileges to read metrics from this port.

7. Apply the custom resource definition by running the following command:

   $ oc apply -f ingress-autoscaler.yaml

Procedure

1. View the current number of available replicas for the default IngressController:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   2

2. Scale the default IngressController to the desired number of replicas using the oc patch command. The following example scales the default IngressController to 3 replicas:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"replicas": 3}}' --type=merge

   Example output

   ingresscontroller.operator.openshift.io/default patched

3. Verify that the default IngressController scaled to the number of replicas that you specified:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   3

   Tip

   You can alternatively apply the following YAML to scale an Ingress Controller to three replicas:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: 3               1

   1

   If you need a different amount of replicas, change the replicas value.

Procedure

1. Create an IngressController custom resource (CR) in a file named <name>-ingress-controller.yaml, such as in the following example:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: <name> 1
   spec:
     domain: <domain> 2
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal 3

   1

   Replace <name> with a name for the IngressController object.

   2

   Specify the domain for the application published by the controller.

   3

   Specify a value of Internal to use an internal load balancer.

2. Create the Ingress Controller defined in the previous step by running the following command:

   $ oc create -f <name>-ingress-controller.yaml 1

   1

   Replace <name> with the name of the IngressController object.

3. Optional: Confirm that the Ingress Controller was created by running the following command:

   $ oc --all-namespaces=true get ingresscontrollers

Procedure

1. Configure the default Ingress Controller for your cluster to be internal by deleting and recreating it.

   $ oc replace --force --wait --filename - <<EOF
   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: default
   spec:
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal
   EOF

Procedure

1. Configure the wildcard policy.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

   2. Under spec, set the wildcardPolicy field to WildcardsDisallowed or WildcardsAllowed:

      spec:
        routeAdmission:
          wildcardPolicy: WildcardsDisallowed # or WildcardsAllowed

Procedure

1. Edit the Ingress Controller resource:

   $ oc -n openshift-ingress-operator edit ingresscontroller/default

2. Replace the X-SSL-Client-Der HTTP request header with the X-Forwarded-Client-Cert HTTP request header:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     httpHeaders:
       actions: 1
         request: 2
         - name: X-Forwarded-Client-Cert 3
           action:
             type: Set 4
             set:
              value: "%{+Q}[ssl_c_der,base64]" 5
         - name: X-SSL-Client-Der
           action:
             type: Delete

   1

   The list of actions you want to perform on the HTTP headers.

   2

   The type of header you want to change. In this case, a request header.

   3

   The name of the header you want to change. For a list of available headers you can set or delete, see HTTP header configuration.

   4

   The type of action being taken on the header. This field can have the value Set or Delete.

   5

   When setting HTTP headers, you must provide a value. The value can be a string from a list of available directives for that header, for example DENY, or it can be a dynamic value that will be interpreted using HAProxy’s dynamic value syntax. In this case, a dynamic value is added.

   Note

   For setting dynamic header values for HTTP responses, allowed sample fetchers are res.hdr and ssl_c_der. For setting dynamic header values for HTTP requests, allowed sample fetchers are req.hdr and ssl_c_der. Both request and response dynamic values can use the lower and base64 converters.

3. Save the file to apply the changes.

Procedure

1. Configure the HTTPHeaders field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

   2. Under spec, set the HTTPHeaders policy field to Append, Replace, IfNone, or Never:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpHeaders:
          forwardedHeaderPolicy: Append

Procedure

1. Edit the Ingress Controller resource by entering the following command in your CLI:

   $ oc -n openshift-ingress-operator edit ingresscontroller/default

2. Set the PROXY configuration:

   • If your Ingress Controller uses the HostNetwork endpoint publishing strategy type, set the spec.endpointPublishingStrategy.hostNetwork.protocol subfield to PROXY:

     Sample hostNetwork configuration to PROXY

     # ...
       spec:
         endpointPublishingStrategy:
           hostNetwork:
             protocol: PROXY
           type: HostNetwork
     # ...

   • If your Ingress Controller uses the NodePortService endpoint publishing strategy type, set the spec.endpointPublishingStrategy.nodePort.protocol subfield to PROXY:

     Sample nodePort configuration to PROXY

     # ...
       spec:
         endpointPublishingStrategy:
           nodePort:
             protocol: PROXY
           type: NodePortService
     # ...

   • If your Ingress Controller uses the Private endpoint publishing strategy type, set the spec.endpointPublishingStrategy.private.protocol subfield to PROXY:

     Sample private configuration to PROXY

     # ...
       spec:
         endpointPublishingStrategy:
           private:
             protocol: PROXY
         type: Private
     # ...

Procedure

1. Configure the appsDomain field by specifying an alternative default domain for user-created routes.

   1. Edit the ingress cluster resource:

      $ oc edit ingresses.config/cluster -o yaml

   2. Edit the YAML file:

      Sample appsDomain configuration to test.example.com

      apiVersion: config.openshift.io/v1
      kind: Ingress
      metadata:
        name: cluster
      spec:
        domain: apps.example.com            1
        appsDomain: <test.example.com>      2

      1

      Specifies the default domain. You cannot modify the default domain after installation.

      2

      Optional: Domain for Red Hat OpenShift Service on AWS infrastructure to use for application routes. Instead of the default prefix, apps, you can use an alternative prefix like test.

2. Verify that an existing route contains the domain name specified in the appsDomain field by exposing the route and verifying the route domain change:

   Note

   Wait for the openshift-apiserver finish rolling updates before exposing the route.

   1. Expose the route:

      $ oc expose service hello-openshift
      route.route.openshift.io/hello-openshift exposed

      Example output

      $ oc get routes
      NAME              HOST/PORT                                   PATH   SERVICES          PORT       TERMINATION   WILDCARD
      hello-openshift   hello_openshift-<my_project>.test.example.com
      hello-openshift   8080-tcp                 None

Procedure

1. Configure the httpCompression field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit -n openshift-ingress-operator ingresscontrollers/default

   2. Under spec, set the httpCompression policy field to mimeTypes and specify a list of MIME types that should have compression applied:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpCompression:
          mimeTypes:
          - "text/html"
          - "text/css; charset=utf-8"
          - "application/json"
         ...

Procedure

1. Get the router pod name by running the following command:

   $ oc get pods -n openshift-ingress

   Example output

   NAME                              READY   STATUS    RESTARTS   AGE
   router-default-76bfffb66c-46qwp   1/1     Running   0          11h

2. Get the router’s username and password, which the router pod stores in the /var/lib/haproxy/conf/metrics-auth/statsUsername and /var/lib/haproxy/conf/metrics-auth/statsPassword files:

   1. Get the username by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsUsername

   2. Get the password by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsPassword

3. Get the router IP and metrics certificates by running the following command:

   $ oc describe pod <router_pod>

4. Get the raw statistics in Prometheus format by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

5. Access the metrics securely by running the following command:

   $ curl -u user:password https://<router_IP>:<stats_port>/metrics -k

6. Access the default stats port, 1936, by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

   Example 4.1. Example output

   ...
   # HELP haproxy_backend_connections_total Total number of connections.
   # TYPE haproxy_backend_connections_total gauge
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route"} 0
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route-alt"} 0
   haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route01"} 0
   ...
   # HELP haproxy_exporter_server_threshold Number of servers tracked and the current threshold value.
   # TYPE haproxy_exporter_server_threshold gauge
   haproxy_exporter_server_threshold{type="current"} 11
   haproxy_exporter_server_threshold{type="limit"} 500
   ...
   # HELP haproxy_frontend_bytes_in_total Current total of incoming bytes.
   # TYPE haproxy_frontend_bytes_in_total gauge
   haproxy_frontend_bytes_in_total{frontend="fe_no_sni"} 0
   haproxy_frontend_bytes_in_total{frontend="fe_sni"} 0
   haproxy_frontend_bytes_in_total{frontend="public"} 119070
   ...
   # HELP haproxy_server_bytes_in_total Current total of incoming bytes.
   # TYPE haproxy_server_bytes_in_total gauge
   haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_no_sni",service=""} 0
   haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_sni",service=""} 0
   haproxy_server_bytes_in_total{namespace="default",pod="docker-registry-5-nk5fz",route="docker-registry",server="10.130.0.89:5000",service="docker-registry"} 0
   haproxy_server_bytes_in_total{namespace="default",pod="hello-rc-vkjqx",route="hello-route",server="10.130.0.90:8080",service="hello-svc-1"} 0
   ...

7. Launch the stats window by entering the following URL in a browser:

   http://<user>:<password>@<router_IP>:<stats_port>

8. Optional: Get the stats in CSV format by entering the following URL in a browser:

   http://<user>:<password>@<router_ip>:1936/metrics;csv

Procedure

1. Create a config map named my-custom-error-code-pages in the openshift-config namespace:

   $ oc -n openshift-config create configmap my-custom-error-code-pages \
   --from-file=error-page-503.http \
   --from-file=error-page-404.http

   Important

   If you do not specify the correct format for the custom error code response page, a router pod outage occurs. To resolve this outage, you must delete or correct the config map and delete the affected router pods so they can be recreated with the correct information.

2. Patch the Ingress Controller to reference the my-custom-error-code-pages config map by name:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"httpErrorCodePages":{"name":"my-custom-error-code-pages"}}}' --type=merge

   The Ingress Operator copies the my-custom-error-code-pages config map from the openshift-config namespace to the openshift-ingress namespace. The Operator names the config map according to the pattern, <your_ingresscontroller_name>-errorpages, in the openshift-ingress namespace.

3. Display the copy:

   $ oc get cm default-errorpages -n openshift-ingress

   Example output

   NAME                       DATA   AGE
   default-errorpages         2      25s  1

   1

   The example config map name is default-errorpages because the default Ingress Controller custom resource (CR) was patched.

4. Confirm that the config map containing the custom error response page mounts on the router volume where the config map key is the filename that has the custom HTTP error code response:

   • For 503 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-503.http

   • For 404 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-404.http

1. Create a test project and application:

    $ oc new-project test-ingress

   $ oc new-app django-psql-example

2. For 503 custom http error code response:

   1. Stop all the pods for the application.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

3. For 404 custom http error code response:

   1. Visit a non-existent route or an incorrect route.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

4. Check if the errorfile attribute is properly in the haproxy.config file:

   $ oc -n openshift-ingress rsh <router> cat /var/lib/haproxy/conf/haproxy.config | grep errorfile

Procedure

1. Navigate to OpenShift Cluster Manager and select your cluster.
2. Under the Virtual Private Cloud (VPC) section on the Networking page, click Edit cluster-wide proxy.

3. On the Edit cluster-wide proxy page, provide your proxy configuration details:

   1. Enter a value in at least one of the following fields:

      • Specify a valid HTTP proxy URL.
      • Specify a valid HTTPS proxy URL.
      • In the Additional trust bundle field, provide a PEM encoded X.509 certificate bundle. If you are replacing an existing trust bundle file, select Replace file to view the field. The bundle is added to the trusted certificate store for the cluster nodes. An additional trust bundle file is required if you use a TLS-inspecting proxy unless the identity certificate for the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle. This requirement applies regardless of whether the proxy is transparent or requires explicit configuration using the http-proxy and https-proxy arguments.
   2. Click Confirm.

Verification

1. List the status of the machine config pools and verify that they are updated:

   $ oc get machineconfigpools

   Example output

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-d9a03f612a432095dcde6dcf44597d90   True      False      False      3              3                   3                     0                      31h
   worker   rendered-worker-f6827a4efe21e155c25c21b43c46f65e   True      False      False      6              6                   6                     0                      31h

2. Display the proxy configuration for your cluster and verify that the details are as expected:

   $ oc get proxy cluster -o yaml

   Example output

   apiVersion: config.openshift.io/v1
   kind: Proxy
   spec:
     httpProxy: http://proxy.host.domain:<port>
     httpsProxy: https://proxy.host.domain:<port>
     <...more...>
   status:
     httpProxy: http://proxy.host.domain:<port>
     httpsProxy: https://proxy.host.domain:<port>
     <...more...>

1. Create the IngressNodeFirewallConfig inside the openshift-ingress-node-firewall namespace named ingressnodefirewallconfig.

2. Run the following command to deploy Ingress Node Firewall Operator rules:

   $ oc apply -f rule.yaml

Procedure

1. Apply the following labels to the ingress-node-firewall-system namespace:

   $ oc label namespace openshift-ingress-node-firewall \
       pod-security.kubernetes.io/enforce=privileged \
       pod-security.kubernetes.io/warn=privileged --overwrite

2. Edit the IngressNodeFirewallConfig object named ingressnodefirewallconfig and set the ebpfProgramManagerMode field:

   Ingress Node Firewall Operator configuration object

   apiVersion: ingressnodefirewall.openshift.io/v1alpha1
   kind: IngressNodeFirewallConfig
   metadata:
     name: ingressnodefirewallconfig
     namespace: openshift-ingress-node-firewall
   spec:
     nodeSelector:
       node-role.kubernetes.io/worker: ""
     ebpfProgramManagerMode: <ebpf_mode>

   where:

   <ebpf_mode>: Specifies whether or not the Ingress Node Firewall Operator uses the eBPF Manager Operator to manage eBPF programs. Must be either true or false. If unset, eBPF Manager is not used.

Procedure

1. Run the following command to view all current rules :

   $ oc get ingressnodefirewall

2. Choose one of the returned <resource> names and run the following command to view the rules or configs:

   $ oc get <resource> <name> -o yaml

1. Change your current project to the project that you enabled multicast for. Replace <project> with the project name.

   $ oc project <project>

2. Create a pod to act as a multicast receiver:

   $ cat <<EOF| oc create -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: mlistener
     labels:
       app: multicast-verify
   spec:
     containers:
       - name: mlistener
         image: registry.access.redhat.com/ubi9
         command: ["/bin/sh", "-c"]
         args:
           ["dnf -y install socat hostname && sleep inf"]
         ports:
           - containerPort: 30102
             name: mlistener
             protocol: UDP
   EOF

3. Create a pod to act as a multicast sender:

   $ cat <<EOF| oc create -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: msender
     labels:
       app: multicast-verify
   spec:
     containers:
       - name: msender
         image: registry.access.redhat.com/ubi9
         command: ["/bin/sh", "-c"]
         args:
           ["dnf -y install socat && sleep inf"]
   EOF

4. In a new terminal window or tab, start the multicast listener.

   1. Get the IP address for the Pod:

      $ POD_IP=$(oc get pods mlistener -o jsonpath='{.status.podIP}')

   2. Start the multicast listener by entering the following command:

      $ oc exec mlistener -i -t -- \
          socat UDP4-RECVFROM:30102,ip-add-membership=224.1.0.1:$POD_IP,fork EXEC:hostname

5. Start the multicast transmitter.

   1. Get the pod network IP address range:

      $ CIDR=$(oc get Network.config.openshift.io cluster \
          -o jsonpath='{.status.clusterNetwork[0].cidr}')

   2. To send a multicast message, enter the following command:

      $ oc exec msender -i -t -- \
          /bin/bash -c "echo | socat STDIO UDP4-DATAGRAM:224.1.0.1:30102,range=$CIDR,ip-multicast-ttl=64"

      If multicast is working, the previous command returns the following output:

      mlistener

Procedure

1. Create a project called hello-openshift by running the following command:

   $ oc new-project hello-openshift

2. Create a pod in the project by running the following command:

   $ oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json

3. Create a service called hello-openshift by running the following command:

   $ oc expose pod/hello-openshift

4. Create an unsecured route to the hello-openshift application by running the following command:

   $ oc expose svc hello-openshift

Procedure

1. Using the oc annotate command, add the timeout to the route:

   $ oc annotate route <route_name> \
       --overwrite haproxy.router.openshift.io/timeout=<timeout><time_unit> 1

   1

   Supported time units are microseconds (us), milliseconds (ms), seconds (s), minutes (m), hours (h), or days (d).

   The following example sets a timeout of two seconds on a route named myroute:

   $ oc annotate route myroute --overwrite haproxy.router.openshift.io/timeout=2s

Procedure

1. Create a route definition and save it in a file called app-example-route.yaml:

   YAML definition of the created route with HTTP header directives

   apiVersion: route.openshift.io/v1
   kind: Route
   # ...
   spec:
     host: app.example.com
     tls:
       termination: edge
     to:
       kind: Service
       name: app-example
     httpHeaders:
       actions: 1
         response: 2
         - name: Content-Location 3
           action:
             type: Set 4
             set:
               value: /lang/en-us 5

   1

   The list of actions you want to perform on the HTTP headers.

   2

   The type of header you want to change. In this case, a response header.

   3

   The name of the header you want to change. For a list of available headers you can set or delete, see HTTP header configuration.

   4

   The type of action being taken on the header. This field can have the value Set or Delete.

   5

   When setting HTTP headers, you must provide a value. The value can be a string from a list of available directives for that header, for example DENY, or it can be a dynamic value that will be interpreted using HAProxy’s dynamic value syntax. In this case, the value is set to the relative location of the content.

2. Create a route to your existing web application using the newly created route definition:

   $ oc -n app-example create -f app-example-route.yaml

1. If the number of IP addresses and CIDR ranges in an allowlist exceeds 61, they are written into a separate file that is then referenced from haproxy.config. This file is stored in the var/lib/haproxy/router/whitelists folder.

   Note

   To ensure that the addresses are written to the allowlist, check that the full list of CIDR ranges are listed in the Ingress Controller configuration file. The etcd object size limit restricts how large a route annotation can be. Because of this, it creates a threshold for the maximum number of IP addresses and CIDR ranges that you can include in an allowlist.

Procedure

1. Create a YAML file for the Ingress object. In this example, the file is called example-ingress.yaml:

   YAML definition of an Ingress object

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: frontend
     ...
   spec:
     rules:
       ...
     tls:
     - {} 1

   1

   Use this exact syntax to specify TLS without specifying a custom certificate.

2. Create the Ingress object by running the following command:

   $ oc create -f example-ingress.yaml

Procedure

1. Add the route.openshift.io/destination-ca-certificate-secret to the Ingress annotations:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: frontend
     annotations:
       route.openshift.io/termination: "reencrypt"
       route.openshift.io/destination-ca-certificate-secret: secret-ca-cert 1
   ...

   1

   The annotation references a kubernetes secret.

2. The secret referenced in this annotation will be inserted into the generated route.

   Example output

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: frontend
     annotations:
       route.openshift.io/termination: reencrypt
       route.openshift.io/destination-ca-certificate-secret: secret-ca-cert
   spec:
   ...
     tls:
       insecureEdgeTerminationPolicy: Redirect
       termination: reencrypt
       destinationCACertificate: |
         -----BEGIN CERTIFICATE-----
         [...]
         -----END CERTIFICATE-----
   ...