Configuring and deploying gateway policies

Procedure

1. Optional: Set the following environment variables:

   export KUADRANT_GATEWAY_NS=api-gateway
   export KUADRANT_GATEWAY_NAME=external
   export KUADRANT_DEVELOPER_NS=toystore
   export KUADRANT_AWS_ACCESS_KEY_ID=xxxx
   export KUADRANT_AWS_SECRET_ACCESS_KEY=xxxx
   export KUADRANT_AWS_DNS_PUBLIC_ZONE_ID=xxxx
   export KUADRANT_ZONE_ROOT_DOMAIN=example.com
   export KUADRANT_CLUSTER_ISSUER_NAME=self-signed

   Copy to Clipboard Toggle word wrap

   These environment variables are described as follows:

   • KUADRANT_GATEWAY_NS: Namespace for your example gateway in OpenShift Container Platform.
   • KUADRANT_GATEWAY_NAME: Name of your example gateway in OpenShift Container Platform.
   • KUADRANT_DEVELOPER_NS: Namespace for the example Toystore app in OpenShift Container Platform.
   • KUADRANT_AWS_ACCESS_KEY_ID: AWS key ID with access to manage your DNS zone.
   • KUADRANT_AWS_SECRET_ACCESS_KEY: AWS secret access key with permissions to manage your DNS zone.
   • KUADRANT_AWS_DNS_PUBLIC_ZONE_ID: AWS Route 53 zone ID for the Gateway. This is the ID of the hosted zone that is displayed in the AWS Route 53 console.
   • KUADRANT_ZONE_ROOT_DOMAIN: Root domain in AWS Route 53 associated with your DNS zone ID.

   • KUADRANT_CLUSTER_ISSUER_NAME: Name of the certificate authority or issuer TLS certificates.

     Note

     If you know your environment variable values, you can set up the required YAML files to suit your environment.

2. Create the namespace for the Toystore app as follows:

   $ kubectl create ns ${KUADRANT_DEVELOPER_NS}

   Copy to Clipboard Toggle word wrap

3. Deploy the Toystore app to the developer namespace:

   $ kubectl apply -f https://raw.githubusercontent.com/Kuadrant/Kuadrant-operator/main/examples/toystore/toystore.yaml -n ${KUADRANT_DEVELOPER_NS}

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the namespace that the Gateway will be deployed in as follows:

   kubectl create ns ${KUADRANT_GATEWAY_NS}

   Copy to Clipboard Toggle word wrap

2. Create the secret credentials in the same namespace as the Gateway as follows:

   $ kubectl -n ${KUADRANT_GATEWAY_NS} create secret generic aws-credentials \
     --type=kuadrant.io/aws \
     --from-literal=AWS_ACCESS_KEY_ID=$KUADRANT_AWS_ACCESS_KEY_ID \
     --from-literal=AWS_SECRET_ACCESS_KEY=$KUADRANT_AWS_SECRET_ACCESS_KEY

   Copy to Clipboard Toggle word wrap

3. Before adding a TLS certificate issuer, create the secret credentials in the cert-manager namespace as follows:

   $ kubectl -n cert-manager create secret generic aws-credentials \
     --type=kuadrant.io/aws \
     --from-literal=AWS_ACCESS_KEY_ID=$KUADRANT_AWS_ACCESS_KEY_ID \
     --from-literal=AWS_SECRET_ACCESS_KEY=$KUADRANT_AWS_SECRET_ACCESS_KEY

   Copy to Clipboard Toggle word wrap

Procedure

1. Enter the following command to define a TLS certificate issuer:

   $ kubectl apply -f - <<EOF
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: ${KUADRANT_CLUSTER_ISSUER_NAME}
   spec:
     selfSigned: {}
   EOF

   Copy to Clipboard Toggle word wrap

2. Wait for the ClusterIssuer to become ready as follows:

   $ kubectl wait clusterissuer/${KUADRANT_CLUSTER_ISSUER_NAME} --for=condition=ready=true

   Copy to Clipboard Toggle word wrap

Procedure

1. Enter the following command to create the Gateway:

   $ kubectl apply -f - <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: ${KUADRANT_GATEWAY_NAME}
     namespace: ${KUADRANT_GATEWAY_NS}
     labels:
       kuadrant.io/gateway: "true"
   spec:
       gatewayClassName: istio
       listeners:
       - allowedRoutes:
           namespaces:
             from: All
         hostname: "api.${KUADRANT_ZONE_ROOT_DOMAIN}"
         name: api
         port: 443
         protocol: HTTPS
         tls:
           certificateRefs:
           - group: ""
             kind: Secret
             name: api-${KUADRANT_GATEWAY_NAME}-tls
           mode: Terminate
   EOF

   Copy to Clipboard Toggle word wrap

2. Check the status of your Gateway as follows:

   kubectl get gateway ${KUADRANT_GATEWAY_NAME} -n ${KUADRANT_GATEWAY_NS} -o=jsonpath='{.status.conditions[?(@.type=="Accepted")].message}{"\n"}{.status.conditions[?(@.type=="Programmed")].message}'

   Copy to Clipboard Toggle word wrap

   Your Gateway should be Accepted and Programmed, which means that it is valid and assigned an external address.

3. Check the status of your HTTPS listener as follows:

   $ kubectl get gateway ${KUADRANT_GATEWAY_NAME} -n ${KUADRANT_GATEWAY_NS} -o=jsonpath='{.status.listeners[0].conditions[?(@.type=="Programmed")].message}'

   Copy to Clipboard Toggle word wrap

   You will see that the HTTPS listener is not yet programmed or ready to accept traffic due to bad TLS configuration. Connectivity Link can help with this by using a TLSPolicy, which is described in the next step.

Procedure

1. Set the TLSPolicy for your Gateway as follows:

   $ kubectl apply -f - <<EOF
   apiVersion: kuadrant.io/v1
   kind: TLSPolicy
   metadata:
     name: ${KUADRANT_GATEWAY_NAME}-tls
     namespace: ${KUADRANT_GATEWAY_NS}
   spec:
     targetRef:
       name: ${KUADRANT_GATEWAY_NAME}
       group: gateway.networking.k8s.io
       kind: Gateway
     issuerRef:
       group: cert-manager.io
       kind: ClusterIssuer
       name: ${KUADRANT_CLUSTER_ISSUER_NAME}
   EOF

   Copy to Clipboard Toggle word wrap

2. Check that your TLS policy has an Accepted and Enforced status as follows:

   $ kubectl get tlspolicy ${KUADRANT_GATEWAY_NAME}-tls -n ${KUADRANT_GATEWAY_NS} -o=jsonpath='{.status.conditions[?(@.type=="Accepted")].message}{"\n"}{.status.conditions[?(@.type=="Enforced")].message}'

   Copy to Clipboard Toggle word wrap

   This may take a few minutes depending on the TLS provider, for example, Let’s Encrypt.

Procedure

1. Ensure that your Connectivity Link system namespace is set correctly as follows:

   $ export KUADRANT_SYSTEM_NS=$(kubectl get kuadrant -A -o jsonpath="{.items[0].metadata.namespace}")

   Copy to Clipboard Toggle word wrap

2. Define API keys for bob and alice users as follows:

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: bob-key
     namespace: ${KUADRANT_SYSTEM_NS}
     labels:
       authorino.kuadrant.io/managed-by: authorino
       app: toystore
     annotations:
       secret.kuadrant.io/user-id: bob
   stringData:
     api_key: IAMBOB
   type: Opaque
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: alice-key
     namespace: ${KUADRANT_SYSTEM_NS}
     labels:
       authorino.kuadrant.io/managed-by: authorino
       app: toystore
     annotations:
       secret.kuadrant.io/user-id: alice
   stringData:
     api_key: IAMALICE
   type: Opaque
   EOF

   Copy to Clipboard Toggle word wrap

3. Create a new AuthPolicy in a different namespace that overrides the deny-all policy created earlier and accepts the API keys as follows:

   kubectl apply -f - <<EOF
   apiVersion: kuadrant.io/v1
   kind: AuthPolicy
   metadata:
     name: toystore-auth
     namespace: ${KUADRANT_DEVELOPER_NS}
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: toystore
     defaults:
      when:
        - predicate: "request.path != '/health'"
      rules:
       authentication:
         "api-key-users":
           apiKey:
             selector:
               matchLabels:
                 app: toystore
           credentials:
             authorizationHeader:
               prefix: APIKEY
       response:
         success:
           filters:
             "identity":
               json:
                 properties:
                   "userid":
                     selector: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
   EOF

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a new RateLimitPolicy in a different namespace to override the default low-limit policy created previously and set rate limits for specific users as follows:

   kubectl apply -f - <<EOF
   apiVersion: kuadrant.io/v1
   kind: RateLimitPolicy
   metadata:
     name: toystore-rlp
     namespace: ${KUADRANT_DEVELOPER_NS}
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: toystore
     limits:
       "general-user":
         rates:
   
         - limit: 5
           window: 10s
         counters:
         - expression: auth.identity.userid
         when:
         - predicate: "auth.identity.userid != 'bob'"
       "bob-limit":
         rates:
         - limit: 2
           window: 10s
         when:
         - predicate: "auth.identity.userid == 'bob'"
   EOF

   Copy to Clipboard Toggle word wrap

   It might take a few minutes for the RateLimitPolicy to be applied, depending on your cluster.

2. Check that the RateLimitPolicy has a status of Accepted and Enforced as follows:

   $ kubectl get ratelimitpolicy -n ${KUADRANT_DEVELOPER_NS} toystore-rlp -o=jsonpath='{.status.conditions[?(@.type=="Accepted")].message}{"\n"}{.status.conditions[?(@.type=="Enforced")].message}'

   Copy to Clipboard Toggle word wrap

3. Check that the status of the HTTPRoute is now affected by the RateLimitPolicy in the same namespace:

   $ kubectl get httproute toystore -n ${KUADRANT_DEVELOPER_NS} -o=jsonpath='{.status.parents[0].conditions[?(@.type=="kuadrant.io/RateLimitPolicyAffected")].message}'

   Copy to Clipboard Toggle word wrap

Verification

1. Send requests as user alice as follows:

   $ while :; do curl -k --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' "https://api.$KUADRANT_ZONE_ROOT_DOMAIN/cars" | grep -E --color "\b(429)\b|$"; sleep 1; done

   Copy to Clipboard Toggle word wrap

   You should see HTTP status 200 every second for 5 seconds, followed by HTTP status 429 every second for 5 seconds.

2. Send requests as user bob as follows:

   $ while :; do curl -k --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMBOB' "https://api.$KUADRANT_ZONE_ROOT_DOMAIN/cars" | grep -E --color "\b(429)\b|$"; sleep 1; done

   Copy to Clipboard Toggle word wrap

   You should see HTTP status 200 every second for 2 seconds, followed by HTTP status 429 every second for 8 seconds.

Procedure

1. Set up your cluster. Set the following environment variables for your cluster context:

   $ export CTX_PRIMARY=$(kubectl config current-context) \
     export KUBECONFIG=~/.kube/config \
     export PRIMARY_CLUSTER_NAME=local-cluster \
     export ONPREM_DOMAIN=<onprem-domain> \
     export KUADRANT_SUBDOMAIN=""

   Copy to Clipboard Toggle word wrap

   For the ONPREM_DOMAIN variable value, use your actual root domain. For the KUADRANT_SUBDOMAIN variable value, valid values are empty or kuadrant.

2. Extract the CoreDNS manifests from dns-operator bundle by running the following commands:

   $ podman create --name bundle registry.redhat.io/rhcl-1/dns-operator-bundle:rhcl-1.3.0

   Copy to Clipboard Toggle word wrap

   $ podman cp bundle:/coredns/manifests.yaml ./coredns-manifests.yaml

   Copy to Clipboard Toggle word wrap

   $ podman rm bundle

   Copy to Clipboard Toggle word wrap

3. Apply the manifests to the cluster by running the following command:

   $ oc apply -f ./coredns-manifests.yaml

   Copy to Clipboard Toggle word wrap

4. Create a ConfigMap to define the authoritative zone for CoreDNS. This minimal configuration enables the kuadrant plugin and GeoIP features.

   $ cat | oc --context $CTX_PRIMARY apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: coredns-kuadrant-config
     namespace: kuadrant-coredns
   data:
     Corefile: |
       ${KUADRANT_SUBDOMAIN}.${ONPREM_DOMAIN}:53 {
           debug
           errors
           health {
               lameduck 5s
           }
           ready
           log
           geoip <GeoIP-database-name>.mmdb {
               edns-subnet
           }
           metadata
           kuadrant
       }

   Copy to Clipboard Toggle word wrap
   Note

   For production or exact GeoIP routing, mount your licensed MaxMind GeoIP database into the CoreDNS pod and update the filename in the data.Corefile.geoip parameter.

5. Update the CoreDNS deployment to use the new configuration by running the following command:

   $ oc --context $CTX_PRIMARY -n kuadrant-system patch deployment kuadrant-coredns --patch '{"spec":{"template":{"spec":{"volumes":[{"name":"config-volume","configMap":{"name":"coredns-kuadrant-config","items":[{"key":"Corefile","path":"Corefile"}]}}]}}}}'

   Copy to Clipboard Toggle word wrap

6. Set a watch-and-wait command for the deployment rollout to complete:

   $ oc --context $CTX_PRIMARY -n kuadrant-system rollout status deployment/kuadrant-coredns

   Copy to Clipboard Toggle word wrap

   Example output

   kuadrant-coredns successfully rolled out

   Copy to Clipboard Toggle word wrap

7. Create the Kubernetes Secret that Connectivity Link uses to interact with CoreDNS. This secret specifies the zones this provider instance is authoritative for.

   $ oc create secret generic coredns-credentials \
     --namespace=kuadrant-system \
     --type=kuadrant.io/coredns \
     --from-literal=ZONES="${KUADRANT_SUBDOMAIN}.${ONPREM_DOMAIN}" \
     --context ${CTX_PRIMARY}

   Copy to Clipboard Toggle word wrap

Procedure

1. Set up the primary cluster. Set the following environment variables for your primary cluster context:

   $ export CTX_PRIMARY=<primary_cluster_context_name> # such as, primary \
     export KUBECONFIG=~/.kube/config # Adjust path if necessary \
     export PRIMARY_CLUSTER_NAME=<primary_cluster_name> # such as, primary \
     export ONPREM_DOMAIN=<onprem-domain> # such as, example.local \
     export KUADRANT_SUBDOMAIN=kuadrant # Subdomain to delegate

   Copy to Clipboard Toggle word wrap

   Replace <primary_cluster_context_name> with the name of the cluster that you are specifying as primary. For the ONPREM_DOMAIN variable value, use your actual root domain.

2. Extract the CoreDNS manifests from the dns-operator bundle by running the following commands:

   $ podman create --name bundle registry.redhat.io/rhcl-1/dns-operator-bundle:rhcl-1.3.0

   Copy to Clipboard Toggle word wrap

   $ podman cp bundle:/coredns/manifests.yaml ./coredns-manifests.yaml

   Copy to Clipboard Toggle word wrap

   $ podman rm bundle

   Copy to Clipboard Toggle word wrap

3. Apply the manifests to the cluster by running the following command:

   $ oc apply -f ./coredns-manifests.yaml

   Copy to Clipboard Toggle word wrap

4. Wait for the CoreDNS service to get an external IP address. You need the IP address to configure delegation on your authoritative on-premise DNS server. Retrieve and store the IP address by running the following command:

   $ export COREDNS_IP_PRIMARY=$(oc --context $CTX_PRIMARY -n kuadrant-system get service kuadrant-coredns -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   echo "CoreDNS Primary IP: ${COREDNS_IP_PRIMARY}"

   Copy to Clipboard Toggle word wrap

5. Create a ConfigMap to define the authoritative zone for CoreDNS on the primary cluster. This minimal configuration enables the kuadrant plugin and GeoIP features.

   $ cat | oc --context $CTX_PRIMARY apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: coredns-kuadrant-config
     namespace: kuadrant-coredns
   data:
     Corefile: |
       ${KUADRANT_SUBDOMAIN}.${ONPREM_DOMAIN}:53 {
           debug
           errors
           health {
               lameduck 5s
           }
           ready
           log
           geoip <GeoIP-database-name>.mmdb {
               edns-subnet
           }
           metadata
           kuadrant
       }

   Copy to Clipboard Toggle word wrap
   Note

   For production or exact GeoIP routing, mount your licensed MaxMind GeoIP database into the CoreDNS pod and update the filename in the data.Corefile.geoip parameter.

6. Update the CoreDNS deployment to use the new configuration:

   $ oc --context $CTX_PRIMARY -n kuadrant-system patch deployment kuadrant-coredns --patch '{"spec":{"template":{"spec":{"volumes":[{"name":"config-volume","configMap":{"name":"coredns-kuadrant-config","items":[{"key":"Corefile","path":"Corefile"}]}}]}}}}'

   Copy to Clipboard Toggle word wrap

7. Set a watch-and-wait command for the deployment rollout to complete:

   $ oc --context $CTX_PRIMARY -n kuadrant-system rollout status deployment/kuadrant-coredns

   Copy to Clipboard Toggle word wrap

   Example output

   kuadrant-coredns successfully rolled out

   Copy to Clipboard Toggle word wrap

8. Create the Kubernetes Secret that Connectivity Link uses to interact with CoreDNS. This secret specifies the zones this provider instance is authoritative for.

   $ oc create secret generic coredns-credentials \
     --namespace=kuadrant-system \
     --type=kuadrant.io/coredns \
     --from-literal=ZONES="${KUADRANT_SUBDOMAIN}.${ONPREM_DOMAIN}" \
     --context ${CTX_PRIMARY}

   Copy to Clipboard Toggle word wrap

9. On your authoritative on-premise DNS server, configure delegation for the ${KUADRANT_SUBDOMAIN}.${ONPREM_DOMAIN} subdomain to the external IP addresses of the CoreDNS services running on your primary and secondary clusters, $COREDNS_IP_PRIMARY and $COREDNS_IP_SECONDARY. The specific steps depend on your DNS server software, for example, BIND, Windows DNS Server. You typically need to add Name Server (NS) records pointing the subdomain to the CoreDNS IP addresses.

   Example delegation

   ; Delegate kuadrant.example.local to CoreDNS instances
   $ORIGIN ${KUADRANT_SUBDOMAIN}.${ONPREM_DOMAIN}.
   @       IN      SOA     ns1.${ONPREM_DOMAIN}. hostmaster.${ONPREM_DOMAIN}. (
                           2023102601 ; serial
                           7200       ; refresh (2 hours)
                           3600       ; retry (1 hour)
                           1209600    ; expire (2 weeks)
                           3600       ; minimum (1 hour)
                           )
           IN      NS      coredns-primary.${KUADRANT_SUBDOMAIN}.${ONPREM_DOMAIN}.
   
   coredns-primary   IN A ${COREDNS_IP_PRIMARY}

   Copy to Clipboard Toggle word wrap

10. Restart CoreDNS by running the following command:

    $ oc -n kuadrant-coredns rollout restart deployment kuadrant-coredns

    Copy to Clipboard Toggle word wrap
    Note

    After configuring delegation, you can test that the DNS resolution for the delegated subdomain works correctly by querying your authoritative DNS server for a record within the kuadrant subdomain. One of the CoreDNS instances is expected to refer to and answer the query.

Verification

1. Launch a temporary pod for testing by running the following command:

   $ oc debug node/<node-name>

   Copy to Clipboard Toggle word wrap

   Replace <node-name> with the node you are testing on.

2. Add transfer to your Corefile by running the following command:

   $ oc patch cm kuadrant-coredns -n kuadrant-coredns --type merge \
      -p "$(kubectl get cm kuadrant-coredns -n kuadrant-coredns -o jsonpath='{.data.Corefile}' | \
      sed 's/kuadrant/transfer {\n        to *\n    }\n    kuadrant/' | \
      jq -Rs '{data: {Corefile: .}}')"

   Copy to Clipboard Toggle word wrap

3. Verify zone delegation by running the following command:

   $ dig @${EDGE_NS} -k config/bind9/ddns.key -t AXFR example.com

   Copy to Clipboard Toggle word wrap

   Example output

   example.com.            30      IN      SOA     example.com. root.example.com. 17 30 30 30 30
   example.com.            30      IN      NS      ns.example.com.
   k.example.com.          300     IN      NS      ns1.k.example.com.
   ns1.k.example.com.      300     IN      A       172.18.0.16
   ns.example.com.         30      IN      A       127.0.0.1
   example.com.            30      IN      SOA     example.com. root.example.com. 17 30 30 30 30

   Copy to Clipboard Toggle word wrap

   In this example, k.example is the delegated zone and ns1.k.example is the primary zone.

4. Optional. Remove the transfer from your Corefile by running the following command:

   $ oc patch cm kuadrant-coredns -n kuadrant-coredns --type merge \
      -p "$(kubectl get cm kuadrant-coredns -n kuadrant-coredns -o jsonpath='{.data.Corefile}' | \
      sed '/transfer {/,/}/d' | \
      jq -Rs '{data: {Corefile: .}}')"

   Copy to Clipboard Toggle word wrap

5. Verify the start of authority (SOA) record for the delegated zone by running the following command:

   $ dig @${EDGE_NS} soa k.example.com

   Copy to Clipboard Toggle word wrap

   Example output

   ;; ANSWER SECTION:
   k.example.com.          60      IN      SOA     ns1.k.example.com. hostmaster.k.example.com. 12345 7200 1800 86400 60

   Copy to Clipboard Toggle word wrap

   The SOA record is expected to show the primary name server (NS) as confirmation that CoreDNS is responding authoritatively. In this example the primary NS is ns1.k.example.com.