Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 7. Serving transport encryption

You can enable OpenShift Serverless Serving transport encryption to allow transporting data over secured and encrypted HTTPS connections using TLS.

Important

OpenShift Serverless Serving transport encryption is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Important

Serving Transport Encryption is only available for Kourier as an ingress layer. For Red Hat OpenShift Service Mesh, use the service mesh mTLS capabilities to ensure encrypted traffic.

7.1. Overview of Serving transport encryption
Copy link

There are three parts to OpenShift Serverless Serving transport encryption:

External domain encryption
Transport encryption on the ingress layer external to the cluster. For example, a cluster-external domain, such as myapp-<namespace>.example.com.
Cluster-local encryption
Transport encryption on the ingress layer internal to the cluster. For example, cluster-local domains, such as myapp.<namespace>.svc.cluster.local.
System-internal encryption
Transport encryption between the ingress-gateway, activator, and queue-proxy Knative internal components.
Important

Control-plane traffic, including Kubernetes PreStopHooks, metadata, and metrics, contains no user data and is not encrypted.

The different parts are independent of each other and can be enabled and disabled individually. They can use the same or different Certificate Authorities (CAs) to sign the necessary certificates.

Note

For diagrams illustrating transport encryption, see OpenShift Serverless Serving Transport Encryption.

7.1.1. External domain encryption
Copy link

The transport encryption for external domains is handled by the ingress layer of the cluster, which is either the OpenShift Container Platform ingress or Red Hat OpenShift Service Mesh.

7.1.2. Cluster-local encryption
Copy link

Cluster-local encryption enables transport encryption for cluster-local domains. It has the following properties:

  • The Certificate Common Name (CN) or Subject Alternative Name (SAN) contains the cluster-local domains of a Knative Service, for example myapp.namespace.svc.cluster.local, myapp.namespace.svc, myapp.namespace.
  • The cluster-local endpoint of the ingress-controller component uses SNI to select the certificates.
  • To create the certificates, Knative relies on cert-manager, which needs to be installed and configured for the feature to work. For more information, see cert-manager Operator for Red Hat OpenShift.
Important

The caller must trust the CA that signed the cluster-local certificates. This is out of the scope of OpenShift Serverless.

7.1.3. System-internal encryption
Copy link

System-internal encryption enables transport encryption for the ingress-gateway, activator, and queue-proxy Knative internal components. These components host TLS endpoints when this configuration is used.

The following prerequisites must be satisfied to use this feature:

  • For OpenShift Serverless to get the certificates, cert-manager must be installed and configured.
  • Specific SANs are used to verify each connection. Each component must trust the CA that signed the certificates. To satisfy this requirement, OpenShift Serverless system components consume and trust a bundle of CAs. The CA bundle must be provided by the cluster administrator.

7.2. Choice of a certificate issuer
Copy link

Issuers refer to cert-manager issuers and cluster issuers. They represent certificate authorities (CAs) that can generate signed certificates by honoring certificate signing requests. For more information, see cert-manager documentation on issuers.

Depending on the encryption features that you use, OpenShift Serverless requires your certificate issuer to be able to sign certain certificates. To identify your certificate issuer, refer to the list of cert-manager integrations, which contains examples for the following:

  • A custom CA stored in a Kubernetes secret
  • HTTP-01 challenges
  • DNS-01 challenges
  • Self-signed issuers

7.2.1. Compatible certificate issuers
Copy link

Not all issuer types work for each Knative Serving encryption feature.

  • For cluster-local encryption, the issuer must be able to sign certificates for the following cluster-local domain types:

    • myapp.<namespace>
    • myapp.<namespace>.svc
    • myapp.<namespace>.svc.cluster.local

    As the CA usually is not within the cluster, verification using the Automated Certificate Management Environment (ACME) protocol (DNS01/HTTP01) is not possible. You can use an issuer that allows creating these certificates, such as the cert-manager CA issuer.

  • For system-internal encryption, the issuer must be able to sign certificates with the following Subject Alternative Names (SANs):

    • kn-routing
    • names of format kn-user-<namespace>, where <namespace> is a namespace where Knative Services are created
    • data-plane.knative.dev

    Knative requires these SANs to verify connections between the internal components. Because this is not possible using the ACME protocol (DNS01/HTTP01), you must configure an issuer that allows creating these certificates, for example, cert-manager CA issuer.

Prerequisites

  • You have access to an OpenShift Container Platform account with cluster administrator access.
  • Install the {oc-first}.
  • Install the cert-manager Operator for Red Hat OpenShift.
  • Install the OpenShift Serverless Operator.
Important

If you install the OpenShift Serverless Operator before installing the cert-manager Operator for Red Hat OpenShift, you must restart the controller and activator deployments in the knative-serving namespace. Failure to restart these deployments prevents Knative from creating the necessary cert-manager resources, which results in pending Knative Services and prevents enabling the Knative Serving cert-manager integration.

7.3.1. Configuring a SelfSigned cluster issuer
Copy link

The following procedure uses a SelfSigned issuer as the root certificate. For information about the implications and limitations of this method, see the SelfSigned cert-manager documentation.

If you manage your own company-specific Private Key Infrastructure (PKI), use the CA issuer. For more information, see cert-manager documentation on CA issuers.

Procedure

  1. Create a SelfSigned ClusterIssuer custom resource (CR):

    Example ClusterIssuer CR

    apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: knative-serving-selfsigned-issuer
spec:
  selfSigned: {}
    Copy to Clipboard Toggle word wrap

  2. Apply the ClusterIssuer CR by running the following command: 

    $ oc apply -f <filename>
    Copy to Clipboard Toggle word wrap

  3. Create a root certificate that refers to the ClusterIssuer CR:

    Example root certificate

    apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: knative-serving-selfsigned-ca
  namespace: cert-manager
    1 
    
spec:
  secretName: knative-serving-ca
    2 
    

  isCA: true
  commonName: selfsigned-ca
  privateKey:
    algorithm: ECDSA
    size: 256

  issuerRef:
    name: knative-serving-selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io
    Copy to Clipboard Toggle word wrap

    1
    The cert-manager Operator for Red Hat OpenShift namespace, cert-manager by default.
    2
    Secret name later used for the ClusterIssuer CR for Knative Serving.

  4. Apply the Certificate CR by running the following command: 

    $ oc apply -f <filename>
    Copy to Clipboard Toggle word wrap

To enable the use of certificates by Serving, you must create a cluster issuer.

Procedure

  1. Create the knative-serving-ca-issuer ClusterIssuer for Serving: 

    apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: knative-serving-ca-issuer
spec:
  ca:
    secretName: knative-serving-ca
    1
    Copy to Clipboard Toggle word wrap
    1
    Secret name in the cert-manager Operator for Red Hat OpenShift namespace (cert-manager by default) containing the certificate that can be used by OpenShift Serverless Serving components for new certificates.

  2. Apply the ClusterIssuer resource by running the following command: 

    $ oc apply -f <filename>
    Copy to Clipboard Toggle word wrap

7.3.3. Configuring transport encryption
Copy link

Configuring transport encryption consists of two parts:

  1. Specifying the ClusterIssuer issuer to use:

    • clusterLocalIssuerRef: issuer for cluster-local-domain certificates used for ingress.
    • systemInternalIssuerRef: issuer for certificates for system-internal-tls certificates used by Knative internal components.

  2. Specifying transport encryption features to use:

    • cluster-local-domain-tls: Enables the transport encryption feature for cluster-local domains
    • system-internal-tls: Enables the transport encryption feature for OpenShift Serverless Serving internal components.

Procedure

  1. Enable transport encryption in the KnativeServing resource: 

    apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  ...
  config:
    certmanager:
      clusterLocalIssuerRef: |
        kind: ClusterIssuer
        name: knative-serving-ca-issuer
    1 
    
      systemInternalIssuerRef: |
        kind: ClusterIssuer
        name: knative-serving-ca-issuer
    2 
    
    network:
      cluster-local-domain-tls: Enabled
    3 
    
      system-internal-tls: Enabled
    4
    Copy to Clipboard Toggle word wrap
    1
    Define the cluster issuer for each feature. The same or individual cluster issuers can be used.
    2
    Define the cluster issuer.
    3
    Enable the cluster-local-domain-tls feature. This and other features can be enabled or disabled individually.
    4
    Enable the system-internal-tls feature.

  2. Apply the KnativeServing resource by running the following command: 

    $ oc apply -f <filename>
    Copy to Clipboard Toggle word wrap

  3. Optionally, change the defaultCertificate value in the Ingress Controller: 

    apiVersion: operator.openshift.io/v1
kind: IngressController
 ...
spec:
  defaultCertificate:
    name: ca-ingress-cert
    Copy to Clipboard Toggle word wrap

  4. If you changed the defaultCertificate value, you must specify the custom certificate name in the openshift-ingress-default-certificate field in the KnativeServing custom resource.

    For example, if the custom certificate name is ca-ingress-cert, add the following configuration: 

    ...
spec:
  config:
    network:
      system-internal-tls: Enabled
      openshift-ingress-default-certificate: "ca-ingress-cert"
...
    Copy to Clipboard Toggle word wrap

  5. If you enabled cluster-local-domain-tls or system-internal-tls, restart the Controller component by running the following command.

    Important

    When either the cluster-local-domain-tls or the system-internal-tls feature is enabled, you must restart the Controller component to enable the Knative Serving cert-manager integration. 

    $ oc rollout restart deploy/controller -n knative-serving
    Copy to Clipboard Toggle word wrap

  6. If you enabled system-internal-tls, restart the Activator component by running the following command.

    Important

    When the system-internal-tls feature is activated, you must restart the Activator component to reconfigure its internal web server, as this is not possible during runtime. 

    $ oc rollout restart deploy/activator -n knative-serving
    Copy to Clipboard Toggle word wrap

7.4. Trust configuration
Copy link

When you enable any of the transport encryption features, you must make sure that all clients calling trust the Certificate Authority (CA) issuing the certificates used for the transport encryption.

There are multiple places where trust must be ensured:

  • Cluster external client, such as Browser or other application. This is out of the scope of OpenShift Serverless.
  • OpenShift Serverless system components, such as Activator, Queue-Proxy, and Ingress-Controller.
  • Cluster internal client, such as a Knative Service or other workload.

To ensure that OpenShift Serverless Serving components and Knative Services trust the CA that issues certificates, you can create a ConfigMap in the following namespaces with the label networking.knative.dev/trust-bundle: true:

knative-serving
for the system components of OpenShift Serverless Serving.
knative-serving-ingress
for the ingress layer of OpenShift Serverless Serving.
istio-system or your own Service Mesh namespace
when the Service Mesh integration is enabled.

Knative reads all data keys in ConfigMaps with this label, regardless of the name. One key can contain one or multiple CAs or intermediate certificates. If they are valid, they are added to the trust store of the Knative components.

This is an example ConfigMap: 

apiVersion: v1
data:
  cacerts.pem: |
1 

    -----BEGIN CERTIFICATE-----
    MIIDDTCCAfWgAwIBAgIQMQuip05h7NLQq2TB+j9ZmTANBgkqhkiG9w0BAQsFADAW
    MRQwEgYDVQQDEwtrbmF0aXZlLmRldjAeFw0yMzExMjIwOTAwNDhaFw0yNDAyMjAw
    OTAwNDhaMBYxFDASBgNVBAMTC2tuYXRpdmUuZGV2MIIBIjANBgkqhkiG9w0BAQEF
    AAOCAQ8AMIIBCgKCAQEA3clC3CV7sy0TpUKNuTku6QmP9z8JUCbLCPCLACCUc1zG
    FEokqOva6TakgvAntXLkB3TEsbdCJlNm6qFbbko6DBfX6rEggqZs40x3/T+KH66u
    4PvMT3fzEtaMJDK/KQOBIvVHrKmPkvccUYK/qWY7rgBjVjjLVSJrCn4dKaEZ2JNr
    Fd0KNnaaW/dP9/FvviLqVJvHnTMHH5qyRRr1kUGTrc8njRKwpHcnUdauiDoWRKxo
    Zlyy+MhQfdbbyapX984WsDjCvrDXzkdGgbRNAf+erl6yUm6pHpQhyFFo/zndx6Uq
    QXA7jYvM2M3qCnXmaFowidoLDsDyhwoxD7WT8zur/QIDAQABo1cwVTAOBgNVHQ8B
    Af8EBAMCAgQwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUwAwEB/zAd
    BgNVHQ4EFgQU7p4VuECNOcnrP9ulOjc4J37Q2VUwDQYJKoZIhvcNAQELBQADggEB
    AAv26Vnk+ptQrppouF7yHV8fZbfnehpm07HIZkmnXO2vAP+MZJDNrHjy8JAVzXjt
    +OlzqAL0cRQLsUptB0btoJuw23eq8RXgJo05OLOPQ2iGNbAATQh2kLwBWd/CMg+V
    KJ4EIEpF4dmwOohsNR6xa/JoArIYH0D7gh2CwjrdGZr/tq1eMSL+uZcuX5OiE44A
    2oXF9/jsqerOcH7QUMejSnB8N7X0LmUvH4jAesQgr7jo1JTOBs7GF6wb+U76NzFa
    8ms2iAWhoplQ+EHR52wffWb0k6trXspq4O6v/J+nq9Ky3vC36so+G1ZFkMhCdTVJ
    ZmrBsSMWeT2l07qeei2UFRU=
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  labels:
    networking.knative.dev/trust-bundle: "true"
  name: knative-bundle
2 

  namespace: knative-serving
Copy to Clipboard Toggle word wrap
1
Serving components trust all keys containing valid PEM-encoded CA bundles.
2
You can use an arbitrary name.
Important

When a CA bundle ConfigMap is created or updated, the Serving components automatically pick them up and add the CAs or intermediate certificates to their CA trust store. The trust store is refreshed for every new HTTP connection.

7.4.2. Trust configuration on your custom workload
Copy link

As OpenShift Serverless Serving does not control all workloads and managing trust is highly dependent on your runtime and language, custom workloads are out of the scope of OpenShift Serverless. The following are other options for custom workloads:

  • Adding a CA bundle to a Container image on build-time. Note that this complicates CA rotation, as you must rebuild and redeploy every application when the CA rotates.
  • Mounting a CA bundle to the filesystem, such as from a Secret or ConfigMap, and making sure your application uses it to verify TLS connections.
  • Reading a CA bundle from an environment variable and making sure that your application uses it to verify TLS connections.
  • Accessing a CA bundle from a secret or ConfigMap using Kubernetes API and making sure your application uses it to verify TLS connections.

7.5. Ensuring seamless CA rotation
Copy link

Ensuring seamless CA rotation is essential to avoid service downtime, or to deal with an emergency.

Procedure

  1. Create a new CA certificate.
  2. Add the public key of the new CA certificate to the CA trust bundles as described in the "Trust configuration for OpenShift Serverless Operator Serving components and Knative Services" section. Retain the public key of the existing CA.
  3. Ensure that all clients have consumed the latest set of CA trust bundles. OpenShift Serverless Serving components will automatically reload the changed CA trust bundles.
  4. If you have custom workload consuming trust bundles, reload or restart them accordingly.
  5. Update the knative-serving-ca-issuer cluster issuer to reference the secret containing the new CA certificate.
  6. Either wait for cert-manager to renew all your certificates or enforce it to renew all the certificates. For more information, see the cert-manager documentation.
  7. Once the CA rotation is fully completed, you can remove the public key of the old CA from the trust bundle configmap. Allow enough time for all components to apply the changes.

7.6. Verifying transport encryption is enabled
Copy link

Procedure

  1. Create a Knative Service: 

    apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: test-webapp
  namespace: test-namespace
spec:
  template:
    spec:
      containers:
        - image: docker.io/openshift/hello-openshift
          env:
            - name: RESPONSE
              value: "Hello Serverless!"
    Copy to Clipboard Toggle word wrap

  2. Apply the Knative Service YAML by running the following command: 

    $ oc apply -f <filename>
    Copy to Clipboard Toggle word wrap

  3. Examine status of the Knative Service:

    Example command

    $ oc get ksvc -n test-namespace -o yaml
    Copy to Clipboard Toggle word wrap

    Example output

    apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: test-webapp
  namespace: test-namespace
# spec:
# ...
status:
  address:
    # cluster-local-domain:
    url: https://helloworld.test.svc.cluster.local
    1
    Copy to Clipboard Toggle word wrap

    1
    If you have enabled cluster-local-domain-tls, you will see the HTTPS URL.

  4. To verify that system-internal-tls is enabled, check the output of Queue-Proxy logs by running the following command:

    Example command

    $ oc logs your-pod -n test-namespace -c queue-proxy | grep -E 'certDir|Certificate|tls'
    Copy to Clipboard Toggle word wrap

    If you see lines similar to the following, system-internal-tls is enabled: 

    {"severity":"INFO","timestamp":"2024-01-03T07:07:32.892810888Z","logger":"queueproxy","caller":"certificate/watcher.go:62","message":"Starting to watch the following directories for changes{certDir 15 0 /var/lib/knative/certs <nil>} {keyDir 15 0 /var/lib/knative/certs <nil>}","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}
{"severity":"INFO","timestamp":"2024-01-03T07:07:32.89397512Z","logger":"queueproxy","caller":"certificate/watcher.go:131","message":"Certificate and/or key have changed on disk and were reloaded.","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}
{"severity":"INFO","timestamp":"2024-01-03T07:07:32.894232939Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server admin:8022","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}
{"severity":"INFO","timestamp":"2024-01-03T07:07:32.894268548Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server main:8112","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}
    Copy to Clipboard Toggle word wrap
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat