Serving
Getting started with Knative Serving and configuring services
Abstract
Chapter 1. Getting started with Knative Serving
1.1. Creating serverless applications
Serverless applications are created and deployed as Kubernetes services, defined by a route and a configuration, and contained in a YAML file. To deploy a serverless application using OpenShift Serverless, you must create a Knative Service
object.
Example Knative Service
object YAML file
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: showcase 1 namespace: default 2 spec: template: spec: containers: - image: quay.io/openshift-knative/showcase 3 env: - name: GREET 4 value: Ciao
You can create a serverless application by using one of the following methods:
Create a Knative service from the OpenShift Container Platform web console.
For OpenShift Container Platform, see Creating applications using the Developer perspective for more information.
-
Create a Knative service by using the Knative (
kn
) CLI. -
Create and apply a Knative
Service
object as a YAML file, by using theoc
CLI.
1.1.1. Creating serverless applications by using the Knative CLI
Using the Knative (kn
) CLI to create serverless applications provides a more streamlined and intuitive user interface over modifying YAML files directly. You can use the kn service create
command to create a basic serverless application.
Prerequisites
- OpenShift Serverless Operator and Knative Serving are installed on your cluster.
-
You have installed the Knative (
kn
) CLI. - You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
Procedure
Create a Knative service:
$ kn service create <service-name> --image <image> --tag <tag-value>
Where:
-
--image
is the URI of the image for the application. --tag
is an optional flag that can be used to add a tag to the initial revision that is created with the service.Example command
$ kn service create showcase \ --image quay.io/openshift-knative/showcase
Example output
Creating service 'showcase' in namespace 'default': 0.271s The Route is still working to reflect the latest desired specification. 0.580s Configuration "showcase" is waiting for a Revision to become ready. 3.857s ... 3.861s Ingress has not yet been reconciled. 4.270s Ready to serve. Service 'showcase' created with latest revision 'showcase-00001' and URL: http://showcase-default.apps-crc.testing
-
1.1.2. Creating serverless applications using YAML
Creating Knative resources by using YAML files uses a declarative API, which enables you to describe applications declaratively and in a reproducible manner. To create a serverless application by using YAML, you must create a YAML file that defines a Knative Service
object, then apply it by using oc apply
.
After the service is created and the application is deployed, Knative creates an immutable revision for this version of the application. Knative also performs network programming to create a route, ingress, service, and load balancer for your application and automatically scales your pods up and down based on traffic.
Prerequisites
- OpenShift Serverless Operator and Knative Serving are installed on your cluster.
- You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
-
Install the OpenShift CLI (
oc
).
Procedure
Create a YAML file containing the following sample code:
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: showcase namespace: default spec: template: spec: containers: - image: quay.io/openshift-knative/showcase env: - name: GREET value: Bonjour
Navigate to the directory where the YAML file is contained, and deploy the application by applying the YAML file:
$ oc apply -f <filename>
If you do not want to switch to the Developer perspective in the OpenShift Container Platform web console or use the Knative (kn
) CLI or YAML files, you can create Knative components by using the Administator perspective of the OpenShift Container Platform web console.
1.1.3. Creating serverless applications using the Administrator perspective
Serverless applications are created and deployed as Kubernetes services, defined by a route and a configuration, and contained in a YAML file. To deploy a serverless application using OpenShift Serverless, you must create a Knative Service
object.
Example Knative Service
object YAML file
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: showcase 1 namespace: default 2 spec: template: spec: containers: - image: quay.io/openshift-knative/showcase 3 env: - name: GREET 4 value: Ciao
After the service is created and the application is deployed, Knative creates an immutable revision for this version of the application. Knative also performs network programming to create a route, ingress, service, and load balancer for your application and automatically scales your pods up and down based on traffic.
Prerequisites
To create serverless applications using the Administrator perspective, ensure that you have completed the following steps.
- The OpenShift Serverless Operator and Knative Serving are installed.
- You have logged in to the web console and are in the Administrator perspective.
Procedure
- Navigate to the Serverless → Serving page.
- In the Create list, select Service.
- Manually enter YAML or JSON definitions, or by dragging and dropping a file into the editor.
- Click Create.
1.1.4. Creating a service using offline mode
You can execute kn service
commands in offline mode, so that no changes happen on the cluster, and instead the service descriptor file is created on your local machine. After the descriptor file is created, you can modify the file before propagating changes to the cluster.
The offline mode of the Knative CLI is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Prerequisites
- OpenShift Serverless Operator and Knative Serving are installed on your cluster.
-
You have installed the Knative (
kn
) CLI.
Procedure
In offline mode, create a local Knative service descriptor file:
$ kn service create showcase \ --image quay.io/openshift-knative/showcase \ --target ./ \ --namespace test
Example output
Service 'showcase' created in namespace 'test'.
The
--target ./
flag enables offline mode and specifies./
as the directory for storing the new directory tree.If you do not specify an existing directory, but use a filename, such as
--target my-service.yaml
, then no directory tree is created. Instead, only the service descriptor filemy-service.yaml
is created in the current directory.The filename can have the
.yaml
,.yml
, or.json
extension. Choosing.json
creates the service descriptor file in the JSON format.The
--namespace test
option places the new service in thetest
namespace.If you do not use
--namespace
, and you are logged in to an OpenShift Container Platform cluster, the descriptor file is created in the current namespace. Otherwise, the descriptor file is created in thedefault
namespace.
Examine the created directory structure:
$ tree ./
Example output
./ └── test └── ksvc └── showcase.yaml 2 directories, 1 file
-
The current
./
directory specified with--target
contains the newtest/
directory that is named after the specified namespace. -
The
test/
directory contains theksvc
directory, named after the resource type. -
The
ksvc
directory contains the descriptor fileshowcase.yaml
, named according to the specified service name.
-
The current
Examine the generated service descriptor file:
$ cat test/ksvc/showcase.yaml
Example output
apiVersion: serving.knative.dev/v1 kind: Service metadata: creationTimestamp: null name: showcase namespace: test spec: template: metadata: annotations: client.knative.dev/user-image: quay.io/openshift-knative/showcase creationTimestamp: null spec: containers: - image: quay.io/openshift-knative/showcase name: "" resources: {} status: {}
List information about the new service:
$ kn service describe showcase --target ./ --namespace test
Example output
Name: showcase Namespace: test Age: URL: Revisions: Conditions: OK TYPE AGE REASON
The
--target ./
option specifies the root directory for the directory structure containing namespace subdirectories.Alternatively, you can directly specify a YAML or JSON filename with the
--target
option. The accepted file extensions are.yaml
,.yml
, and.json
.The
--namespace
option specifies the namespace, which communicates tokn
the subdirectory that contains the necessary service descriptor file.If you do not use
--namespace
, and you are logged in to an OpenShift Container Platform cluster,kn
searches for the service in the subdirectory that is named after the current namespace. Otherwise,kn
searches in thedefault/
subdirectory.
Use the service descriptor file to create the service on the cluster:
$ kn service create -f test/ksvc/showcase.yaml
Example output
Creating service 'showcase' in namespace 'test': 0.058s The Route is still working to reflect the latest desired specification. 0.098s ... 0.168s Configuration "showcase" is waiting for a Revision to become ready. 23.377s ... 23.419s Ingress has not yet been reconciled. 23.534s Waiting for load balancer to be ready 23.723s Ready to serve. Service 'showcase' created to latest revision 'showcase-00001' is available at URL: http://showcase-test.apps.example.com
1.1.5. Verifying your serverless application deployment
To verify that your serverless application has been deployed successfully, you must get the application URL created by Knative, and then send a request to that URL and observe the output. OpenShift Serverless supports the use of both HTTP and HTTPS URLs, however the output from oc get ksvc
always prints URLs using the http://
format.
Prerequisites
- OpenShift Serverless Operator and Knative Serving are installed on your cluster.
-
You have installed the
oc
CLI. - You have created a Knative service.
Prerequisites
-
Install the OpenShift CLI (
oc
).
Procedure
Find the application URL:
$ oc get ksvc <service_name>
Example output
NAME URL LATESTCREATED LATESTREADY READY REASON showcase http://showcase-default.example.com showcase-00001 showcase-00001 True
Make a request to your cluster and observe the output.
Example HTTP request (using HTTPie tool)
$ http showcase-default.example.com
Example HTTPS request
$ https showcase-default.example.com
Example output
HTTP/1.1 200 OK Content-Type: application/json Server: Quarkus/2.13.7.Final-redhat-00003 Java/17.0.7 X-Config: {"sink":"http://localhost:31111","greet":"Ciao","delay":0} X-Version: v0.7.0-4-g23d460f content-length: 49 { "artifact": "knative-showcase", "greeting": "Ciao" }
Optional. If you don’t have the HTTPie tool installed on your system, you can likely use curl tool instead:
Example HTTPS request
$ curl http://showcase-default.example.com
Example output
{"artifact":"knative-showcase","greeting":"Ciao"}
Optional. If you receive an error relating to a self-signed certificate in the certificate chain, you can add the
--verify=no
flag to the HTTPie command to ignore the error:$ https --verify=no showcase-default.example.com
Example output
HTTP/1.1 200 OK Content-Type: application/json Server: Quarkus/2.13.7.Final-redhat-00003 Java/17.0.7 X-Config: {"sink":"http://localhost:31111","greet":"Ciao","delay":0} X-Version: v0.7.0-4-g23d460f content-length: 49 { "artifact": "knative-showcase", "greeting": "Ciao" }
ImportantSelf-signed certificates must not be used in a production deployment. This method is only for testing purposes.
Optional. If your OpenShift Container Platform cluster is configured with a certificate that is signed by a certificate authority (CA) but not yet globally configured for your system, you can specify this with the
curl
command. The path to the certificate can be passed to the curl command by using the--cacert
flag:$ curl https://showcase-default.example.com --cacert <file>
Example output
{"artifact":"knative-showcase","greeting":"Ciao"}
1.1.6. Additional resources
Chapter 2. Scalability and performance of OpenShift Serverless Serving
OpenShift Serverless consists of several different components that have different resource requirements and scaling behaviors. These components are horizontally and vertically scalable, but their resource requirements and configuration highly depend on the actual use-case.
- Control-plane components
- These components are responsible for observing and reacting to custom resources and continuously reconfiguring the system, for example, the controller pods.
- Data-plane components
- These components are directly involved in requests and response handling, for example, the Knative Servings activator component.
The following metrics and findings were recorded using the following test setup:
- A cluster running OpenShift Container Platform 4.13
- The cluster running 4 compute nodes in AWS with a machine type of m6.xlarge
- OpenShift Serverless 1.30
2.1. Overhead of OpenShift Serverless Serving
As components of OpenShift Serverless Serving are part of the data-plane, requests from clients are routed through:
- The ingress-gateway (Kourier or Service Mesh)
- The activator component
- The queue-proxy sidecar container in each Knative Service
These components introduce an additional hop in networking and perform additional tasks, for example, adding observability and request queuing. The following are the measured latency overheads:
- Each additional network hop adds 0.5 ms to 1 ms latency to a request. Depending on the current load of the Knative Service and if the Knative Service was scaled to zero before the request, the activator component is not always a part of the data-plane.
- Depending on the payload size, each of the components is consuming up to 1 vCPU of CPU for handling 2500 requests per second.
2.2. Known limitations of OpenShift Serverless Serving
The maximum number of Knative Services that can be created is 3,000. This corresponds to the OpenShift Container Platform Kubernetes services limit of 10,000, since 1 Knative Service creates 3 Kubernetes services.
2.3. Scaling and performance of OpenShift Serverless Serving
OpenShift Serverless Serving has to be scaled and configured based on the following parameters:
- Number of Knative Services
- Number of Revisions
- Amount of concurrent requests in the system
- Size of payloads of the requests
- The startup-latency and response latency of the Knative Service added by the user’s web application
- Number of changes of the KnativeService custom resource (CR) over time
2.3.1. KnativeServing default configuration
Per default, OpenShift Serverless Serving is configured to run all components with high-availability and medium-sized CPU and memory requests and limits. This means that the high-available
field in KnativeServing
CR is automatically set to a value of 2
and all system components are scaled to two replicas. This configuration is suitable for medium workload scenarios and has been tested with:
- 170 Knative Services
- 1-2 Revisions per Knative Service
- 89 test scenarios mainly focused on testing the control plane
- 48 re-creating scenarios where Knative Services are deleted and re-created
- 41 stable scenarios, in which requests are slowly but continuously sent to the system
During these test cases, the system components effectively consumed:
Component | Measured Resources |
---|---|
Operator in project | 1 GB Memory, 0.2 Cores of CPU |
Serving components in project | 5 GB Memory, 2.5 Cores of CPU |
2.3.2. Minimal requirements of OpenShift Serverless Serving
While the default setup is suitable for medium-sized workloads, it might be over-sized for smaller setups or under-sized for high-workload scenarios. To configure OpenShift Serverless Serving for a minimal workload scenario, you need to know the idle consumption of the system components.
2.3.2.1. Idle consumption
The idle consumption is dependent on the number of Knative Services. The following memory usage has been measured for the components in the knative-serving
and knative-serving-ingress
OpenShift Container Platform projects:
Component | 0 Services | 100 Services | 500 Services | 1000 Services |
---|---|---|---|---|
| 55Mi | 86Mi | 300Mi | 450Mi |
| 52Mi | 102Mi | 225Mi | 350Mi |
| 100Mi | 135Mi | 310Mi | 500Mi |
| 60Mi | 60Mi | 60Mi | 60Mi |
| 20Mi | 60Mi | 190Mi | 330Mi |
| 90Mi | 170Mi | 340Mi | 430Mi |
Either 3scale-kourier-gateway
and net-kourier-controller
components or istio-ingressgateway
and net-istio-controller
components are installed.
The memory consumption of net-istio
is based on the total number of pods within the mesh.
2.3.3. Configuring Serving for minimal workloads
Procedure
You can configure Knative Serving for minimal workloads using the
KnativeServing
custom resource (CR):A minimal workload configuration in KnativeServing CR
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: high-availability: replicas: 1 1 workloads: - name: activator replicas: 2 2 resources: - container: activator requests: cpu: 250m 3 memory: 60Mi 4 limits: cpu: 1000m memory: 600Mi - name: controller replicas: 1 5 resources: - container: controller requests: cpu: 10m memory: 100Mi limits: 6 cpu: 200m memory: 300Mi - name: webhook replicas: 2 resources: - container: webhook requests: cpu: 100m 7 memory: 60Mi limits: cpu: 200m memory: 200Mi podDisruptionBudgets: 8 - name: activator-pdb minAvailable: 1 - name: webhook-pdb minAvailable: 1
- 1
- Setting this to
1
scales all system components to one replica. - 2
- Activator should always be scaled to a minimum of
2
instances to avoid downtime. - 3
- Activator CPU requests should not be set lower than
250m
, as aHorizontalPodAutoscaler
will use this as a reference to scale up and down. - 4
- Adjust memory requests to the idle values from the previous table. Also adjust memory limits according to your expected load (this might need custom testing to find the best values).
- 5
- One webhook and one controller are sufficient for a minimal-workload scenario
- 6
- These limits are sufficient for a minimal-workload scenario, but they also might need adjustments depending on your concrete workload.
- 7
- Webhook CPU requests should not be set lower than
100m
, as a HorizontalPodAutoscaler will use this as a reference to scale up and down. - 8
- Adjust the
PodDistruptionBudgets
to a value lower thanreplicas
, to avoid problems during node maintenance.
2.3.4. Configuring Serving for high workloads
You can configure Knative Serving for high workloads using the KnativeServing
custom resource (CR). The following findings are relevant to configuring Knative Serving for a high workload:
These findings have been tested with requests with a payload size of 0-32 kb. The Knative Service backends used in those tests had a startup latency between 0 to 10 seconds and response times between 0 to 5 seconds.
- All data-plane components are mostly increasing CPU usage on higher requests and payload scenarios, so the CPU requests and limits have to be tested and potentially increased.
- The activator component also might need more memory, when it has to buffer more or bigger request payloads, so the memory requests and limits might need to be increased as well.
- One activator pod can handle approximately 2500 requests per second before it starts to increase latency and, at some point, leads to errors.
-
One
3scale-kourier-gateway
oristio-ingressgateway
pod can also handle approximately 2500 requests per second before it starts to increase latency and, at some point, leads to errors. - Each of the data-plane components consumes up to 1 vCPU of CPU for handling 2500 requests per second. Note that this highly depends on the payload size and the response times of the Knative Service backend.
Fast startup and fast response-times of your Knative Service user workloads are critical for good performance of the overall system. The Knative Serving components are buffering incoming requests when the Knative Service user backend is scaling up or when request concurrency has reached its capacity. If your Knative Service user workload introduces long startup or request latency, it will either overload the activator
component (when the CPU and memory configuration is too low) or lead to errors for the calling clients.
Procedure
To fine-tune your installation, use the previous findings combined with your own test results to configure the
KnativeServing
custom resource:A high workload configuration in KnativeServing CR
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: high-availability: replicas: 2 1 workloads: - name: component-name 2 replicas: 2 3 resources: - container: container-name requests: cpu: 4 memory: limits: cpu: memory: podDisruptionBudgets: 5 - name: name-of-pod-disruption-budget minAvailable: 1
- 1
- Set this parameter to at least
2
to make sure you always have at least two instances of every component running. You can also useworkloads
to override the replicas for certain components. - 2
- Use the
workloads
list to configure specific components. Use thedeployment
name of the component and set thereplicas
field. - 3
- For the
activator
,webhook
, and3scale-kourier-gateway
components, which use horizontal pod autoscalers (HPAs), thereplicas
field sets the minimum number of replicas. The actual number of replicas depends on the CPU load and scaling done by the HPAs. - 4
- Set the requested and limited CPU and memory according to at least the idle consumption while also taking the previous findings and your own test results into consideration.
- 5
- Adjust the
PodDistruptionBudgets
to a value lower thanreplicas
to avoid problems during node maintenance. The defaultminAvailable
is set to1
, so if you increase the required replicas, you must also increaseminAvailable
.
As each environment is highly specific, it is essential to test and find your own ideal configuration. Use the monitoring and alerting functionality of OpenShift Container Platform to continuously monitor your actual resource consumption and make adjustments if needed.
If you are using the OpenShift Serverless and Service Mesh integration, additional CPU processing is added by the istio-proxy
sidecar containers. For more information about this, see the Service Mesh documentation.
Chapter 3. Autoscaling
3.1. Autoscaling
Knative Serving provides automatic scaling, or autoscaling, for applications to match incoming demand. For example, if an application is receiving no traffic, and scale-to-zero is enabled, Knative Serving scales the application down to zero replicas. If scale-to-zero is disabled, the application is scaled down to the minimum number of replicas configured for applications on the cluster. Replicas can also be scaled up to meet demand if traffic to the application increases.
Autoscaling settings for Knative services can be global settings that are configured by cluster administrators (or dedicated administrators for Red Hat OpenShift Service on AWS and OpenShift Dedicated), or per-revision settings that are configured for individual services.
You can modify per-revision settings for your services by using the OpenShift Container Platform web console, by modifying the YAML file for your service, or by using the Knative (kn
) CLI.
Any limits or targets that you set for a service are measured against a single instance of your application. For example, setting the target
annotation to 50
configures the autoscaler to scale the application so that each revision handles 50 requests at a time.
3.2. Scale bounds
Scale bounds determine the minimum and maximum numbers of replicas that can serve an application at any given time. You can set scale bounds for an application to help prevent cold starts or control computing costs.
3.2.1. Minimum scale bounds
The minimum number of replicas that can serve an application is determined by the min-scale
annotation. If scale to zero is not enabled, the min-scale
value defaults to 1
.
The min-scale
value defaults to 0
replicas if the following conditions are met:
-
The
min-scale
annotation is not set - Scaling to zero is enabled
-
The class
KPA
is used
Example service spec with min-scale
annotation
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: showcase namespace: default spec: template: metadata: annotations: autoscaling.knative.dev/min-scale: "0" ...
3.2.1.1. Setting the min-scale annotation by using the Knative CLI
Using the Knative (kn
) CLI to set the min-scale
annotation provides a more streamlined and intuitive user interface over modifying YAML files directly. You can use the kn service
command with the --scale-min
flag to create or modify the min-scale
value for a service.
Prerequisites
- Knative Serving is installed on the cluster.
-
You have installed the Knative (
kn
) CLI.
Procedure
Set the minimum number of replicas for the service by using the
--scale-min
flag:$ kn service create <service_name> --image <image_uri> --scale-min <integer>
Example command
$ kn service create showcase --image quay.io/openshift-knative/showcase --scale-min 2
3.2.2. Maximum scale bounds
The maximum number of replicas that can serve an application is determined by the max-scale
annotation. If the max-scale
annotation is not set, there is no upper limit for the number of replicas created.
Example service spec with max-scale
annotation
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: showcase namespace: default spec: template: metadata: annotations: autoscaling.knative.dev/max-scale: "10" ...
3.2.2.1. Setting the max-scale annotation by using the Knative CLI
Using the Knative (kn
) CLI to set the max-scale
annotation provides a more streamlined and intuitive user interface over modifying YAML files directly. You can use the kn service
command with the --scale-max
flag to create or modify the max-scale
value for a service.
Prerequisites
- Knative Serving is installed on the cluster.
-
You have installed the Knative (
kn
) CLI.
Procedure
Set the maximum number of replicas for the service by using the
--scale-max
flag:$ kn service create <service_name> --image <image_uri> --scale-max <integer>
Example command
$ kn service create showcase --image quay.io/openshift-knative/showcase --scale-max 10
3.3. Concurrency
Concurrency determines the number of simultaneous requests that can be processed by each replica of an application at any given time. Concurrency can be configured as a soft limit or a hard limit:
- A soft limit is a targeted requests limit, rather than a strictly enforced bound. For example, if there is a sudden burst of traffic, the soft limit target can be exceeded.
A hard limit is a strictly enforced upper bound requests limit. If concurrency reaches the hard limit, surplus requests are buffered and must wait until there is enough free capacity to execute the requests.
ImportantUsing a hard limit configuration is only recommended if there is a clear use case for it with your application. Having a low, hard limit specified may have a negative impact on the throughput and latency of an application, and might cause cold starts.
Adding a soft target and a hard limit means that the autoscaler targets the soft target number of concurrent requests, but imposes a hard limit of the hard limit value for the maximum number of requests.
If the hard limit value is less than the soft limit value, the soft limit value is tuned down, because there is no need to target more requests than the number that can actually be handled.
3.3.1. Configuring a soft concurrency target
A soft limit is a targeted requests limit, rather than a strictly enforced bound. For example, if there is a sudden burst of traffic, the soft limit target can be exceeded. You can specify a soft concurrency target for your Knative service by setting the autoscaling.knative.dev/target
annotation in the spec, or by using the kn service
command with the correct flags.
Procedure
Optional: Set the
autoscaling.knative.dev/target
annotation for your Knative service in the spec of theService
custom resource:Example service spec
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: showcase namespace: default spec: template: metadata: annotations: autoscaling.knative.dev/target: "200"
Optional: Use the
kn service
command to specify the--concurrency-target
flag:$ kn service create <service_name> --image <image_uri> --concurrency-target <integer>
Example command to create a service with a concurrency target of 50 requests
$ kn service create showcase --image quay.io/openshift-knative/showcase --concurrency-target 50
3.3.2. Configuring a hard concurrency limit
A hard concurrency limit is a strictly enforced upper bound requests limit. If concurrency reaches the hard limit, surplus requests are buffered and must wait until there is enough free capacity to execute the requests. You can specify a hard concurrency limit for your Knative service by modifying the containerConcurrency
spec, or by using the kn service
command with the correct flags.
Procedure
Optional: Set the
containerConcurrency
spec for your Knative service in the spec of theService
custom resource:Example service spec
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: showcase namespace: default spec: template: spec: containerConcurrency: 50
The default value is
0
, which means that there is no limit on the number of simultaneous requests that are permitted to flow into one replica of the service at a time.A value greater than
0
specifies the exact number of requests that are permitted to flow into one replica of the service at a time. This example would enable a hard concurrency limit of 50 requests.Optional: Use the
kn service
command to specify the--concurrency-limit
flag:$ kn service create <service_name> --image <image_uri> --concurrency-limit <integer>
Example command to create a service with a concurrency limit of 50 requests
$ kn service create showcase --image quay.io/openshift-knative/showcase --concurrency-limit 50
3.3.3. Concurrency target utilization
This value specifies the percentage of the concurrency limit that is actually targeted by the autoscaler. This is also known as specifying the hotness at which a replica runs, which enables the autoscaler to scale up before the defined hard limit is reached.
For example, if the containerConcurrency
value is set to 10, and the target-utilization-percentage
value is set to 70 percent, the autoscaler creates a new replica when the average number of concurrent requests across all existing replicas reaches 7. Requests numbered 7 to 10 are still sent to the existing replicas, but additional replicas are started in anticipation of being required after the containerConcurrency
value is reached.
Example service configured using the target-utilization-percentage annotation
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: showcase namespace: default spec: template: metadata: annotations: autoscaling.knative.dev/target-utilization-percentage: "70" ...
3.4. Scale-to-zero
Knative Serving provides automatic scaling, or autoscaling, for applications to match incoming demand.
3.4.1. Enabling scale-to-zero
You can use the enable-scale-to-zero
spec to enable or disable scale-to-zero globally for applications on the cluster.
Prerequisites
- You have installed OpenShift Serverless Operator and Knative Serving on your cluster.
- You have cluster administrator permissions on OpenShift Container Platform, or you have cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
- You are using the default Knative Pod Autoscaler. The scale to zero feature is not available if you are using the Kubernetes Horizontal Pod Autoscaler.
Procedure
Modify the
enable-scale-to-zero
spec in theKnativeServing
custom resource (CR):Example KnativeServing CR
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving spec: config: autoscaler: enable-scale-to-zero: "false" 1
- 1
- The
enable-scale-to-zero
spec can be either"true"
or"false"
. If set to true, scale-to-zero is enabled. If set to false, applications are scaled down to the configured minimum scale bound. The default value is"true"
.
3.4.2. Configuring the scale-to-zero grace period
Knative Serving provides automatic scaling down to zero pods for applications. You can use the scale-to-zero-grace-period
spec to define an upper bound time limit that Knative waits for scale-to-zero machinery to be in place before the last replica of an application is removed.
Prerequisites
- You have installed OpenShift Serverless Operator and Knative Serving on your cluster.
- You have cluster administrator permissions on OpenShift Container Platform, or you have cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
- You are using the default Knative Pod Autoscaler. The scale-to-zero feature is not available if you are using the Kubernetes Horizontal Pod Autoscaler.
Procedure
Modify the
scale-to-zero-grace-period
spec in theKnativeServing
custom resource (CR):Example KnativeServing CR
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving spec: config: autoscaler: scale-to-zero-grace-period: "30s" 1
- 1
- The grace period time in seconds. The default value is 30 seconds.
Chapter 4. Configuring OpenShift Serverless applications
4.1. Multi-container support for Serving
You can deploy a multi-container pod by using a single Knative service. This method is useful for separating application responsibilities into smaller, specialized parts.
4.1.1. Configuring a multi-container service
Multi-container support is enabled by default. You can create a multi-container pod by specifiying multiple containers in the service.
Procedure
Modify your service to include additional containers. Only one container can handle requests, so specify
ports
for exactly one container. Here is an example configuration with two containers:Multiple containers configuration
apiVersion: serving.knative.dev/v1 kind: Service ... spec: template: spec: containers: - name: first-container 1 image: gcr.io/knative-samples/helloworld-go ports: - containerPort: 8080 2 - name: second-container 3 image: gcr.io/knative-samples/helloworld-java
4.1.2. Probing a multi-container service
You can specify readiness and liveness probes for multiple containers. This feature is not enabled by default and you must configure it using the KnativeServing
custom resource (CR).
Procedure
Configure multi-container probing for your service by enabling the
multi-container-probing
feature in theKnativeServing
CR.Multi-container probing configuration
... spec: config: features: "multi-container-probing": enabled 1 ...
- 1
- Enabled multi-container-probing feature
Apply the updated
KnativeServing
CR.$ oc apply -f <filename>
Modify your multi-container service to include the specified probes.
Multi-container probing
apiVersion: serving.knative.dev/v1 kind: Service ... spec: template: spec: containers: - name: first-container image: ghcr.io/knative/helloworld-go:latest ports: - containerPort: 8080 readinessProbe: 1 httpGet: port: 8080 - name: second-container image: gcr.io/knative-samples/helloworld-java readinessProbe: 2 httpGet: port: 8090
4.1.2.1. Additional resources
4.2. EmptyDir volumes
emptyDir
volumes are empty volumes that are created when a pod is created, and are used to provide temporary working disk space. emptyDir
volumes are deleted when the pod they were created for is deleted.
4.2.1. Configuring the EmptyDir extension
The kubernetes.podspec-volumes-emptydir
extension controls whether emptyDir
volumes can be used with Knative Serving. To enable using emptyDir
volumes, you must modify the KnativeServing
custom resource (CR) to include the following YAML:
Example KnativeServing CR
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving spec: config: features: kubernetes.podspec-volumes-emptydir: enabled ...
4.3. Persistent Volume Claims for Serving
Some serverless applications require permanent data storage. By configuring different volume types, you can provide data storage for Knative services. Serving supports mounting of the volume types such as secret
, configMap
, projected
, and emptyDir
.
You can configure persistent volume claims (PVCs) for your Knative services. The Persistent volume types are implemented as plugins. To determine if there are any persistent volume types available, you can check the available or installed storage classes in your cluster. Persistent volumes are supported, but require a feature flag to be enabled.
The mounting of large volumes can lead to a considerable delay in the start time of the application.
4.3.1. Enabling PVC support
Procedure
To enable Knative Serving to use PVCs and write to them, modify the
KnativeServing
custom resource (CR) to include the following YAML:Enabling PVCs with write access
... spec: config: features: "kubernetes.podspec-persistent-volume-claim": enabled "kubernetes.podspec-persistent-volume-write": enabled ...
-
The
kubernetes.podspec-persistent-volume-claim
extension controls whether persistent volumes (PVs) can be used with Knative Serving. -
The
kubernetes.podspec-persistent-volume-write
extension controls whether PVs are available to Knative Serving with the write access.
-
The
To claim a PV, modify your service to include the PV configuration. For example, you might have a persistent volume claim with the following configuration:
NoteUse the storage class that supports the access mode you are requesting. For example, you can use the
ocs-storagecluster-cephfs
storage class for theReadWriteMany
access mode.The
ocs-storagecluster-cephfs
storage class is supported and comes from Red Hat OpenShift Data Foundation.PersistentVolumeClaim configuration
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: example-pv-claim namespace: my-ns spec: accessModes: - ReadWriteMany storageClassName: ocs-storagecluster-cephfs resources: requests: storage: 1Gi
In this case, to claim a PV with write access, modify your service as follows:
Knative service PVC configuration
apiVersion: serving.knative.dev/v1 kind: Service metadata: namespace: my-ns ... spec: template: spec: containers: ... volumeMounts: 1 - mountPath: /data name: mydata readOnly: false volumes: - name: mydata persistentVolumeClaim: 2 claimName: example-pv-claim readOnly: false 3
NoteTo successfully use persistent storage in Knative services, you need additional configuration, such as the user permissions for the Knative container user.
4.3.2. Additional resources for OpenShift Container Platform
4.4. Init containers
Init containers are specialized containers that are run before application containers in a pod. They are generally used to implement initialization logic for an application, which may include running setup scripts or downloading required configurations. You can enable the use of init containers for Knative services by modifying the KnativeServing
custom resource (CR).
Init containers may cause longer application start-up times and should be used with caution for serverless applications, which are expected to scale up and down frequently.
4.4.1. Enabling init containers
Prerequisites
- You have installed OpenShift Serverless Operator and Knative Serving on your cluster.
- You have cluster administrator permissions on OpenShift Container Platform, or you have cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
Procedure
Enable the use of init containers by adding the
kubernetes.podspec-init-containers
flag to theKnativeServing
CR:Example KnativeServing CR
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving spec: config: features: kubernetes.podspec-init-containers: enabled ...
4.5. Startup probes
Startup probes verify whether a service has started successfully, helping to reduce cold start times for containers with slow startup processes. Startup probes run only during the container’s initialization phase and do not execute periodically. If a startup probe fails, the container adheres to the defined restartPolicy
.
4.5.1. Progress deadline
By default, services have a progress deadline that defines the time limit for a service to complete its initial startup. When using startup probes, ensure that the progress deadline is set to exceed the maximum time required by the startup probes. If the progress deadline is set too low, the startup probes might not finish before the deadline is reached, which can prevent the service from starting.
Consider increasing the progress deadline if you encounter any of these conditions in your deployment:
- The service image takes a long time to pull due to its size.
-
The service takes a long time to become
READY
because of initial cache priming. - The cluster relies on autoscaling to allocate resources for new pods.
4.5.2. Configuring startup probing
For OpenShift Serverless Serving, startup probes are not defined by default. You can define startup probes for your containers in your deployment configuration.
Procedure
Define startup probes for your service by modifying your deployment configuration. The following example shows a configuration with two containers:
Example of defined starup probes
apiVersion: serving.knative.dev/v1 kind: Service # ... spec: template: spec: containers: - name: first-container image: <image> ports: - containerPort: 8080 # ... startupProbe: 1 httpGet: port: 8080 path: "/" - name: second-container image: <image> # ... startupProbe: 2 httpGet: port: 8081 path: "/"
4.5.3. Configuring the progress deadline
You can configure progress deadline settings to specify the maximum time allowed for your deployment to progress before the system reports a failure for the Knative Revision. This time limit can be specified in seconds or minutes.
To configure the progress deadline effectively, consider the following parameters:
-
initialDelaySeconds
-
failureThreshold
-
periodSeconds
-
timeoutSeconds
If the initial scale is not achieved within the specified time limit, the Knative Autoscaler component scales the revision to 0
, and the Knative service enters a terminal Failed
state.
By default, the progress deadline is set to 600 seconds. This value is specified as a Golang time.Duration
string and must be rounded to the nearest second.
Procedure
To configure the progress deadline setting, use an annotation in your deployment configuration.
Example of progress deadline set to 60 seconds
apiVersion: serving.knative.dev/v1 kind: Service ... spec: template: metadata: annotations: serving.knative.dev/progress-deadline: "60s" spec: containers: - image: ghcr.io/knative/helloworld-go:latest
4.6. Resolving image tags to digests
If the Knative Serving controller has access to the container registry, Knative Serving resolves image tags to a digest when you create a revision of a service. This is known as tag-to-digest resolution, and helps to provide consistency for deployments.
4.6.1. Tag-to-digest resolution
To give the controller access to the container registry on OpenShift Container Platform, you must create a secret and then configure controller custom certificates. You can configure controller custom certificates by modifying the controller-custom-certs
spec in the KnativeServing
custom resource (CR). The secret must reside in the same namespace as the KnativeServing
CR.
If a secret is not included in the KnativeServing
CR, this setting defaults to using public key infrastructure (PKI). When using PKI, the cluster-wide certificates are automatically injected into the Knative Serving controller by using the config-service-sa
config map. The OpenShift Serverless Operator populates the config-service-sa
config map with cluster-wide certificates and mounts the config map as a volume to the controller.
4.6.1.1. Configuring tag-to-digest resolution by using a secret
If the controller-custom-certs
spec uses the Secret
type, the secret is mounted as a secret volume. Knative components consume the secret directly, assuming that the secret has the required certificates.
Prerequisites
- You have cluster administrator permissions on OpenShift Container Platform, or you have cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
- You have installed the OpenShift Serverless Operator and Knative Serving on your cluster.
Procedure
Create a secret:
Example command
$ oc -n knative-serving create secret generic custom-secret --from-file=<secret_name>.crt=<path_to_certificate>
Configure the
controller-custom-certs
spec in theKnativeServing
custom resource (CR) to use theSecret
type:Example KnativeServing CR
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: controller-custom-certs: name: custom-secret type: Secret
4.7. Configuring deployment resources
In Knative Serving, the config-deployment
config map contains settings that determine how Kubernetes Deployment
resources are configured for Knative services. In OpenShift Serverless Serving, you can configure these settings in the deployment
section of your KnativeServing
custom resource (CR).
You can use the deployment
section to configure the following:
- Tag resolution
- Runtime environments
- Progress deadlines
4.7.1. Skipping tag resolution
Skipping tag resolution in OpenShift Serverless Serving can speed up deployments by avoiding unnecessary queries to the container registry, reducing latency and dependency on registry availability.
You can configure Serving to skip tag resolution by modifying the registriesSkippingTagResolving
setting in your KnativeServing
custom resource (CR).
Procedure
In your
KnativeServing
CR, modify theregistriesSkippingTagResolving
setting with the list of registries for which tag resoution will be skipped:Example of configured tag resolution skipping
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving spec: config: deployment: registriesSkippingTagResolving: "registry.example.com, another.registry.com"
4.7.2. Configuring selectable RuntimeClassName
You can configure OpenShift Serverless Serving to set a specific RuntimeClassName
resource for Deployments by updating the runtime-class-name
setting in your KnativeServing
custom resource (CR).
This setting interacts with service labels, applying either the default RuntimeClassName
or the one that matches the most labels associated with the service.
Procedure
In your
KnativeServing
CR, configure theruntime-class-name
setting:Example of configured
runtime-class-name
settingapiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving spec: config: deployment: runtime-class-name: | kata: {} gvisor: selector: my-label: selector
4.7.3. Progress deadline
By default, services have a progress deadline that defines the time limit for a service to complete its initial startup.
Consider increasing the progress deadline if you encounter any of these conditions in your deployment:
- The service image takes a long time to pull due to its size.
-
The service takes a long time to become
READY
because of initial cache priming. - The cluster relies on autoscaling to allocate resources for new pods.
If the initial scale is not achieved within the specified time limit, the Knative Autoscaler component scales the revision to 0
, and the service enters a terminal Failed
state.
4.7.3.1. Configuring the progress deadline
Configure progress deadline settings to set the maximum time allowed in seconds or minutes for deployment progress before the system reports a Knative Revision failure.
By default, the progress deadline is set to 600 seconds. This value is specified as a Go time.Duration
string and must be rounded to the nearest second.
Procedure
Configure progress deadline by modifying your KnativeServing
custom resource (CR).
In your
KnativeServing
CR, set the value ofprogressDeadline
:Example of progress deadline set to 60 seconds
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving spec: config: deployment: progressDeadline: "60s"
4.8. Configuring Kourier
Kourier is a lightweight Kubernetes-native Ingress for Knative Serving. Kourier acts as a gateway for Knative, routing HTTP traffic to Knative services.
4.8.1. Accessing the current Envoy bootstrap configuration
The Envoy proxy component in Kourier handles inbound and outbound HTTP traffic for the Knative services. By default, Kourier contains an Envoy bootstrap configuration in the kourier-bootstrap
configuration map in the knative-serving-ingress
namespace.
Procedure
To get the current Envoy bootstrap configuration, run the following command:
Example command
$ oc get cm kourier-bootstrap -n knative-serving-ingress -o yaml
For example, with the default configuration, the example command produces the output that contains the following excerpts:
Example output
Name: kourier-bootstrap Namespace: knative-serving-ingress Labels: app.kubernetes.io/component=net-kourier app.kubernetes.io/name=knative-serving app.kubernetes.io/version=release-v1.10 networking.knative.dev/ingress-provider=kourier serving.knative.openshift.io/ownerName=knative-serving serving.knative.openshift.io/ownerNamespace=knative-serving Annotations: manifestival: new
Example
Data
outputdynamic_resources: ads_config: transport_api_version: V3 api_type: GRPC rate_limit_settings: {} grpc_services: - envoy_grpc: {cluster_name: xds_cluster} cds_config: resource_api_version: V3 ads: {} lds_config: resource_api_version: V3 ads: {} node: cluster: kourier-knative id: 3scale-kourier-gateway static_resources: listeners: - name: stats_listener address: socket_address: address: 0.0.0.0 port_value: 9000 filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager stat_prefix: stats_server http_filters: - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router route_config: virtual_hosts: - name: admin_interface domains: - "*" routes: - match: safe_regex: regex: '/(certs|stats(/prometheus)?|server_info|clusters|listeners|ready)?' headers: - name: ':method' string_match: exact: GET route: cluster: service_stats clusters: - name: service_stats connect_timeout: 0.250s type: static load_assignment: cluster_name: service_stats endpoints: lb_endpoints: endpoint: address: pipe: path: /tmp/envoy.admin - name: xds_cluster # This keepalive is recommended by envoy docs. # https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol typed_extension_protocol_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions: "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions explicit_http_config: http2_protocol_options: connection_keepalive: interval: 30s timeout: 5s connect_timeout: 1s load_assignment: cluster_name: xds_cluster endpoints: lb_endpoints: endpoint: address: socket_address: address: "net-kourier-controller.knative-serving-ingress.svc.cluster.local." port_value: 18000 type: STRICT_DNS admin: access_log: - name: envoy.access_loggers.stdout typed_config: "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog address: pipe: path: /tmp/envoy.admin layered_runtime: layers: - name: static-layer static_layer: envoy.reloadable_features.override_request_timeout_by_gateway_timeout: false
Example
BinaryData
outputEvents: <none>
4.8.2. Customizing kourier-bootstrap for Kourier getaways
The Envoy proxy component in Kourier handles inbound and outbound HTTP traffic for the Knative services. By default, Kourier contains an Envoy bootstrap configuration in the kourier-bootstrap
configuration map in the knative-serving-ingress
namespace. You can change this configuration map to a custom one.
Prerequisites
- You have installed the OpenShift Serverless Operator and Knative Serving.
- You have cluster administrator permissions on OpenShift Container Platform, or you have cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
Procedure
Specify a custom bootstrapping configuration map by changing the
spec.ingress.kourier.bootstrap-configmap
field in theKnativeServing
custom resource (CR):Example KnativeServing CR
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: config: network: ingress-class: kourier.ingress.networking.knative.dev ingress: kourier: bootstrap-configmap: my-configmap enabled: true # ...
4.8.3. Enabling administrator interface access
You can change the envoy bootstrap configuration to enable access to the administrator interface.
This procedure assumes sufficient knowledge of Knative, as changing envoy bootstrap configuration might result in Knative failure. Red Hat does not support custom configurations that are not tested or shipped with the product.
Prerequisites
- You have installed the OpenShift Serverless Operator and Knative Serving.
- You have cluster administrator permissions on OpenShift Container Platform, or you have cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
Procedure
To enable administrator interface access, locate this configuration in your bootstrapping configuration map:
pipe: path: /tmp/envoy.admin
Substitute it with the following configuration:
socket_address: 1 address: 127.0.0.1 port_value: 9901
- 1
- This configuration enables access to the Envoy admin interface on the loopback address (127.0.0.1) and port 9901.
Apply the
socket_address
configuration in theservice_stats
cluster configuration and in theadmin
configuration:The first is in the
service_stats
cluster configuration:clusters: - name: service_stats connect_timeout: 0.250s type: static load_assignment: cluster_name: service_stats endpoints: lb_endpoints: endpoint: address: socket_address: address: 127.0.0.1 port_value: 9901
The second is in the
admin
configuration:admin: access_log: - name: envoy.access_loggers.stdout typed_config: "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog address: socket_address: address: 127.0.0.1 port_value: 9901
4.9. Restrictive network policies
4.9.1. Clusters with restrictive network policies
If you are using a cluster that multiple users have access to, your cluster might use network policies to control which pods, services, and namespaces can communicate with each other over the network. If your cluster uses restrictive network policies, it is possible that Knative system pods are not able to access your Knative application. For example, if your namespace has the following network policy, which denies all requests, Knative system pods cannot access your Knative application:
Example NetworkPolicy object that denies all requests to the namespace
kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: deny-by-default namespace: example-namespace spec: podSelector: ingress: []
4.9.2. Enabling communication with Knative applications on a cluster with restrictive network policies
To allow access to your applications from Knative system pods, you must add a label to each of the Knative system namespaces, and then create a NetworkPolicy
object in your application namespace that allows access to the namespace for other namespaces that have this label.
A network policy that denies requests to non-Knative services on your cluster still prevents access to these services. However, by allowing access from Knative system namespaces to your Knative application, you are allowing access to your Knative application from all namespaces in the cluster.
If you do not want to allow access to your Knative application from all namespaces on the cluster, you might want to use JSON Web Token authentication for Knative services instead. JSON Web Token authentication for Knative services requires Service Mesh.
Prerequisites
-
Install the OpenShift CLI (
oc
). - OpenShift Serverless Operator and Knative Serving are installed on your cluster.
Procedure
Add the
knative.openshift.io/system-namespace=true
label to each Knative system namespace that requires access to your application:Label the
knative-serving
namespace:$ oc label namespace knative-serving knative.openshift.io/system-namespace=true
Label the
knative-serving-ingress
namespace:$ oc label namespace knative-serving-ingress knative.openshift.io/system-namespace=true
Label the
knative-eventing
namespace:$ oc label namespace knative-eventing knative.openshift.io/system-namespace=true
Label the
knative-kafka
namespace:$ oc label namespace knative-kafka knative.openshift.io/system-namespace=true
Create a
NetworkPolicy
object in your application namespace to allow access from namespaces with theknative.openshift.io/system-namespace
label:Example
NetworkPolicy
objectapiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: <network_policy_name> 1 namespace: <namespace> 2 spec: ingress: - from: - namespaceSelector: matchLabels: knative.openshift.io/system-namespace: "true" podSelector: {} policyTypes: - Ingress
4.10. Configuring revision timeouts
You can configure timeout durations for revisions globally or individually to control the time spent on requests.
4.10.1. Configuring revision timeout
You can configure the default number of seconds for the revision timeout based on the request.
Prerequisites
- You have installed the OpenShift Serverless Operator and Knative Serving.
- You have cluster administrator permissions on OpenShift Container Platform, or cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
Procedure
Choose the appropriate method to configure the revision timeout:
To configure the revision timeout globally, set the
revision-timeout-seconds
field in theKnativeServing
custom resource (CR):apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: config: defaults: revision-timeout-seconds: "300"
To configure the timeout per revision by setting the
timeoutSeconds
field in your service definition:apiVersion: serving.knative.dev/v1 kind: Service metadata: namespace: my-ns spec: template: spec: timeoutSeconds: 300 containers: - image: ghcr.io/knative/helloworld-go:latest
4.10.2. Configuring maximum revision timeout
By seting the maximum revision timeout, you can ensure that no revision can exceed a specific limit.
Prerequisites
- You have installed the OpenShift Serverless Operator and Knative Serving.
- You have cluster administrator permissions on OpenShift Container Platform, or cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
Procedure
To configure the maximum revision timeout, set the
max-revision-timeout-seconds
field in theKnativeServing
custom resource (CR):If this value is increased, the activator `terminationGracePeriodSeconds` should also be increased to prevent in-flight requests being disrupted.
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: config: defaults: max-revision-timeout-seconds: "600"
Chapter 5. Debugging Serverless applications
You can use a variety of methods to troubleshoot a Serverless application.
5.1. Checking terminal output
You can check your deploy command output to see whether deployment succeeded or not. If your deployment process was terminated, you should see an error message in the output that describes the reason why the deployment failed. This kind of failure is most likely due to either a misconfigured manifest or an invalid command.
Procedure
Open the command output on the client where you deploy and manage your application. The following example is an error that you might see after a failed
oc apply
command:Error from server (InternalError): error when applying patch: {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"serving.knative.dev/v1\",\"kind\":\"Route\",\"metadata\":{\"annotations\":{},\"name\":\"route-example\",\"namespace\":\"default\"},\"spec\":{\"traffic\":[{\"configurationName\":\"configuration-example\",\"percent\":50}]}}\n"}},"spec":{"traffic":[{"configurationName":"configuration-example","percent":50}]}} to: &{0xc421d98240 0xc421e77490 default route-example STDIN 0xc421db0488 264682 false} for: "STDIN": Internal error occurred: admission webhook "webhook.knative.dev" denied the request: mutation failed: The route must have traffic percent sum equal to 100. ERROR: Non-zero return code '1' from command: Process exited with status 1
This output indicates that you must configure the route traffic percent to be equal to 100.
5.2. Checking pod status
You might need to check the status of your Pod
object to identify the issue with your Serverless application.
Procedure
List all pods for your deployment by running the following command:
$ oc get pods
Example output
NAME READY STATUS RESTARTS AGE configuration-example-00001-deployment-659747ff99-9bvr4 2/2 Running 0 3h configuration-example-00002-deployment-5f475b7849-gxcht 1/2 CrashLoopBackOff 2 36s
In the output, you can see all pods with selected data about their status.
View the detailed information on the status of a pod by running the following command:
Example output
$ oc get pod <pod_name> --output yaml
In the output, the
conditions
andcontainerStatuses
fields might be particularly useful for debugging.
5.3. Checking revision status
You might need to check the status of your revision to identify the issue with your Serverless application.
Procedure
If you configure your route with a
Configuration
object, get the name of theRevision
object created for your deployment by running the following command:$ oc get configuration <configuration_name> --output jsonpath="{.status.latestCreatedRevisionName}"
You can find the configuration name in the
Route.yaml
file, which specifies routing settings by defining an OpenShiftRoute
resource.If you configure your route with revision directly, look up the revision name in the
Route.yaml
file.Query for the status of the revision by running the following command:
$ oc get revision <revision-name> --output yaml
A ready revision should have the
reason: ServiceReady
,status: "True"
, andtype: Ready
conditions in its status. If these conditions are present, you might want to check pod status or Istio routing. Otherwise, the resource status contains the error message.
5.3.1. Additional resources
5.4. Checking Ingress status
You might need to check the status of your Ingress to identify the issue with your Serverless application.
Procedure
Check the IP address of your Ingress by running the following command:
$ oc get svc -n istio-system istio-ingressgateway
The
istio-ingressgateway
service is theLoadBalancer
service used by Knative.If there is no external IP address, run the following command:
$ oc describe svc istio-ingressgateway -n istio-system
This command prints the reason why IP addresses were not provisioned. Most likely, it is due to a quota issue.
5.5. Checking route status
In some cases, the Route
object has issues. You can check its status by using the OpenShift CLI (oc
).
Procedure
View the status of the
Route
object with which you deployed your application by running the following command:$ oc get route <route_name> --output yaml
Substitute
<route_name>
with the name of yourRoute
object.The
conditions
object in thestatus
object states the reason in case of a failure.
5.6. Checking Ingress and Istio routing
Sometimes, when Istio is used as an Ingress layer, the Ingress and Istio routing have issues. You can see the details on them by using the OpenShift CLI (oc
).
Procedure
List all Ingress resources and their corresponding labels by running the following command:
$ oc get ingresses.networking.internal.knative.dev -o=custom-columns='NAME:.metadata.name,LABELS:.metadata.labels'
Example output
NAME LABELS helloworld-go map[serving.knative.dev/route:helloworld-go serving.knative.dev/routeNamespace:default serving.knative.dev/service:helloworld-go]
In this output, labels
serving.knative.dev/route
andserving.knative.dev/routeNamespace
indicate theRoute
where the Ingress resource resides. YourRoute
and Ingress should be listed.If your Ingress does not exist, the route controller assumes that the
Revision
objects targeted by yourRoute
orService
object are not ready. Proceed with other debugging procedures to diagnoseRevision
readiness status.If your Ingress is listed, examine the
ClusterIngress
object created for your route by running the following command:$ oc get ingresses.networking.internal.knative.dev <ingress_name> --output yaml
In the status section of the output, if the condition with
type=Ready
has the status ofTrue
, then Ingress is working correctly. Otherwise, the output contains error messages.If Ingress has the status of
Ready
, then there is a correspondingVirtualService
object. Verify the configuration of theVirtualService
object by running the following command:$ oc get virtualservice -l networking.internal.knative.dev/ingress=<ingress_name> -n <ingress_namespace> --output yaml
The network configuration in the
VirtualService
object must match that of theIngress
andRoute
objects. Because theVirtualService
object does not expose aStatus
field, you might need to wait for its settings to propagate.
5.6.1. Additional resources
Chapter 6. Kourier and Istio ingresses
OpenShift Serverless supports the following two ingress solutions:
- Kourier
- Istio using Red Hat OpenShift Service Mesh
The default is Kourier.
6.1. Kourier and Istio ingress solutions
6.1.1. Kourier
Kourier is the default ingress solution for OpenShift Serverless. It has the following properties:
- It is based on envoy proxy.
- It is simple and lightweight.
- It provides the basic routing functionality that Serverless needs to provide its set of features.
- It supports basic observability and metrics.
- It supports basic TLS termination of Knative Service routing.
- It provides only limited configuration and extension options.
6.1.2. Istio using OpenShift Service Mesh
Using Istio as the ingress solution for OpenShift Serverless enables an additional feature set that is based on what Red Hat OpenShift Service Mesh offers:
- Native mTLS between all connections
- Serverless components are part of a service mesh
- Additional observability and metrics
- Authorization and authentication support
- Custom rules and configuration, as supported by Red Hat OpenShift Service Mesh
However, the additional features come with a higher overhead and resource consumption. For details, see the Red Hat OpenShift Service Mesh documentation.
See the "Integrating Service Mesh with OpenShift Serverless" section of Serverless documentation for Istio requirements and installation instructions.
6.1.3. Traffic configuration and routing
Regardless of whether you use Kourier or Istio, the traffic for a Knative Service is configured in the knative-serving
namespace by the net-kourier-controller
or the net-istio-controller
respectively.
The controller reads the KnativeService
and its child custom resources to configure the ingress solution. Both ingress solutions provide an ingress gateway pod that becomes part of the traffic path. Both ingress solutions are based on Envoy. By default, Serverless has two routes for each KnativeService
object:
-
A cluster-external route that is forwarded by the OpenShift router, for example
myapp-namespace.example.com
. -
A cluster-local route containing the cluster domain, for example
myapp.namespace.svc.cluster.local
. This domain can and should be used to call Knative services from Knative or other user workloads.
The ingress gateway can forward requests either in the serve mode or the proxy mode:
- In the serve mode, requests go directly to the Queue-Proxy sidecar container of the Knative service.
-
In the proxy mode, requests first go through the Activator component in the
knative-serving
namespace.
The choice of mode depends on the configuration of Knative, the Knative service, and the current traffic. For example, if a Knative Service is scaled to zero, requests are sent to the Activator component, which acts as a buffer until a new Knative service pod is started.
Chapter 7. Serving transport encryption
You can enable OpenShift Serverless Serving transport encryption to allow transporting data over secured and encrypted HTTPS connections using TLS.
OpenShift Serverless Serving transport encryption is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Serving Transport Encryption is only available for Kourier as an ingress layer. For Red Hat OpenShift Service Mesh, use the service mesh mTLS capabilities to ensure encrypted traffic.
7.1. Overview of Serving transport encryption
There are three parts to OpenShift Serverless Serving transport encryption:
- External domain encryption
-
Transport encryption on the ingress layer external to the cluster. For example, a cluster-external domain, such as
myapp-<namespace>.example.com
. - Cluster-local encryption
-
Transport encryption on the ingress layer internal to the cluster. For example, cluster-local domains, such as
myapp.<namespace>.svc.cluster.local
. - System-internal encryption
-
Transport encryption between the
ingress-gateway
,activator
, andqueue-proxy
Knative internal components.
Control-plane traffic, including Kubernetes PreStopHooks, metadata, and metrics, contains no user data and is not encrypted.
The different parts are independent of each other and can be enabled and disabled individually. They can use the same or different Certificate Authorities (CAs) to sign the necessary certificates.
For diagrams illustrating transport encryption, see OpenShift Serverless Serving Transport Encryption.
7.1.1. External domain encryption
The transport encryption for external domains is handled by the ingress layer of the cluster, which is either the OpenShift Container Platform ingress or Red Hat OpenShift Service Mesh.
7.1.2. Cluster-local encryption
Cluster-local encryption enables transport encryption for cluster-local domains. It has the following properties:
-
The Certificate Common Name (CN) or Subject Alternative Name (SAN) contains the cluster-local domains of a Knative Service, for example
myapp.namespace.svc.cluster.local
,myapp.namespace.svc
,myapp.namespace
. -
The cluster-local endpoint of the
ingress-controller
component uses SNI to select the certificates. -
To create the certificates, Knative relies on
cert-manager
, which needs to be installed and configured for the feature to work. For more information, see cert-manager Operator for Red Hat OpenShift.
The caller must trust the CA that signed the cluster-local certificates. This is out of the scope of OpenShift Serverless.
7.1.3. System-internal encryption
System-internal encryption enables transport encryption for the ingress-gateway
, activator
, and queue-proxy
Knative internal components. These components host TLS endpoints when this configuration is used.
The following prerequisites must be satisfied to use this feature:
-
For OpenShift Serverless to get the certificates,
cert-manager
must be installed and configured. - Specific SANs are used to verify each connection. Each component must trust the CA that signed the certificates. To satisfy this requirement, OpenShift Serverless system components consume and trust a bundle of CAs. The CA bundle must be provided by the cluster administrator.
7.2. Choice of a certificate issuer
Issuers refer to cert-manager
issuers and cluster issuers. They represent certificate authorities (CAs) that can generate signed certificates by honoring certificate signing requests. For more information, see cert-manager documentation on issuers.
Depending on the encryption features that you use, OpenShift Serverless requires your certificate issuer to be able to sign certain certificates. To identify your certificate issuer, refer to the list of cert-manager integrations, which contains examples for the following:
- A custom CA stored in a Kubernetes secret
- HTTP-01 challenges
- DNS-01 challenges
- Self-signed issuers
7.2.1. Compatible certificate issuers
Not all issuer types work for each Knative Serving encryption feature.
For cluster-local encryption, the issuer must be able to sign certificates for the following cluster-local domain types:
-
myapp.<namespace>
-
myapp.<namespace>.svc
-
myapp.<namespace>.svc.cluster.local
As the CA usually is not within the cluster, verification using the Automated Certificate Management Environment (ACME) protocol (DNS01/HTTP01) is not possible. You can use an issuer that allows creating these certificates, such as the
cert-manager
CA issuer.-
For system-internal encryption, the issuer must be able to sign certificates with the following Subject Alternative Names (SANs):
-
kn-routing
-
names of format
kn-user-<namespace>
, where<namespace>
is a namespace where Knative Services are created -
data-plane.knative.dev
Knative requires these SANs to verify connections between the internal components. Because this is not possible using the ACME protocol (DNS01/HTTP01), you must configure an issuer that allows creating these certificates, for example,
cert-manager
CA issuer.-
7.3. Setting up OpenShift Serverless transport encryption
Prerequisites
- You have access to an OpenShift Container Platform account with cluster administrator access.
- Install the {oc-first}.
- Install the cert-manager Operator for Red Hat OpenShift.
- Install the OpenShift Serverless Operator.
If you install the OpenShift Serverless Operator before installing the cert-manager Operator for Red Hat OpenShift, you must restart the controller and activator deployments in the knative-serving
namespace. Failure to restart these deployments prevents Knative from creating the necessary cert-manager
resources, which results in pending Knative Services and prevents enabling the Knative Serving cert-manager
integration.
7.3.1. Configuring a SelfSigned cluster issuer
The following procedure uses a SelfSigned
issuer as the root certificate. For information about the implications and limitations of this method, see the SelfSigned cert-manager documentation.
If you manage your own company-specific Private Key Infrastructure (PKI), use the CA issuer. For more information, see cert-manager documentation on CA issuers.
Procedure
Create a
SelfSigned
ClusterIssuer
custom resource (CR):Example ClusterIssuer CR
apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: knative-serving-selfsigned-issuer spec: selfSigned: {}
Apply the
ClusterIssuer
CR by running the following command:$ oc apply -f <filename>
Create a root certificate that refers to the
ClusterIssuer
CR:Example root certificate
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: knative-serving-selfsigned-ca namespace: cert-manager 1 spec: secretName: knative-serving-ca 2 isCA: true commonName: selfsigned-ca privateKey: algorithm: ECDSA size: 256 issuerRef: name: knative-serving-selfsigned-issuer kind: ClusterIssuer group: cert-manager.io
Apply the
Certificate
CR by running the following command:$ oc apply -f <filename>
7.3.2. Creating a ClusterIssuer to be used by Serving
To enable the use of certificates by Serving, you must create a cluster issuer.
Procedure
Create the
knative-serving-ca-issuer
ClusterIssuer
for Serving:apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: knative-serving-ca-issuer spec: ca: secretName: knative-serving-ca 1
- 1
- Secret name in the cert-manager Operator for Red Hat OpenShift namespace (
cert-manager
by default) containing the certificate that can be used by OpenShift Serverless Serving components for new certificates.
Apply the
ClusterIssuer
resource by running the following command:$ oc apply -f <filename>
7.3.3. Configuring transport encryption
Configuring transport encryption consists of two parts:
Specifying the
ClusterIssuer
issuer to use:-
clusterLocalIssuerRef
: issuer for cluster-local-domain certificates used for ingress. -
systemInternalIssuerRef
: issuer for certificates for system-internal-tls certificates used by Knative internal components.
-
Specifying transport encryption features to use:
-
cluster-local-domain-tls
: Enables the transport encryption feature for cluster-local domains -
system-internal-tls
: Enables the transport encryption feature for OpenShift Serverless Serving internal components.
-
Procedure
Enable transport encryption in the
KnativeServing
resource:apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: ... config: certmanager: clusterLocalIssuerRef: | kind: ClusterIssuer name: knative-serving-ca-issuer 1 systemInternalIssuerRef: | kind: ClusterIssuer name: knative-serving-ca-issuer 2 network: cluster-local-domain-tls: Enabled 3 system-internal-tls: Enabled 4
Apply the
KnativeServing
resource by running the following command:$ oc apply -f <filename>
Optionally, change the
defaultCertificate
value in the Ingress Controller:apiVersion: operator.openshift.io/v1 kind: IngressController ... spec: defaultCertificate: name: ca-ingress-cert
If you changed the
defaultCertificate
value, you must specify the custom certificate name in theopenshift-ingress-default-certificate
field in theKnativeServing
custom resource.For example, if the custom certificate name is
ca-ingress-cert
, add the following configuration:... spec: config: network: system-internal-tls: Enabled openshift-ingress-default-certificate: "ca-ingress-cert" ...
If you enabled
cluster-local-domain-tls
orsystem-internal-tls
, restart the Controller component by running the following command.ImportantWhen either the
cluster-local-domain-tls
or thesystem-internal-tls
feature is enabled, you must restart the Controller component to enable the Knative Servingcert-manager
integration.$ oc rollout restart deploy/controller -n knative-serving
If you enabled
system-internal-tls
, restart the Activator component by running the following command.ImportantWhen the
system-internal-tls
feature is activated, you must restart the Activator component to reconfigure its internal web server, as this is not possible during runtime.$ oc rollout restart deploy/activator -n knative-serving
7.4. Trust configuration
When you enable any of the transport encryption features, you must make sure that all clients calling trust the Certificate Authority (CA) issuing the certificates used for the transport encryption.
There are multiple places where trust must be ensured:
- Cluster external client, such as Browser or other application. This is out of the scope of OpenShift Serverless.
- OpenShift Serverless system components, such as Activator, Queue-Proxy, and Ingress-Controller.
- Cluster internal client, such as a Knative Service or other workload.
7.4.1. Trust configuration for OpenShift Serverless Serving components and Knative Services
To ensure that OpenShift Serverless Serving components and Knative Services trust the CA that issues certificates, you can create a ConfigMap in the following namespaces with the label networking.knative.dev/trust-bundle: true
:
knative-serving
- for the system components of OpenShift Serverless Serving.
knative-serving-ingress
- for the ingress layer of OpenShift Serverless Serving.
istio-system
or your own Service Mesh namespace- when the Service Mesh integration is enabled.
Knative reads all data keys in ConfigMaps with this label, regardless of the name. One key can contain one or multiple CAs or intermediate certificates. If they are valid, they are added to the trust store of the Knative components.
This is an example ConfigMap:
apiVersion: v1 data: cacerts.pem: | 1 -----BEGIN CERTIFICATE----- MIIDDTCCAfWgAwIBAgIQMQuip05h7NLQq2TB+j9ZmTANBgkqhkiG9w0BAQsFADAW MRQwEgYDVQQDEwtrbmF0aXZlLmRldjAeFw0yMzExMjIwOTAwNDhaFw0yNDAyMjAw OTAwNDhaMBYxFDASBgNVBAMTC2tuYXRpdmUuZGV2MIIBIjANBgkqhkiG9w0BAQEF AAOCAQ8AMIIBCgKCAQEA3clC3CV7sy0TpUKNuTku6QmP9z8JUCbLCPCLACCUc1zG FEokqOva6TakgvAntXLkB3TEsbdCJlNm6qFbbko6DBfX6rEggqZs40x3/T+KH66u 4PvMT3fzEtaMJDK/KQOBIvVHrKmPkvccUYK/qWY7rgBjVjjLVSJrCn4dKaEZ2JNr Fd0KNnaaW/dP9/FvviLqVJvHnTMHH5qyRRr1kUGTrc8njRKwpHcnUdauiDoWRKxo Zlyy+MhQfdbbyapX984WsDjCvrDXzkdGgbRNAf+erl6yUm6pHpQhyFFo/zndx6Uq QXA7jYvM2M3qCnXmaFowidoLDsDyhwoxD7WT8zur/QIDAQABo1cwVTAOBgNVHQ8B Af8EBAMCAgQwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUwAwEB/zAd BgNVHQ4EFgQU7p4VuECNOcnrP9ulOjc4J37Q2VUwDQYJKoZIhvcNAQELBQADggEB AAv26Vnk+ptQrppouF7yHV8fZbfnehpm07HIZkmnXO2vAP+MZJDNrHjy8JAVzXjt +OlzqAL0cRQLsUptB0btoJuw23eq8RXgJo05OLOPQ2iGNbAATQh2kLwBWd/CMg+V KJ4EIEpF4dmwOohsNR6xa/JoArIYH0D7gh2CwjrdGZr/tq1eMSL+uZcuX5OiE44A 2oXF9/jsqerOcH7QUMejSnB8N7X0LmUvH4jAesQgr7jo1JTOBs7GF6wb+U76NzFa 8ms2iAWhoplQ+EHR52wffWb0k6trXspq4O6v/J+nq9Ky3vC36so+G1ZFkMhCdTVJ ZmrBsSMWeT2l07qeei2UFRU= -----END CERTIFICATE----- kind: ConfigMap metadata: labels: networking.knative.dev/trust-bundle: "true" name: knative-bundle 2 namespace: knative-serving
When a CA bundle ConfigMap is created or updated, the Serving components automatically pick them up and add the CAs or intermediate certificates to their CA trust store. The trust store is refreshed for every new HTTP connection.
7.4.2. Trust configuration on your custom workload
As OpenShift Serverless Serving does not control all workloads and managing trust is highly dependent on your runtime and language, custom workloads are out of the scope of OpenShift Serverless. The following are other options for custom workloads:
- Adding a CA bundle to a Container image on build-time. Note that this complicates CA rotation, as you must rebuild and redeploy every application when the CA rotates.
- Mounting a CA bundle to the filesystem, such as from a Secret or ConfigMap, and making sure your application uses it to verify TLS connections.
- Reading a CA bundle from an environment variable and making sure that your application uses it to verify TLS connections.
- Accessing a CA bundle from a secret or ConfigMap using Kubernetes API and making sure your application uses it to verify TLS connections.
7.5. Ensuring seamless CA rotation
Ensuring seamless CA rotation is essential to avoid service downtime, or to deal with an emergency.
Procedure
- Create a new CA certificate.
- Add the public key of the new CA certificate to the CA trust bundles as described in the "Trust configuration for OpenShift Serverless Operator Serving components and Knative Services" section. Retain the public key of the existing CA.
- Ensure that all clients have consumed the latest set of CA trust bundles. OpenShift Serverless Serving components will automatically reload the changed CA trust bundles.
- If you have custom workload consuming trust bundles, reload or restart them accordingly.
-
Update the
knative-serving-ca-issuer
cluster issuer to reference the secret containing the new CA certificate. -
Either wait for
cert-manager
to renew all your certificates or enforce it to renew all the certificates. For more information, see thecert-manager
documentation. - Once the CA rotation is fully completed, you can remove the public key of the old CA from the trust bundle configmap. Allow enough time for all components to apply the changes.
Additional resources
7.6. Verifying transport encryption is enabled
Procedure
Create a Knative Service:
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: test-webapp namespace: test-namespace spec: template: spec: containers: - image: docker.io/openshift/hello-openshift env: - name: RESPONSE value: "Hello Serverless!"
Apply the Knative Service YAML by running the following command:
$ oc apply -f <filename>
Examine status of the Knative Service:
Example command
$ oc get ksvc -n test-namespace -o yaml
Example output
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: test-webapp namespace: test-namespace # spec: # ... status: address: # cluster-local-domain: url: https://helloworld.test.svc.cluster.local 1
- 1
- If you have enabled
cluster-local-domain-tls
, you will see the HTTPS URL.
To verify that
system-internal-tls
is enabled, check the output of Queue-Proxy logs by running the following command:Example command
$ oc logs your-pod -n test-namespace -c queue-proxy | grep -E 'certDir|Certificate|tls'
If you see lines similar to the following,
system-internal-tls
is enabled:{"severity":"INFO","timestamp":"2024-01-03T07:07:32.892810888Z","logger":"queueproxy","caller":"certificate/watcher.go:62","message":"Starting to watch the following directories for changes{certDir 15 0 /var/lib/knative/certs <nil>} {keyDir 15 0 /var/lib/knative/certs <nil>}","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"} {"severity":"INFO","timestamp":"2024-01-03T07:07:32.89397512Z","logger":"queueproxy","caller":"certificate/watcher.go:131","message":"Certificate and/or key have changed on disk and were reloaded.","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"} {"severity":"INFO","timestamp":"2024-01-03T07:07:32.894232939Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server admin:8022","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"} {"severity":"INFO","timestamp":"2024-01-03T07:07:32.894268548Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server main:8112","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}
Chapter 8. Traffic splitting
8.1. Traffic splitting overview
In a Knative application, traffic can be managed by creating a traffic split. A traffic split is configured as part of a route, which is managed by a Knative service.

Configuring a route allows requests to be sent to different revisions of a service. This routing is determined by the traffic
spec of the Service
object.
A traffic
spec declaration consists of one or more revisions, each responsible for handling a portion of the overall traffic. The percentages of traffic routed to each revision must add up to 100%, which is ensured by a Knative validation.
The revisions specified in a traffic
spec can either be a fixed, named revision, or can point to the “latest” revision, which tracks the head of the list of all revisions for the service. The "latest" revision is a type of floating reference that updates if a new revision is created. Each revision can have a tag attached that creates an additional access URL for that revision.
The traffic
spec can be modified by:
-
Editing the YAML of a
Service
object directly. -
Using the Knative (
kn
) CLI--traffic
flag. - Using the OpenShift Container Platform web console.
When you create a Knative service, it does not have any default traffic
spec settings.
8.2. Traffic spec examples
The following example shows a traffic
spec where 100% of traffic is routed to the latest revision of the service. Under status
, you can see the name of the latest revision that latestRevision
resolves to:
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: example-service namespace: default spec: ... traffic: - latestRevision: true percent: 100 status: ... traffic: - percent: 100 revisionName: example-service
The following example shows a traffic
spec where 100% of traffic is routed to the revision tagged as current
, and the name of that revision is specified as example-service
. The revision tagged as latest
is kept available, even though no traffic is routed to it:
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: example-service namespace: default spec: ... traffic: - tag: current revisionName: example-service percent: 100 - tag: latest latestRevision: true percent: 0
The following example shows how the list of revisions in the traffic
spec can be extended so that traffic is split between multiple revisions. This example sends 50% of traffic to the revision tagged as current
, and 50% of traffic to the revision tagged as candidate
. The revision tagged as latest
is kept available, even though no traffic is routed to it:
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: example-service namespace: default spec: ... traffic: - tag: current revisionName: example-service-1 percent: 50 - tag: candidate revisionName: example-service-2 percent: 50 - tag: latest latestRevision: true percent: 0
8.3. Traffic splitting using the Knative CLI
Using the Knative (kn
) CLI to create traffic splits provides a more streamlined and intuitive user interface over modifying YAML files directly. You can use the kn service update
command to split traffic between revisions of a service.
8.3.1. Creating a traffic split by using the Knative CLI
Prerequisites
- The OpenShift Serverless Operator and Knative Serving are installed on your cluster.
-
You have installed the Knative (
kn
) CLI. - You have created a Knative service.
Procedure
Specify the revision of your service and what percentage of traffic you want to route to it by using the
--traffic
tag with a standardkn service update
command:Example command
$ kn service update <service_name> --traffic <revision>=<percentage>
Where:
-
<service_name>
is the name of the Knative service that you are configuring traffic routing for. -
<revision>
is the revision that you want to configure to receive a percentage of traffic. You can either specify the name of the revision, or a tag that you assigned to the revision by using the--tag
flag. -
<percentage>
is the percentage of traffic that you want to send to the specified revision.
-
Optional: The
--traffic
flag can be specified multiple times in one command. For example, if you have a revision tagged as@latest
and a revision namedstable
, you can specify the percentage of traffic that you want to split to each revision as follows:Example command
$ kn service update showcase --traffic @latest=20,stable=80
If you have multiple revisions and do not specify the percentage of traffic that should be split to the last revision, the
--traffic
flag can calculate this automatically. For example, if you have a third revision namedexample
, and you use the following command:Example command
$ kn service update showcase --traffic @latest=10,stable=60
The remaining 30% of traffic is split to the
example
revision, even though it was not specified.
8.4. CLI flags for traffic splitting
The Knative (kn
) CLI supports traffic operations on the traffic block of a service as part of the kn service update
command.
8.4.1. Knative CLI traffic splitting flags
The following table displays a summary of traffic splitting flags, value formats, and the operation the flag performs. The Repetition column denotes whether repeating the particular value of flag is allowed in a kn service update
command.
Flag | Value(s) | Operation | Repetition |
---|---|---|---|
|
|
Gives | Yes |
|
|
Gives | Yes |
|
|
Gives | No |
|
|
Gives | Yes |
|
|
Gives | No |
|
|
Removes | Yes |
8.4.1.1. Multiple flags and order precedence
All traffic-related flags can be specified using a single kn service update
command. kn
defines the precedence of these flags. The order of the flags specified when using the command is not taken into account.
The precedence of the flags as they are evaluated by kn
are:
-
--untag
: All the referenced revisions with this flag are removed from the traffic block. -
--tag
: Revisions are tagged as specified in the traffic block. -
--traffic
: The referenced revisions are assigned a portion of the traffic split.
You can add tags to revisions and then split traffic according to the tags you have set.
8.4.1.2. Custom URLs for revisions
Assigning a --tag
flag to a service by using the kn service update
command creates a custom URL for the revision that is created when you update the service. The custom URL follows the pattern https://<tag>-<service_name>-<namespace>.<domain>
or http://<tag>-<service_name>-<namespace>.<domain>
.
The --tag
and --untag
flags use the following syntax:
- Require one value.
- Denote a unique tag in the traffic block of the service.
- Can be specified multiple times in one command.
8.4.1.2.1. Example: Assign a tag to a revision
The following example assigns the tag latest
to a revision named example-revision
:
$ kn service update <service_name> --tag @latest=example-tag
8.4.1.2.2. Example: Remove a tag from a revision
You can remove a tag to remove the custom URL, by using the --untag
flag.
If a revision has its tags removed, and it is assigned 0% of the traffic, the revision is removed from the traffic block entirely.
The following command removes all tags from the revision named example-revision
:
$ kn service update <service_name> --untag example-tag
8.5. Splitting traffic between revisions
After you create a serverless application, the application is displayed in the Topology view of the Developer perspective in the OpenShift Container Platform web console. The application revision is represented by the node, and the Knative service is indicated by a quadrilateral around the node.
Any new change in the code or the service configuration creates a new revision, which is a snapshot of the code at a given time. For a service, you can manage the traffic between the revisions of the service by splitting and routing it to the different revisions as required.
8.5.1. Managing traffic between revisions by using the OpenShift Container Platform web console
Prerequisites
- The OpenShift Serverless Operator and Knative Serving are installed on your cluster.
- You have logged in to the OpenShift Container Platform web console.
Procedure
To split traffic between multiple revisions of an application in the Topology view:
- Click the Knative service to see its overview in the side panel.
Click the Resources tab, to see a list of Revisions and Routes for the service.
Figure 8.1. Serverless application
- Click the service, indicated by the S icon at the top of the side panel, to see an overview of the service details.
-
Click the YAML tab and modify the service configuration in the YAML editor, and click Save. For example, change the
timeoutseconds
from 300 to 301 . This change in the configuration triggers a new revision. In the Topology view, the latest revision is displayed and the Resources tab for the service now displays the two revisions. In the Resources tab, click to see the traffic distribution dialog box:
- Add the split traffic percentage portion for the two revisions in the Splits field.
- Add tags to create custom URLs for the two revisions.
Click Save to see two nodes representing the two revisions in the Topology view.
Figure 8.2. Serverless application revisions
8.6. Rerouting traffic using blue-green strategy
You can safely reroute traffic from a production version of an app to a new version, by using a blue-green deployment strategy.
8.6.1. Routing and managing traffic by using a blue-green deployment strategy
Prerequisites
- The OpenShift Serverless Operator and Knative Serving are installed on the cluster.
-
Install the OpenShift CLI (
oc
).
Procedure
- Create and deploy an app as a Knative service.
Find the name of the first revision that was created when you deployed the service, by viewing the output from the following command:
$ oc get ksvc <service_name> -o=jsonpath='{.status.latestCreatedRevisionName}'
Example command
$ oc get ksvc showcase -o=jsonpath='{.status.latestCreatedRevisionName}'
Example output
$ showcase-00001
Add the following YAML to the service
spec
to send inbound traffic to the revision:... spec: traffic: - revisionName: <first_revision_name> percent: 100 # All traffic goes to this revision ...
Verify that you can view your app at the URL output you get from running the following command:
$ oc get ksvc <service_name>
-
Deploy a second revision of your app by modifying at least one field in the
template
spec of the service and redeploying it. For example, you can modify theimage
of the service, or anenv
environment variable. You can redeploy the service by applying the service YAML file, or by using thekn service update
command if you have installed the Knative (kn
) CLI. Find the name of the second, latest revision that was created when you redeployed the service, by running the command:
$ oc get ksvc <service_name> -o=jsonpath='{.status.latestCreatedRevisionName}'
At this point, both the first and second revisions of the service are deployed and running.
Update your existing service to create a new, test endpoint for the second revision, while still sending all other traffic to the first revision:
Example of updated service spec with test endpoint
... spec: traffic: - revisionName: <first_revision_name> percent: 100 # All traffic is still being routed to the first revision - revisionName: <second_revision_name> percent: 0 # No traffic is routed to the second revision tag: v2 # A named route ...
After you redeploy this service by reapplying the YAML resource, the second revision of the app is now staged. No traffic is routed to the second revision at the main URL, and Knative creates a new service named
v2
for testing the newly deployed revision.Get the URL of the new service for the second revision, by running the following command:
$ oc get ksvc <service_name> --output jsonpath="{.status.traffic[*].url}"
You can use this URL to validate that the new version of the app is behaving as expected before you route any traffic to it.
Update your existing service again, so that 50% of traffic is sent to the first revision, and 50% is sent to the second revision:
Example of updated service spec splitting traffic 50/50 between revisions
... spec: traffic: - revisionName: <first_revision_name> percent: 50 - revisionName: <second_revision_name> percent: 50 tag: v2 ...
When you are ready to route all traffic to the new version of the app, update the service again to send 100% of traffic to the second revision:
Example of updated service spec sending all traffic to the second revision
... spec: traffic: - revisionName: <first_revision_name> percent: 0 - revisionName: <second_revision_name> percent: 100 tag: v2 ...
TipYou can remove the first revision instead of setting it to 0% of traffic if you do not plan to roll back the revision. Non-routeable revision objects are then garbage-collected.
- Visit the URL of the first revision to verify that no more traffic is being sent to the old version of the app.
Chapter 9. External and Ingress routing
9.1. Routing overview
Knative leverages OpenShift Container Platform TLS termination to provide routing for Knative services. When a Knative service is created, an OpenShift Container Platform route is automatically created for the service. This route is managed by the OpenShift Serverless Operator. The OpenShift Container Platform route exposes the Knative service through the same domain as the OpenShift Container Platform cluster.
You can disable Operator control of OpenShift Container Platform routing so that you can configure a Knative route to directly use your TLS certificates instead.
Knative routes can also be used alongside the OpenShift Container Platform route to provide additional fine-grained routing capabilities, such as traffic splitting.
9.1.1. Additional resources for OpenShift Container Platform
9.2. Customizing labels and annotations
OpenShift Container Platform routes support the use of custom labels and annotations, which you can configure by modifying the metadata
spec of a Knative service. Custom labels and annotations are propagated from the service to the Knative route, then to the Knative ingress, and finally to the OpenShift Container Platform route.
9.2.1. Customizing labels and annotations for OpenShift Container Platform routes
Prerequisites
- You must have the OpenShift Serverless Operator and Knative Serving installed on your OpenShift Container Platform cluster.
-
Install the OpenShift CLI (
oc
).
Procedure
Create a Knative service that contains the label or annotation that you want to propagate to the OpenShift Container Platform route:
To create a service by using YAML:
Example service created by using YAML
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: <service_name> labels: <label_name>: <label_value> annotations: <annotation_name>: <annotation_value> ...
To create a service by using the Knative (
kn
) CLI, enter:Example service created by using a
kn
command$ kn service create <service_name> \ --image=<image> \ --annotation <annotation_name>=<annotation_value> \ --label <label_value>=<label_value>
Verify that the OpenShift Container Platform route has been created with the annotation or label that you added by inspecting the output from the following command:
Example command for verification
$ oc get routes.route.openshift.io \ -l serving.knative.openshift.io/ingressName=<service_name> \ 1 -l serving.knative.openshift.io/ingressNamespace=<service_namespace> \ 2 -n knative-serving-ingress -o yaml \ | grep -e "<label_name>: \"<label_value>\"" -e "<annotation_name>: <annotation_value>" 3
9.3. Configuring routes for Knative services
If you want to configure a Knative service to use your TLS certificate on OpenShift Container Platform, you must disable the automatic creation of a route for the service by the OpenShift Serverless Operator and instead manually create a route for the service.
When you complete the following procedure, the default OpenShift Container Platform route in the knative-serving-ingress
namespace is not created. However, the Knative route for the application is still created in this namespace.
9.3.1. Configuring OpenShift Container Platform routes for Knative services
Prerequisites
- The OpenShift Serverless Operator and Knative Serving component must be installed on your OpenShift Container Platform cluster.
-
Install the OpenShift CLI (
oc
).
Procedure
Create a Knative service that includes the
serving.knative.openshift.io/disableRoute=true
annotation:ImportantThe
serving.knative.openshift.io/disableRoute=true
annotation instructs OpenShift Serverless to not automatically create a route for you. However, the service still shows a URL and reaches a status ofReady
. This URL does not work externally until you create your own route with the same hostname as the hostname in the URL.Create a Knative
Service
resource:Example resource
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: <service_name> annotations: serving.knative.openshift.io/disableRoute: "true" spec: template: spec: containers: - image: <image> ...
Apply the
Service
resource:$ oc apply -f <filename>
Optional. Create a Knative service by using the
kn service create
command:Example
kn
command$ kn service create <service_name> \ --image=gcr.io/knative-samples/helloworld-go \ --annotation serving.knative.openshift.io/disableRoute=true
Verify that no OpenShift Container Platform route has been created for the service:
Example command
$ $ oc get routes.route.openshift.io \ -l serving.knative.openshift.io/ingressName=$KSERVICE_NAME \ -l serving.knative.openshift.io/ingressNamespace=$KSERVICE_NAMESPACE \ -n knative-serving-ingress
You will see the following output:
No resources found in knative-serving-ingress namespace.
Create a
Route
resource in theknative-serving-ingress
namespace:apiVersion: route.openshift.io/v1 kind: Route metadata: annotations: haproxy.router.openshift.io/timeout: 600s 1 name: <route_name> 2 namespace: knative-serving-ingress 3 spec: host: <service_host> 4 port: targetPort: http2 to: kind: Service name: kourier weight: 100 tls: insecureEdgeTerminationPolicy: Allow termination: edge 5 key: |- -----BEGIN PRIVATE KEY----- [...] -----END PRIVATE KEY----- certificate: |- -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- caCertificate: |- -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE---- wildcardPolicy: None
- 1
- The timeout value for the OpenShift Container Platform route. You must set the same value as the
max-revision-timeout-seconds
setting (600s
by default). - 2
- The name of the OpenShift Container Platform route.
- 3
- The namespace for the OpenShift Container Platform route. This must be
knative-serving-ingress
. - 4
- The hostname for external access. You can set this to
<service_name>-<service_namespace>.<domain>
. - 5
- The certificates you want to use. Currently, only
edge
termination is supported.
Apply the
Route
resource:$ oc apply -f <filename>
9.4. URL scheme for external routes
The URL scheme of external routes defaults to HTTPS for enhanced security. This scheme is determined by the default-external-scheme
key in the KnativeServing
custom resource (CR) spec.
9.4.1. Setting the URL scheme for external routes
Default spec
... spec: config: network: default-external-scheme: "https" ...
You can override the default spec to use HTTP by modifying the default-external-scheme
key:
HTTP override spec
... spec: config: network: default-external-scheme: "http" ...
9.5. Cluster local availability
By default, Knative services are published to a public IP address. Being published to a public IP address means that Knative services are public applications, and have a publicly accessible URL.
Publicly accessible URLs are accessible from outside of the cluster. However, developers may need to build back-end services that are only be accessible from inside the cluster, known as private services. Developers can label individual services in the cluster with the networking.knative.dev/visibility=cluster-local
label to make them private.
For OpenShift Serverless 1.15.0 and newer versions, the serving.knative.dev/visibility
label is no longer available. You must update existing services to use the networking.knative.dev/visibility
label instead.
9.5.1. Setting cluster availability to cluster local
Prerequisites
- The OpenShift Serverless Operator and Knative Serving are installed on the cluster.
- You have created a Knative service.
Procedure
Set the visibility for your service by adding the
networking.knative.dev/visibility=cluster-local
label:$ oc label ksvc <service_name> networking.knative.dev/visibility=cluster-local
Verification
Check that the URL for your service is now in the format
http://<service_name>.<namespace>.svc.cluster.local
, by entering the following command and reviewing the output:$ oc get ksvc
Example output
NAME URL LATESTCREATED LATESTREADY READY REASON hello http://hello.default.svc.cluster.local hello-tx2g7 hello-tx2g7 True
9.5.2. Enabling TLS authentication for cluster local services
For cluster local services, the Kourier local gateway kourier-internal
is used. If you want to use TLS traffic against the Kourier local gateway, you must configure your own server certificates in the local gateway.
Prerequisites
- You have installed the OpenShift Serverless Operator and Knative Serving.
- You have administrator permissions.
-
You have installed the OpenShift (
oc
) CLI.
Procedure
Deploy server certificates in the
knative-serving-ingress
namespace:$ export san="knative"
NoteSubject Alternative Name (SAN) validation is required so that these certificates can serve the request to
<app_name>.<namespace>.svc.cluster.local
.Generate a root key and certificate:
$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \ -subj '/O=Example/CN=Example' \ -keyout ca.key \ -out ca.crt
Generate a server key that uses SAN validation:
$ openssl req -out tls.csr -newkey rsa:2048 -nodes -keyout tls.key \ -subj "/CN=Example/O=Example" \ -addext "subjectAltName = DNS:$san"
Create server certificates:
$ openssl x509 -req -extfile <(printf "subjectAltName=DNS:$san") \ -days 365 -in tls.csr \ -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt
Configure a secret for the Kourier local gateway:
Deploy a secret in
knative-serving-ingress
namespace from the certificates created by the previous steps:$ oc create -n knative-serving-ingress secret tls server-certs \ --key=tls.key \ --cert=tls.crt --dry-run=client -o yaml | oc apply -f -
Update the
KnativeServing
custom resource (CR) spec to use the secret that was created by the Kourier gateway:Example KnativeServing CR
... spec: config: kourier: cluster-cert-secret: server-certs ...
The Kourier controller sets the certificate without restarting the service, so that you do not need to restart the pod.
You can access the Kourier internal service with TLS through port 443
by mounting and using the ca.crt
from the client.
9.6. Kourier Gateway service type
The Kourier Gateway is exposed by default as the ClusterIP
service type. This service type is determined by the service-type
ingress spec in the KnativeServing
custom resource (CR).
Default spec
... spec: ingress: kourier: service-type: ClusterIP ...
9.6.1. Setting the Kourier Gateway service type
You can override the default service type to use a load balancer service type instead by modifying the service-type
spec:
LoadBalancer override spec
... spec: ingress: kourier: service-type: LoadBalancer ...
9.7. Using HTTP2 and gRPC
OpenShift Serverless supports only insecure or edge-terminated routes. Insecure or edge-terminated routes do not support HTTP2 on OpenShift Container Platform. These routes also do not support gRPC because gRPC is transported by HTTP2. If you use these protocols in your application, you must call the application using the ingress gateway directly. To do this you must find the ingress gateway’s public address and the application’s specific host.
9.7.1. Interacting with a serverless application using HTTP2 and gRPC
This method applies to OpenShift Container Platform 4.10 and later. For older versions, see the following section.
Prerequisites
- Install OpenShift Serverless Operator and Knative Serving on your cluster.
-
Install the OpenShift CLI (
oc
). - Create a Knative service.
- Upgrade OpenShift Container Platform 4.10 or later.
- Enable HTTP/2 on OpenShift Ingress controller.
Procedure
Add the
serverless.openshift.io/default-enable-http2=true
annotation to theKnativeServing
Custom Resource:$ oc annotate knativeserving <your_knative_CR> -n knative-serving serverless.openshift.io/default-enable-http2=true
After the annotation is added, you can verify that the
appProtocol
value of the Kourier service ish2c
:$ oc get svc -n knative-serving-ingress kourier -o jsonpath="{.spec.ports[0].appProtocol}"
Example output
h2c
Now you can use the gRPC framework over the HTTP/2 protocol for external traffic, for example:
import "google.golang.org/grpc" grpc.Dial( YOUR_URL, 1 grpc.WithTransportCredentials(insecure.NewCredentials())), 2 )
Additional resources
9.8. Using Serving with OpenShift ingress sharding
You can use Knative Serving with OpenShift ingress sharding to split ingress traffic based on domains. This allows you to manage and route network traffic to different parts of a cluster more efficiently.
Even with OpenShift ingress sharding in place, OpenShift Serverless traffic is still routed through a single Knative Ingress Gateway and the activator component in the knative-serving
project.
For more information about isolating the network traffic, see Using Service Mesh to isolate network traffic with OpenShift Serverless.
Prerequisites
- You have installed the OpenShift Serverless Operator and Knative Serving.
- You have cluster administrator permissions on OpenShift Container Platform, or you have cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
9.8.1. Configuring OpenShift ingress shards
Before configuring Knative Serving, you must configure OpenShift ingress shards.
Procedure
Use a label selector in the
IngressController
CR to configure OpenShift Serverless to match specific ingress shards with different domains:Example
IngressController
CRapiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: ingress-dev 1 namespace: openshift-ingress-operator spec: routeSelector: matchLabels: router: dev 2 domain: "dev.serverless.cluster.example.com" 3 # ... --- apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: ingress-prod 4 namespace: openshift-ingress-operator spec: routeSelector: matchLabels: router: prod 5 domain: "prod.serverless.cluster.example.com" 6 # ...
9.8.2. Configuring custom domains in the KnativeServing CR
After configuring OpenShift ingress shards, you must configure Knative Serving to match them.
Procedure
In the
KnativeServing
CR, configure Serving to use the same domains and labels as your ingress shards by adding thespec.config.domain
field:Example
KnativeServing
CRspec: config: domain: 1 dev.serverless.cluster.example.com: | selector: router: dev prod.serverless.cluster.example.com: | selector: router: prod # ...
- 1
- These values need to match the values in the ingress shard configuration.
9.8.3. Targeting a specific ingress shard in the Knative Service
After configuring ingress sharding and Knative Serving, you can target a specific ingress shard in your Knative Service resources using a label.
Procedure
In your
Service
CR, add the label selector that matches a specific shard:Example Service CR
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: hello-dev labels: router: dev 1 spec: template: spec: containers: - image: docker.io/openshift/hello-openshift --- apiVersion: serving.knative.dev/v1 kind: Service metadata: name: hello-prod labels: router: prod 2 spec: template: spec: containers: - image: docker.io/openshift/hello-openshift # ...
9.8.4. Verifying Serving with OpenShift ingress sharding configuration
After configuring ingress sharding, Knative Serving, and your service, you can verify that your service uses the correct route and the selected ingress shard.
Procedure
Print information about the services in the cluster by running the following command:
$ oc get ksvc
Example output
NAME URL LATESTCREATED LATESTREADY READY REASON hello-dev https://hello-dev-default.dev.serverless.cluster.example.com hello-dev-00001 hello-dev-00001 True hello-prod https://hello-prod-default.prod.serverless.cluster.example.com hello-prod-00001 hello-prod-00001 True
Verify that your service uses the correct route and the selected ingress shard by running the following command:
$ oc get route -n knative-serving-ingress -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.host}{" "}{@.status.ingress[*].routerName}{"\n"}{end}'
Example output
route-19e6628b-77af-4da0-9b4c-1224934b2250-323461616533 hello-prod-default.prod.serverless.cluster.example.com ingress-prod route-cb5085d9-b7da-4741-9a56-96c88c6adaaa-373065343266 hello-dev-default.dev.serverless.cluster.example.com ingress-dev
Chapter 10. HTTP configuration
10.1. Global HTTPS redirection
HTTPS redirection provides redirection for incoming HTTP requests. These redirected HTTP requests are encrypted. You can enable HTTPS redirection for all services on the cluster by configuring the httpProtocol
spec for the KnativeServing
custom resource (CR).
10.1.1. HTTPS redirection global settings
Example KnativeServing
CR that enables HTTPS redirection
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving spec: config: network: httpProtocol: "redirected" ...
10.2. HTTPS redirection per service
You can enable or disable HTTPS redirection for a service by configuring the networking.knative.dev/http-option
annotation.
10.2.1. Redirecting HTTPS for a service
The following example shows how you can use this annotation in a Knative Service
YAML object:
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: example namespace: default annotations: networking.knative.dev/http-protocol: "redirected" spec: ...
10.3. Full duplex support for HTTP/1
You can enable the HTTP/1 full duplex support for a service by configuring the features.knative.dev/http-full-duplex
annotation.
Verify your HTTP clients before enabling, as earlier version clients might not provide support for HTTP/1 full duplex.
The following example shows how you can use this annotation in a Knative Service
YAML object at the revision spec level:
Example KnativeServing
CR that provides full duplex support for HTTP/1
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: example-service namespace: default spec: template: metadata: annotations: features.knative.dev/http-full-duplex: "Enabled" ...
Chapter 11. Configuring access to Knative services
11.1. Configuring JSON Web Token authentication for Knative services
OpenShift Serverless does not currently have user-defined authorization features. To add user-defined authorization to your deployment, you must integrate OpenShift Serverless with Red Hat OpenShift Service Mesh, and then configure JSON Web Token (JWT) authentication and sidecar injection for Knative services.
11.2. Using JSON Web Token authentication with Service Mesh 2.x
You can use JSON Web Token (JWT) authentication with Knative services by using Service Mesh 2.x and OpenShift Serverless. To do this, you must create authentication requests and policies in the application namespace that is a member of the ServiceMeshMemberRoll
object. You must also enable sidecar injection for the service.
11.2.1. Configuring JSON Web Token authentication for Service Mesh 2.x and OpenShift Serverless
Adding sidecar injection to pods in system namespaces, such as knative-serving
and knative-serving-ingress
, is not supported when Kourier is enabled.
For OpenShift Container Platform, if you require sidecar injection for pods in these namespaces, see the OpenShift Serverless documentation on Integrating Service Mesh with OpenShift Serverless natively.
Prerequisites
- You have installed the OpenShift Serverless Operator, Knative Serving, and Red Hat OpenShift Service Mesh on your cluster.
-
Install the OpenShift CLI (
oc
). - You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
Procedure
Add the
sidecar.istio.io/inject="true"
annotation to your service:Example service
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: <service_name> spec: template: metadata: annotations: sidecar.istio.io/inject: "true" 1 sidecar.istio.io/rewriteAppHTTPProbers: "true" 2 ...
Apply the
Service
resource:$ oc apply -f <filename>
Create a
RequestAuthentication
resource in each serverless application namespace that is a member in theServiceMeshMemberRoll
object:apiVersion: security.istio.io/v1beta1 kind: RequestAuthentication metadata: name: jwt-example namespace: <namespace> spec: jwtRules: - issuer: testing@secure.istio.io jwksUri: https://raw.githubusercontent.com/istio/istio/release-1.8/security/tools/jwt/samples/jwks.json
Apply the
RequestAuthentication
resource:$ oc apply -f <filename>
Allow access to the
RequestAuthenticaton
resource from system pods for each serverless application namespace that is a member in theServiceMeshMemberRoll
object, by creating the followingAuthorizationPolicy
resource:apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: allowlist-by-paths namespace: <namespace> spec: action: ALLOW rules: - to: - operation: paths: - /metrics 1 - /healthz 2
Apply the
AuthorizationPolicy
resource:$ oc apply -f <filename>
For each serverless application namespace that is a member in the
ServiceMeshMemberRoll
object, create the followingAuthorizationPolicy
resource:apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: require-jwt namespace: <namespace> spec: action: ALLOW rules: - from: - source: requestPrincipals: ["testing@secure.istio.io/testing@secure.istio.io"]
Apply the
AuthorizationPolicy
resource:$ oc apply -f <filename>
Verification
If you try to use a
curl
request to get the Knative service URL, it is denied:Example command
$ curl http://hello-example-1-default.apps.mycluster.example.com/
Example output
RBAC: access denied
Verify the request with a valid JWT.
Get the valid JWT token:
$ TOKEN=$(curl https://raw.githubusercontent.com/istio/istio/release-1.8/security/tools/jwt/samples/demo.jwt -s) && echo "$TOKEN" | cut -d '.' -f2 - | base64 --decode -
Access the service by using the valid token in the
curl
request header:$ curl -H "Authorization: Bearer $TOKEN" http://hello-example-1-default.apps.example.com
The request is now allowed:
Example output
Hello OpenShift!
11.3. Using JSON Web Token authentication with Service Mesh 1.x
You can use JSON Web Token (JWT) authentication with Knative services by using Service Mesh 1.x and OpenShift Serverless. To do this, you must create a policy in the application namespace that is a member of the ServiceMeshMemberRoll
object. You must also enable sidecar injection for the service.
11.3.1. Configuring JSON Web Token authentication for Service Mesh 1.x and OpenShift Serverless
Adding sidecar injection to pods in system namespaces, such as knative-serving
and knative-serving-ingress
, is not supported when Kourier is enabled.
For OpenShift Container Platform, if you require sidecar injection for pods in these namespaces, see the OpenShift Serverless documentation on Integrating Service Mesh with OpenShift Serverless natively.
Prerequisites
- You have installed the OpenShift Serverless Operator, Knative Serving, and Red Hat OpenShift Service Mesh on your cluster.
-
Install the OpenShift CLI (
oc
). - You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
Procedure
Add the
sidecar.istio.io/inject="true"
annotation to your service:Example service
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: <service_name> spec: template: metadata: annotations: sidecar.istio.io/inject: "true" 1 sidecar.istio.io/rewriteAppHTTPProbers: "true" 2 ...
Apply the
Service
resource:$ oc apply -f <filename>
Create a policy in a serverless application namespace which is a member in the
ServiceMeshMemberRoll
object, that only allows requests with valid JSON Web Tokens (JWT):ImportantThe paths
/metrics
and/healthz
must be included inexcludedPaths
because they are accessed from system pods in theknative-serving
namespace.apiVersion: authentication.istio.io/v1alpha1 kind: Policy metadata: name: default namespace: <namespace> spec: origins: - jwt: issuer: testing@secure.istio.io jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.6/security/tools/jwt/samples/jwks.json" triggerRules: - excludedPaths: - prefix: /metrics 1 - prefix: /healthz 2 principalBinding: USE_ORIGIN
Apply the
Policy
resource:$ oc apply -f <filename>
Verification
If you try to use a
curl
request to get the Knative service URL, it is denied:$ curl http://hello-example-default.apps.mycluster.example.com/
Example output
Origin authentication failed.
Verify the request with a valid JWT.
Get the valid JWT token:
$ TOKEN=$(curl https://raw.githubusercontent.com/istio/istio/release-1.6/security/tools/jwt/samples/demo.jwt -s) && echo "$TOKEN" | cut -d '.' -f2 - | base64 --decode -
Access the service by using the valid token in the
curl
request header:$ curl http://hello-example-default.apps.mycluster.example.com/ -H "Authorization: Bearer $TOKEN"
The request is now allowed:
Example output
Hello OpenShift!
Chapter 12. Configuring kube-rbac-proxy for Serving
The kube-rbac-proxy
component provides internal authentication and authorization capabilities for Knative Serving.
12.1. Configuring kube-rbac-proxy resources for Serving
You can globally override resource allocation for the kube-rbac-proxy
container by using the OpenShift Serverless Operator CR.
You can also override resource allocation for a specific deployment.
The following configuration sets Knative Serving kube-rbac-proxy
minimum and maximum CPU and memory allocation:
KnativeServing CR example
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: config: deployment: "kube-rbac-proxy-cpu-request": "10m" 1 "kube-rbac-proxy-memory-request": "20Mi" 2 "kube-rbac-proxy-cpu-limit": "100m" 3 "kube-rbac-proxy-memory-limit": "100Mi" 4
Chapter 13. Configuring burst and QPS for net-kourier
The queries per second (QPS) and burst values determine the frequency of requests or API calls to the API server.
13.1. Configuring burst and QPS values for net-kourier
The queries per second (QPS) value determines the number of client requests or API calls that are sent to the API server.
The burst value determines how many requests from the client can be stored for processing. Requests exceeding this buffer will be dropped. This is helpful for controllers that are bursty and do not spread their requests uniformly in time.
When the net-kourier-controller
restarts, it parses all ingress
resources deployed on the cluster, which leads to a significant number of API calls. Due to this, the net-kourier-controller
can take a long time to start.
You can adjust the QPS and burst values for the net-kourier-controller
in the KnativeServing CR:
KnativeServing CR example
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: workloads: - name: net-kourier-controller env: - container: controller envVars: - name: KUBE_API_BURST value: "200" 1 - name: KUBE_API_QPS value: "200" 2
Chapter 14. Configuring custom domains for Knative services
14.1. Configuring a custom domain for a Knative service
Knative services are automatically assigned a default domain name based on your cluster configuration. For example, <service_name>-<namespace>.example.com
. You can customize the domain for your Knative service by mapping a custom domain name that you own to a Knative service.
You can do this by creating a DomainMapping
resource for the service. You can also create multiple DomainMapping
resources to map multiple domains and subdomains to a single service.
14.2. Custom domain mapping
You can customize the domain for your Knative service by mapping a custom domain name that you own to a Knative service. To map a custom domain name to a custom resource (CR), you must create a DomainMapping
CR that maps to an Addressable target CR, such as a Knative service or a Knative route.
14.2.1. Creating a custom domain mapping
You can customize the domain for your Knative service by mapping a custom domain name that you own to a Knative service. To map a custom domain name to a custom resource (CR), you must create a DomainMapping
CR that maps to an Addressable target CR, such as a Knative service or a Knative route.
Prerequisites
- The OpenShift Serverless Operator and Knative Serving are installed on your cluster.
-
Install the OpenShift CLI (
oc
). - You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
You have created a Knative service and control a custom domain that you want to map to that service.
NoteYour custom domain must point to the IP address of the OpenShift Container Platform cluster.
Procedure
Create a YAML file containing the
DomainMapping
CR in the same namespace as the target CR you want to map to:apiVersion: serving.knative.dev/v1beta1 kind: DomainMapping metadata: name: <domain_name> 1 namespace: <namespace> 2 spec: ref: name: <target_name> 3 kind: <target_type> 4 apiVersion: serving.knative.dev/v1
Example service domain mapping
apiVersion: serving.knative.dev/v1beta1 kind: DomainMapping metadata: name: example.com namespace: default spec: ref: name: showcase kind: Service apiVersion: serving.knative.dev/v1
Example route domain mapping
apiVersion: serving.knative.dev/v1beta1 kind: DomainMapping metadata: name: example.com namespace: default spec: ref: name: example-route kind: Route apiVersion: serving.knative.dev/v1
Apply the
DomainMapping
CR as a YAML file:$ oc apply -f <filename>
14.3. Custom domains for Knative services using the Knative CLI
You can customize the domain for your Knative service by mapping a custom domain name that you own to a Knative service. You can use the Knative (kn
) CLI to create a DomainMapping
custom resource (CR) that maps to an Addressable target CR, such as a Knative service or a Knative route.
14.3.1. Creating a custom domain mapping by using the Knative CLI
Prerequisites
- The OpenShift Serverless Operator and Knative Serving are installed on your cluster.
You have created a Knative service or route, and control a custom domain that you want to map to that CR.
NoteYour custom domain must point to the DNS of the OpenShift Container Platform cluster.
-
You have installed the Knative (
kn
) CLI. - You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
Procedure
Map a domain to a CR in the current namespace:
$ kn domain create <domain_mapping_name> --ref <target_name>
Example command
$ kn domain create example.com --ref showcase
The
--ref
flag specifies an Addressable target CR for domain mapping.If a prefix is not provided when using the
--ref
flag, it is assumed that the target is a Knative service in the current namespace.Map a domain to a Knative service in a specified namespace:
$ kn domain create <domain_mapping_name> --ref <ksvc:service_name:service_namespace>
Example command
$ kn domain create example.com --ref ksvc:showcase:example-namespace
Map a domain to a Knative route:
$ kn domain create <domain_mapping_name> --ref <kroute:route_name>
Example command
$ kn domain create example.com --ref kroute:example-route
14.4. Domain mapping using the Developer perspective
You can customize the domain for your Knative service by mapping a custom domain name that you own to a Knative service. You can use the Developer perspective of the OpenShift Container Platform web console to map a DomainMapping
custom resource (CR) to a Knative service.
14.4.1. Mapping a custom domain to a service by using the Developer perspective
Prerequisites
- You have logged in to the web console.
- You are in the Developer perspective.
- The OpenShift Serverless Operator and Knative Serving are installed on your cluster. This must be completed by a cluster administrator.
- You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
You have created a Knative service and control a custom domain that you want to map to that service.
NoteYour custom domain must point to the IP address of the OpenShift Container Platform cluster.
Procedure
- Navigate to the Topology page.
-
Right-click the service you want to map to a domain, and select the Edit option that contains the service name. For example, if the service is named
showcase
, select the Edit showcase option. In the Advanced options section, click Show advanced Routing options.
- If the domain mapping CR that you want to map to the service already exists, you can select it in the Domain mapping list.
-
If you want to create a new domain mapping CR, type the domain name into the box, and select the Create option. For example, if you type in
example.com
, the Create option is Create "example.com".
- Click Save to save the changes to your service.
Verification
- Navigate to the Topology page.
- Click on the service that you have created.
- In the Resources tab of the service information window, you can see the domain you have mapped to the service listed under Domain mappings.
14.5. Domain mapping using the Administrator perspective
If you do not want to switch to the Developer perspective in the OpenShift Container Platform web console or use the Knative (kn
) CLI or YAML files, you can use the Administator perspective of the OpenShift Container Platform web console.
14.5.1. Mapping a custom domain to a service by using the Administrator perspective
Knative services are automatically assigned a default domain name based on your cluster configuration. For example, <service_name>-<namespace>.example.com
. You can customize the domain for your Knative service by mapping a custom domain name that you own to a Knative service.
You can do this by creating a DomainMapping
resource for the service. You can also create multiple DomainMapping
resources to map multiple domains and subdomains to a single service.
If you have cluster administrator permissions on OpenShift Container Platform (or cluster or dedicated administrator permissions on OpenShift Dedicated or Red Hat OpenShift Service on AWS), you can create a DomainMapping
custom resource (CR) by using the Administrator perspective in the web console.
Prerequisites
- You have logged in to the web console.
- You are in the Administrator perspective.
- You have installed the OpenShift Serverless Operator.
- You have installed Knative Serving.
- You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads.
You have created a Knative service and control a custom domain that you want to map to that service.
NoteYour custom domain must point to the IP address of the cluster.
Procedure
- Navigate to CustomResourceDefinitions and use the search box to find the DomainMapping custom resource definition (CRD).
- Click the DomainMapping CRD, then navigate to the Instances tab.
- Click Create DomainMapping.
Modify the YAML for the
DomainMapping
CR so that it includes the following information for your instance:apiVersion: serving.knative.dev/v1beta1 kind: DomainMapping metadata: name: <domain_name> 1 namespace: <namespace> 2 spec: ref: name: <target_name> 3 kind: <target_type> 4 apiVersion: serving.knative.dev/v1
Example domain mapping to a Knative service
apiVersion: serving.knative.dev/v1beta1 kind: DomainMapping metadata: name: custom-ksvc-domain.example.com namespace: default spec: ref: name: showcase kind: Service apiVersion: serving.knative.dev/v1
Verification
Access the custom domain by using a
curl
request. For example:Example command
$ curl custom-ksvc-domain.example.com
Example output
{"artifact":"knative-showcase","greeting":"Welcome"}
14.5.2. Restricting cipher suites by using the Administrator perspective
When you specify net-kourier
for ingress and use DomainMapping
, the TLS for OpenShift routing is set to passthrough, and TLS is handled by the Kourier Gateway. In such cases, you might need to restrict which TLS cipher suites for Kourier are allowed for users.
Prerequisites
- You have logged in to the web console.
- You are in the Administrator perspective.
- You have installed the OpenShift Serverless Operator.
- You have installed Knative Serving.
You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads.
NoteYour custom domain must point to the IP address of the cluster.
Procedure
In the
KnativeServing
CR, use thecipher-suites
value to specify the cipher suites you want to enable:KnativeServing CR example
spec: config: kourier: cipher-suites: ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-ECDSA-CHACHA20-POLY1305
Other cipher suites will be disabled. You can specify multiple suites by separating them with commas.
NoteThe Kourier Gateway’s container image utilizes the Envoy proxy image, and the default enabled cipher suites depend on the version of the Envoy proxy.
14.6. Securing a mapped service using a TLS certificate
14.6.1. Securing a service with a custom domain by using a TLS certificate
After you have configured a custom domain for a Knative service, you can use a TLS certificate to secure the mapped service. To do this, you must create a Kubernetes TLS secret, and then update the DomainMapping
CR to use the TLS secret that you have created.
Prerequisites
-
You configured a custom domain for a Knative service and have a working
DomainMapping
CR. - You have a TLS certificate from your Certificate Authority provider or a self-signed certificate.
-
You have obtained the
cert
andkey
files from your Certificate Authority provider, or a self-signed certificate. -
Install the OpenShift CLI (
oc
).
Procedure
Create a Kubernetes TLS secret:
$ oc create secret tls <tls_secret_name> --cert=<path_to_certificate_file> --key=<path_to_key_file>
Add the
networking.internal.knative.dev/certificate-uid: <id>`
label to the Kubernetes TLS secret:$ oc label secret <tls_secret_name> networking.internal.knative.dev/certificate-uid="<id>"
If you are using a third-party secret provider such as
cert-manager
, you can configure your secret manager to label the Kubernetes TLS secret automatically.cert-manager
users can use the secret template offered to automatically generate secrets with the correct label. In this case, secret filtering is done based on the key only, but this value can carry useful information such as the certificate ID that the secret contains.NoteThe cert-manager Operator for Red Hat OpenShift is a Technology Preview feature. For more information, see the Installing the cert-manager Operator for Red Hat OpenShift documentation.
Update the
DomainMapping
CR to use the TLS secret that you have created:apiVersion: serving.knative.dev/v1beta1 kind: DomainMapping metadata: name: <domain_name> namespace: <namespace> spec: ref: name: <service_name> kind: Service apiVersion: serving.knative.dev/v1 # TLS block specifies the secret to be used tls: secretName: <tls_secret_name>
Verification
Verify that the
DomainMapping
CR status isTrue
, and that theURL
column of the output shows the mapped domain with the schemehttps
:$ oc get domainmapping <domain_name>
Example output
NAME URL READY REASON example.com https://example.com True
Optional: If the service is exposed publicly, verify that it is available by running the following command:
$ curl https://<domain_name>
If the certificate is self-signed, skip verification by adding the
-k
flag to thecurl
command.
14.6.2. Improving net-kourier memory usage by using secret filtering
By default, the informers implementation for the Kubernetes client-go
library fetches all resources of a particular type. This can lead to a substantial overhead when many resources are available, which can cause the Knative net-kourier
ingress controller to fail on large clusters due to memory leaking. However, a filtering mechanism is available for the Knative net-kourier
ingress controller, which enables the controller to only fetch Knative related secrets.
The secret filtering is enabled by default on the OpenShift Serverless Operator side. An environment variable, ENABLE_SECRET_INFORMER_FILTERING_BY_CERT_UID=true
, is added by default to the net-kourier
controller pods.
If you enable secret filtering, all of your secrets need to be labeled with networking.internal.knative.dev/certificate-uid: "<id>"
. Otherwise, Knative Serving does not detect them, which leads to failures. You must label both new and existing secrets.
Prerequisites
- You have cluster administrator permissions on OpenShift Container Platform, or you have cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
- A project that you created or that you have roles and permissions for to create applications and other workloads.
- Install the OpenShift Serverless Operator and Knative Serving.
-
Install the OpenShift CLI (
oc
).
You can disable the secret filtering by setting the ENABLE_SECRET_INFORMER_FILTERING_BY_CERT_UID
variable to false
by using the workloads
field in the KnativeServing
custom resource (CR).
Example KnativeServing CR
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: ... workloads: - env: - container: controller envVars: - name: ENABLE_SECRET_INFORMER_FILTERING_BY_CERT_UID value: 'false' name: net-kourier-controller
Chapter 15. High availability configuration for Knative Serving
15.1. High availability for Knative services
High availability (HA) is a standard feature of Kubernetes APIs that helps to ensure that APIs stay operational if a disruption occurs. In an HA deployment, if an active controller crashes or is deleted, another controller is readily available. This controller takes over processing of the APIs that were being serviced by the controller that is now unavailable.
HA in OpenShift Serverless is available through leader election, which is enabled by default after the Knative Serving or Eventing control plane is installed. When using a leader election HA pattern, instances of controllers are already scheduled and running inside the cluster before they are required. These controller instances compete to use a shared resource, known as the leader election lock. The instance of the controller that has access to the leader election lock resource at any given time is called the leader.
15.2. High availability for Knative deployments
High availability (HA) is available by default for the Knative Serving activator
, autoscaler
, autoscaler-hpa
, controller
, webhook
, domain-mapping
, domainmapping-webhook
, kourier-control
, and kourier-gateway
components, which are configured to have two replicas each. You can change the number of replicas for these components by modifying the spec.high-availability.replicas
value in the KnativeServing
custom resource (CR).
15.2.1. Configuring high availability replicas for Knative Serving
To specify three minimum replicas for the eligible deployment resources, set the value of the field spec.high-availability.replicas
in the custom resource to 3
.
Prerequisites
- You have cluster administrator permissions on OpenShift Container Platform, or you have cluster or dedicated administrator permissions on Red Hat OpenShift Service on AWS or OpenShift Dedicated.
- The OpenShift Serverless Operator and Knative Serving are installed on your cluster.
Procedure
- In the OpenShift Container Platform web console Administrator perspective, navigate to OperatorHub → Installed Operators.
-
Select the
knative-serving
namespace. - Click Knative Serving in the list of Provided APIs for the OpenShift Serverless Operator to go to the Knative Serving tab.
Click knative-serving, then go to the YAML tab in the knative-serving page.
Modify the number of replicas in the
KnativeServing
CR:Example YAML
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: high-availability: replicas: 3
You can also specify the number of replicas for a specific workload.
NoteWorkload-specific configuration overrides the global setting for Knative Serving.
Example YAML
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: high-availability: replicas: 3 workloads: - name: webhook replicas: 4
Verify that the high availability limits are respected:
Example command
$ oc get hpa -n knative-serving
Example output
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE activator Deployment/activator 0%/100% 3 22 3 2m24s webhook Deployment/webhook 2%/100% 4 8 4 2m23s
15.2.2. Overriding disruption budgets
A Pod Disruption Budget (PDB) is a standard feature of Kubernetes APIs that helps limit the disruption to an application when its pods need to be rescheduled for maintenance reasons.
Procedure
-
Override the default PDB for a specific resource by modifying the
minAvailable
configuration value in theKnativeServing
custom resource (CR).
Example PDB with a minAvailable
seting of 70%
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: podDisruptionBudgets: - name: activator-pdb minAvailable: 70%
If you disable high-availability, for example, by changing the high-availability.replicas
value to 1
, make sure you also update the corresponding PDB minAvailable
value to 0
. Otherwise, the pod disruption budget prevents automatic cluster or Operator updates.
Chapter 16. Tuning serving configuration
16.1. Overriding Knative Serving system deployment configurations
You can override the default configurations for some specific deployments by modifying the workloads
spec in the KnativeServing
custom resources (CRs).
16.1.1. Overriding system deployment configurations
Currently, overriding default configuration settings is supported for the resources
, replicas
, labels
, annotations
, and nodeSelector
fields, as well as for the readiness
and liveness
fields for probes.
In the following example, a KnativeServing
CR overrides the webhook
deployment so that:
-
The
readiness
probe timeout fornet-kourier-controller
is set to be 10 seconds. - The deployment has specified CPU and memory resource limits.
- The deployment has 3 replicas.
-
The
example-label: label
label is added. -
The
example-annotation: annotation
annotation is added. -
The
nodeSelector
field is set to select nodes with thedisktype: hdd
label.
The KnativeServing
CR label and annotation settings override the deployment’s labels and annotations for both the deployment itself and the resulting pods.
KnativeServing CR example
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
name: ks
namespace: knative-serving
spec:
high-availability:
replicas: 2
workloads:
- name: net-kourier-controller
readinessProbes: 1
- container: controller
timeoutSeconds: 10
- name: webhook
resources:
- container: webhook
requests:
cpu: 300m
memory: 60Mi
limits:
cpu: 1000m
memory: 1000Mi
replicas: 3
labels:
example-label: label
annotations:
example-annotation: annotation
nodeSelector:
disktype: hdd
- 1
- You can use the
readiness
andliveness
probe overrides to override all fields of a probe in a container of a deployment as specified in the Kubernetes API except for the fields related to the probe handler:exec
,grpc
,httpGet
, andtcpSocket
.
Additional resources
Chapter 17. Configuring Queue Proxy resources
The Queue Proxy is a sidecar container to each application container within a service. It improves managing Serverless workloads, ensuring efficient resource usage. You can configure the Queue Proxy.
17.1. Configuring Queue Proxy Resources for a Knative Service
Apart from configuring the Queue Proxy resource requests and limits globally in the deployment configmap, you can set them at the service level using the corresponding annotations targeting CPU, memory, and ephemeral storage resource types.
Prerequisites
- Red Hat OpenShift Pipelines must be installed on your cluster.
-
You have installed the OpenShift (
oc
) CLI. -
You have installed the Knative (
kn
) CLI.
Procedure
Modify the configmap of your service with resource requests and limits:
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: example-service namespace: default spec: template: metadata: annotations: queue.sidecar.serving.knative.dev/cpu-resource-request: "1" queue.sidecar.serving.knative.dev/cpu-resource-limit: "2" queue.sidecar.serving.knative.dev/memory-resource-request: "1Gi" queue.sidecar.serving.knative.dev/memory-resource-limit: "2Gi" queue.sidecar.serving.knative.dev/ephemeral-storage-resource-request: "400Mi" queue.sidecar.serving.knative.dev/ephemeral-storage-resource-limit: "450Mi"
Alternatively, you can use the special
queue.sidecar.serving.knative.dev/resource-percentage
annotation, which calculates the Queue Proxy resources as a percentage of the application container. When CPU and memory resource requirements are calculated from the application container requirements and they are outside the boundaries below, the values are adjusted to fit within the boundaries. In this case, the following minimum and maximum boundaries are applied to the CPU and memory resource requirements:Table 17.1. Resource requirements boundaries Resource requirements Min Max CPU request
25m
100m
CPU limit
40m
500m
Memory request
50Mi
200Mi
Memory limit
200Mi
500Mi
NoteIf you simultaneously set a percentage annotation and a specific resource value using the corresponding resource annotation, then the latter takes precedence.
WarningThe
queue.sidecar.serving.knative.dev/resource-percentage
annotation is now deprecated and will be removed in a future version of OpenShift Serverless.