Chapter 1. Release notes

In OpenShift Serverless 1.29 or later, the following components of Knative Eventing will be scaled down from two pods to one:

The serverless.openshift.io/enable-secret-informer-filtering annotation for the Serving CR is now deprecated. The annotation is valid only for Istio, and not for Kourier.

With OpenShift Serverless 1.28, the OpenShift Serverless Operator allows injecting the environment variable ENABLE_SECRET_INFORMER_FILTERING_BY_CERT_UID for both net-istio and net-kourier.

To prevent problems when upgrading from OpenShift Serverless 1.28 to some future version, users must annotate their secrets with networking.internal.knative.dev/certificate-uid:some_cuid.

Currently, runtimes for Python are not supported for OpenShift Serverless Functions on IBM Power, IBM zSystems, and IBM® LinuxONE.

Node.js, TypeScript, and Quarkus functions are supported on these architectures.

On the Windows platform, Python functions cannot be locally built, run, or deployed using the Source-to-Image builder due to the app.sh file permissions.

To work around this problem, use the Windows Subsystem for Linux.

Namespace-scoped brokers might leave ClusterRoleBindings in the user namespace even after deletion of namespace-scoped brokers.

If this happens, delete the ClusterRoleBinding named rbac-proxy-reviews-prom-rb-knative-kafka-broker-data-plane-{{.Namespace}} in the user namespace.

If you use net-istio for Ingress and enable mTLS via SMCP using security.dataPlane.mtls: true, Service Mesh deploys DestinationRules for the *.local host, which does not allow DomainMapping for OpenShift Serverless.

To work around this issue, enable mTLS by deploying PeerAuthentication instead of using security.dataPlane.mtls: true.

The new trigger filters feature is now available as a Developer Preview. It allows users to specify a set of filter expressions, where each expression evaluates to either true or false for each event.

To enable new trigger filters, add the new-trigger-filters: enabled entry in the section of the KnativeEventing type in the operator config map:

apiVersion: operator.knative.dev/v1beta1
kind: KnativeEventing
...
...
spec:
  config:
    features:
      new-trigger-filters: enabled
...

Knative Operator 1.3 adds the updated v1beta1 version of the API for operator.knative.dev.

To update from v1alpha1 to v1beta1 in your KnativeServing and KnativeEventing custom resource config maps, edit the apiVersion key:

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
...

apiVersion: operator.knative.dev/v1beta1
kind: KnativeEventing
...

If you use net-istio for Ingress and enable mTLS via SMCP using security.dataPlane.mtls: true, Service Mesh deploys DestinationRules for the *.local host, which does not allow DomainMapping for OpenShift Serverless.

To work around this issue, enable mTLS by deploying PeerAuthentication instead of using security.dataPlane.mtls: true.

Knative Kafka broker is now GA. Knative Kafka broker is a highly performant implementation of the Knative broker API, directly targeting Apache Kafka.

It is recommended to not use the MT-Channel-Broker, but the Knative Kafka broker instead.

The Knative Serving Controller pod adds a new informer to watch secrets in the cluster. The informer includes the secrets in the cache, which increases memory consumption of the controller pod.

If the pod runs out of memory, you can work around the issue by increasing the memory limit for the deployment.

If you use net-istio for Ingress and enable mTLS via SMCP using security.dataPlane.mtls: true, Service Mesh deploys DestinationRules for the *.local host, which does not allow DomainMapping for OpenShift Serverless.

To work around this issue, enable mTLS by deploying PeerAuthentication instead of using security.dataPlane.mtls: true.

Integrating OpenShift Serverless with Red Hat OpenShift Service Mesh causes the net-istio-controller pod to run out of memory on startup when too many secrets are present on the cluster.

It is now possible to enable secret filtering, which causes net-istio-controller to consider only secrets with a networking.internal.knative.dev/certificate-uid label, thus reducing the amount of memory needed.

In OpenShift Serverless 1.23, support for KafkaBindings and the kafka-binding webhook were removed. However, an existing kafkabindings.webhook.kafka.sources.knative.dev MutatingWebhookConfiguration might remain, pointing to the kafka-source-webhook service, which no longer exists.

For certain specifications of KafkaBindings on the cluster, kafkabindings.webhook.kafka.sources.knative.dev MutatingWebhookConfiguration might be configured to pass any create and update events to various resources, such as Deployments, Knative Services, or Jobs, through the webhook, which would then fail.

To work around this issue, manually delete kafkabindings.webhook.kafka.sources.knative.dev MutatingWebhookConfiguration from the cluster after upgrading to OpenShift Serverless 1.23:

$ oc delete mutatingwebhookconfiguration kafkabindings.webhook.kafka.sources.knative.dev

If you use net-istio for Ingress and enable mTLS via SMCP using security.dataPlane.mtls: true, Service Mesh deploys DestinationRules for the *.local host, which does not allow DomainMapping for OpenShift Serverless.

To work around this issue, enable mTLS by deploying PeerAuthentication instead of using security.dataPlane.mtls: true.

Running OpenShift Serverless with a large number of Knative services can cause Knative activator pods to run close to their default memory limits of 600MB. These pods might be restarted if memory consumption reaches this limit. Requests and limits for the activator deployment can be configured by modifying the KnativeServing custom resource:

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  deployments:
  - name: activator
    resources:
    - container: activator
      requests:
        cpu: 300m
        memory: 60Mi
      limits:
        cpu: 1000m
        memory: 1000Mi

If you use net-istio for Ingress and enable mTLS via SMCP using security.dataPlane.mtls: true, Service Mesh deploys DestinationRules for the *.local host, which does not allow DomainMapping for OpenShift Serverless.

To work around this issue, enable mTLS by deploying PeerAuthentication instead of using security.dataPlane.mtls: true.

Currently, the domain mapping controller cannot process the URI of a broker, which contains a path that is currently not supported.

This means that, if you want to use a DomainMapping custom resource (CR) to map a custom domain to a broker, you must configure the DomainMapping CR with the broker’s ingress service, and append the exact path of the broker to the custom domain:

apiVersion: serving.knative.dev/v1alpha1
kind: DomainMapping
metadata:
  name: <domain-name>
  namespace: knative-eventing
spec:
  ref:
    name: broker-ingress
    kind: Service
    apiVersion: v1

The URI for the broker is then <domain-name>/<broker-namespace>/<broker-name>.

The Kafka broker is now available as a Technology Preview.

OpenShift Serverless deploys Knative services with a default address that uses HTTPS. When sending an event to a resource inside the cluster, the sender does not have the cluster certificate authority (CA) configured. This causes event delivery to fail, unless the cluster uses globally accepted certificates.

For example, an event delivery to a publicly accessible address works:

$ kn event send --to-url https://ce-api.foo.example.com/

On the other hand, this delivery fails if the service uses a public address with an HTTPS certificate issued by a custom CA:

$ kn event send --to Service:serving.knative.dev/v1:event-display

Sending an event to other addressable objects, such as brokers or channels, is not affected by this issue and works as expected.

If you create a Springboot function project directory with the kn func create command, subsequent running of the kn func build command fails with this error message:

[analyzer] no stack metadata found at path ''
[analyzer] ERROR: failed to : set API for buildpack 'paketo-buildpacks/ca-certificates@3.0.2': buildpack API version '0.7' is incompatible with the lifecycle

As a workaround, you can change the builder property to gcr.io/paketo-buildpacks/builder:base in the function configuration file func.yaml.

Deploying a function using the gcr.io registry fails with this error message:

Error: failed to get credentials: failed to verify credentials: status code: 404

As a workaround, use a different registry than gcr.io, such as quay.io or docker.io.

TypeScript functions created with the http template fail to deploy on the cluster.

As a workaround, in the func.yaml file, replace the following section:

buildEnvs: []

with this:

buildEnvs:
- name: BP_NODE_RUN_SCRIPTS
  value: build

In func version 0.20, some runtimes might be unable to build a function by using podman. You might see an error message similar to the following:

ERROR: failed to image: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.40/info": EOF

In func version 0.19, some runtimes might be unable to build a function by using podman. You might see an error message similar to the following:

ERROR: failed to image: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.40/info": EOF

In the upcoming OpenShift Serverless 1.19.0 release, the URL scheme of external routes will default to HTTPS for enhanced security.

If you do not want this change to apply for your workloads, you can override the default setting before upgrading to 1.19.0, by adding the following YAML to your KnativeServing custom resource (CR):

...
spec:
  config:
    network:
      defaultExternalScheme: "http"
...

If you want the change to apply in 1.18.0 already, add the following YAML:

...
spec:
  config:
    network:
      defaultExternalScheme: "https"
...

In the upcoming OpenShift Serverless 1.19.0 release, the default service type by which the Kourier Gateway is exposed will be ClusterIP and not LoadBalancer.

If you do not want this change to apply to your workloads, you can override the default setting before upgrading to 1.19.0, by adding the following YAML to your KnativeServing custom resource (CR):

...
spec:
  ingress:
    kourier:
      service-type: LoadBalancer
...

Previously, if you created a new subscription for a Kafka channel, or a new Kafka source, a delay was possible in the Kafka data plane becoming ready to dispatch messages after the newly created subscription or sink reported a ready status.

As a result, messages that were sent during the time when the data plane was not reporting a ready status, might not have been delivered to the subscriber or sink.

In OpenShift Serverless 1.18.0, the issue is fixed and the initial messages are no longer lost. For more information about the issue, see Knowledgebase Article #6343981.

Older versions of the Knative kn CLI might use older versions of the Knative Serving and Knative Eventing APIs. For example, version 0.23.2 of the kn CLI uses the v1alpha1 API version.

On the other hand, newer releases of OpenShift Serverless might no longer support older API versions. For example, OpenShift Serverless 1.18.0 no longer supports version v1alpha1 of the kafkasources.sources.knative.dev API.

Consequently, using an older version of the Knative kn CLI with a newer OpenShift Serverless might produce an error because the kn cannot find the outdated API. For example, version 0.23.2 of the kn CLI does not work with OpenShift Serverless 1.18.0.

To avoid issues, use the latest kn CLI version available for your OpenShift Serverless release. For OpenShift Serverless 1.18.0, use Knative kn CLI 0.24.0.

In the upcoming OpenShift Serverless 1.19.0 release, the URL scheme of external routes will default to HTTPS for enhanced security.

If you do not want this change to apply for your workloads, you can override the default setting before upgrading to 1.19.0, by adding the following YAML to your KnativeServing custom resource (CR):

...
spec:
  config:
    network:
      defaultExternalScheme: "http"
...

Changes to API versions in Knative Eventing 0.23.0:

If you try to use an older version of the Knative kn CLI with a newer OpenShift Serverless release, the API is not found and an error occurs.

For example, if you use the 1.16.0 release of the kn CLI, which uses version 0.22.0, with the 1.17.0 OpenShift Serverless release, which uses the 0.23.0 versions of the Knative Serving and Knative Eventing APIs, the CLI does not work because it continues to look for the outdated 0.22.0 API versions.

Ensure that you are using the latest kn CLI version for your OpenShift Serverless release to avoid issues.

If you create a new subscription for a Kafka channel, or a new Kafka source, there might be a delay in the Kafka data plane becoming ready to dispatch messages after the newly created subscription or sink reports a ready status.

As a result, messages that are sent during the time when the data plane is not reporting a ready status might not be delivered to the subscriber or sink.

For more information about this issue and possible workarounds, see Knowledge Article #6343981.

The Camel-K 1.4 release is not compatible with OpenShift Serverless version 1.17.0. This is because Camel-K 1.4 uses APIs that were removed in Knative version 0.23.0. There is currently no workaround available for this issue. If you need to use Camel-K 1.4 with OpenShift Serverless, do not upgrade to OpenShift Serverless version 1.17.0.

The AMQ Streams Operator might prevent the installation or upgrade of the OpenShift Serverless Operator. If this happens, the following error is thrown by Operator Lifecycle Manager (OLM):

WARNING: found multiple channel heads: [amqstreams.v1.7.2 amqstreams.v1.6.2], please check the `replaces`/`skipRange` fields of the operator bundles.

You can fix this issue by uninstalling the AMQ Streams Operator before installing or upgrading the OpenShift Serverless Operator. You can then reinstall the AMQ Streams Operator.

If you deploy Service Mesh CRs with the Istio ingress enabled, you might see the following warning in the istio-ingressgateway pod:

2021-05-02T12:56:17.700398Z warning envoy config [external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8081: duplicate listener 0.0.0.0_8081 found

Your Knative services might also not be accessible.

You can use the following workaround to fix this issue by recreating the knative-local-gateway service:

If you have 1000 Knative services on a cluster, and then perform a reinstall or upgrade of Knative Serving, there is a delay when you create the first new service after the KnativeServing custom resource (CR) becomes Ready.

The 3scale-kourier-control service reconciles all previously existing Knative services before processing the creation of a new service, which causes the new service to spend approximately 800 seconds in an IngressNotConfigured or Unknown state before the state updates to Ready.

If you create a new subscription for a Kafka channel, or a new Kafka source, there might be a delay in the Kafka data plane becoming ready to dispatch messages after the newly created subscription or sink reports a ready status.

As a result, messages that are sent during the time when the data plane is not reporting a ready status might not be delivered to the subscriber or sink.

For more information about this issue and possible workarounds, see Knowledge Article #6343981.