Chapter 1. Service Mesh 2.x

• Added an option to the must-gather data collection tool that gathers information from a specified Service Mesh control plane namespace. For more information, see OSSM-351.
• Improved performance for Service Mesh control planes with hundreds of namespaces

• Simplifies installation, upgrades, and management of the Service Mesh control plane.
• Reduces the Service Mesh control plane’s resource usage and startup time.

• Improves performance by reducing inter-control plane communication over networking.

  • Adds support for Envoy’s Secret Discovery Service (SDS). SDS is a more secure and efficient mechanism for delivering secrets to Envoy side car proxies.
• Removes the need to use Kubernetes Secrets, which have well known security risks.

• Improves performance during certificate rotation, as proxies no longer require a restart to recognize new certificates.

  • Adds support for Istio’s Telemetry v2 architecture, which is built using WebAssembly extensions. This new architecture brings significant performance improvements.
  • Updates the ServiceMeshControlPlane resource to v2 with a streamlined configuration to make it easier to manage the Service Mesh Control Plane.
  • Introduces WebAssembly extensions as a Technology Preview feature.

• ECDHE-ECDSA-AES128-SHA
• ECDHE-RSA-AES128-SHA
• AES128-GCM-SHA256
• AES128-SHA
• ECDHE-ECDSA-AES256-SHA
• ECDHE-RSA-AES256-SHA
• AES256-GCM-SHA384
• AES256-SHA

• ECDHE-ECDSA-AES128-SHA
• ECDHE-RSA-AES128-SHA
• AES128-GCM-SHA256
• AES128-SHA
• ECDHE-ECDSA-AES256-SHA
• ECDHE-RSA-AES256-SHA
• AES256-GCM-SHA384
• AES256-SHA

• Policy (authentication.istio.io/v1alpha1) is no longer supported. Depending on the specific configuration in your Policy resource, you may have to configure multiple resources to achieve the same effect.

  • Use RequestAuthentication (security.istio.io/v1beta1)
  • Use PeerAuthentication (security.istio.io/v1beta1)

• ServiceMeshPolicy (maistra.io/v1) is no longer supported.

  • Use RequestAuthentication or PeerAuthentication, as mentioned above, but place in the Service Mesh control plane namespace.

• RbacConfig (rbac.istio.io/v1alpha1) is no longer supported.

  • Replaced by AuthorizationPolicy (security.istio.io/v1beta1), which encompasses behavior of RbacConfig, ServiceRole, and ServiceRoleBinding.

• ServiceMeshRbacConfig (maistra.io/v1) is no longer supported.

  • Use AuthorizationPolicy as above, but place in Service Mesh control plane namespace.
• ServiceRole (rbac.istio.io/v1alpha1) is no longer supported.
• ServiceRoleBinding (rbac.istio.io/v1alpha1) is no longer supported.
• In Kiali, the login and LDAP strategies are deprecated. A future version will introduce authentication using OpenID providers.

• OSSM-3890 Attempting to use the Gateway API in a multitenant mesh deployment generates an error message similar to the following:

  2023-05-02T15:20:42.541034Z	error	watch error in cluster Kubernetes: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
  2023-05-02T15:20:42.616450Z	info	kube	controller "gateway.networking.k8s.io/v1alpha2/TCPRoute" is syncing...

  To support Gateway API in a multitenant mesh deployment, all Gateway API Custom Resource Definition (CRD) files must be present in the cluster.

  In a multitenant mesh deployment, CRD scan is disabled, and Istio has no way to discover which CRDs are present in a cluster. As a result, Istio attempts to watch all supported Gateway API CRDs, but generates errors if some of those CRDs are not present.

  Service Mesh 2.3.1 and later versions support both v1alpha2 and v1beta1 CRDs. Therefore, both CRD versions must be present for a multitenant mesh deployment to support the Gateway API.

  Workaround: In the following example, the kubectl get operation installs the v1alpha2 and v1beta1 CRDs. Note the URL contains the additional experimental segment and updates any of your existing scripts accordingly:

  $ kubectl get crd gateways.gateway.networking.k8s.io ||   { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v0.5.1" | kubectl apply -f -; }

• OSSM-2042 Deployment of SMCP named default fails. If you are creating an SMCP object, and set its version field to v2.3, the name of the object cannot be default. If the name is default, then the control plane fails to deploy, and OpenShift generates a Warning event with the following message:

  Error processing component mesh-config: error: [mesh-config/templates/telemetryv2_1.6.yaml: Internal error occurred: failed calling webhook "rev.validation.istio.io": Post "https://istiod-default.istio-system.svc:443/validate?timeout=10s": x509: certificate is valid for istiod.istio-system.svc, istiod-remote.istio-system.svc, istio-pilot.istio-system.svc, not istiod-default.istio-system.svc, mesh-config/templates/enable-mesh-permissive.yaml

• OSSM-1655 Kiali dashboard shows error after enabling mTLS in SMCP.

  After enabling the spec.security.controlPlane.mtls setting in the SMCP, the Kiali console displays the following error message No subsets defined.

• OSSM-1505 This issue only occurs when using the ServiceMeshExtension resource on OpenShift Container Platform 4.11. When you use ServiceMeshExtension on OpenShift Container Platform 4.11 the resource never becomes ready. If you inspect the issue using oc describe ServiceMeshExtension you will see the following error: stderr: Error creating mount namespace before pivot: function not implemented.

  Workaround: ServiceMeshExtension was deprecated in Service Mesh 2.2. Migrate from ServiceMeshExtension to the WasmPlugin resource. For more information, see Migrating from ServiceMeshExtension to WasmPlugin resources.

• OSSM-1396 If a gateway resource contains the spec.externalIPs setting, instead of being recreated when the ServiceMeshControlPlane is updated, the gateway is removed and never recreated.
• OSSM-1168 When service mesh resources are created as a single YAML file, the Envoy proxy sidecar is not reliably injected into pods. When the SMCP, SMMR, and Deployment resources are created individually, the deployment works as expected.

• OSSM-1115 The concurrency field of the spec.proxy API did not propagate to the istio-proxy. The concurrency field works when set with ProxyConfig. The concurrency field specifies the number of worker threads to run. If the field is set to 0, then the number of worker threads available is equal to the number of CPU cores. If the field is not set, then the number of worker threads available defaults to 2.

  In the following example, the concurrency field is set to 0.

  apiVersion: networking.istio.io/v1beta1
  kind: ProxyConfig
  metadata:
    name: mesh-wide-concurrency
    namespace: <istiod-namespace>
  spec:
    concurrency: 0

• OSSM-1052 When configuring a Service ExternalIP for the ingressgateway in the Service Mesh control plane, the service is not created. The schema for the SMCP is missing the parameter for the service.

  Workaround: Disable the gateway creation in the SMCP spec and manage the gateway deployment entirely manually (including Service, Role and RoleBinding).

• OSSM-882 This applies for Service Mesh 2.1 and earlier. Namespace is in the accessible_namespace list but does not appear in Kiali UI. By default, Kiali will not show any namespaces that start with "kube" because these namespaces are typically internal-use only and not part of a mesh.

  For example, if you create a namespace called 'akube-a' and add it to the Service Mesh member roll, then the Kiali UI does not display the namespace. For defined exclusion patterns, the software excludes namespaces that start with or contain the pattern.

  Workaround: Change the Kiali Custom Resource setting so it prefixes the setting with a carat (^). For example:

  api:
    namespaces:
      exclude:
      - "^istio-operator"
      - "^kube-.*"
      - "^openshift.*"
      - "^ibm.*"
      - "^kiali-operator"

• MAISTRA-2692 With Mixer removed, custom metrics that have been defined in Service Mesh 2.0.x cannot be used in 2.1. Custom metrics can be configured using EnvoyFilter. Red Hat is unable to support EnvoyFilter configuration except where explicitly documented. This is due to tight coupling with the underlying Envoy APIs, meaning that backward compatibility cannot be maintained.
• MAISTRA-2648 Service mesh extensions are currently not compatible with meshes deployed on IBM Z.

• MAISTRA-1959 Migration to 2.0 Prometheus scraping (spec.addons.prometheus.scrape set to true) does not work when mTLS is enabled. Additionally, Kiali displays extraneous graph data when mTLS is disabled.

  This problem can be addressed by excluding port 15020 from proxy configuration, for example,

  spec:
    proxy:
      networking:
        trafficControl:
          inbound:
            excludedPorts:
            - 15020

• MAISTRA-453 If you create a new project and deploy pods immediately, sidecar injection does not occur. The operator fails to add the maistra.io/member-of before the pods are created, therefore the pods must be deleted and recreated for sidecar injection to occur.
• MAISTRA-158 Applying multiple gateways referencing the same hostname will cause all gateways to stop functioning.

• KIALI-2206 When you are accessing the Kiali console for the first time, and there is no cached browser data for Kiali, the “View in Grafana” link on the Metrics tab of the Kiali Service Details page redirects to the wrong location. The only way you would encounter this issue is if you are accessing Kiali for the first time.
• KIALI-507 Kiali does not support Internet Explorer 11. This is because the underlying frameworks do not support Internet Explorer. To access the Kiali console, use one of the two most recent versions of the Chrome, Edge, Firefox or Safari browser.

• Apache Spark is not supported.
• The streaming deployment via AMQ/Kafka is unsupported on IBM Z and IBM Power Systems.

• OBSDA-220 In some cases, if you try to pull an image using distributed tracing data collection, the image pull fails and a Failed to pull image error message appears. There is no workaround for this issue.

• TRACING-2057 The Kafka API has been updated to v1beta2 to support the Strimzi Kafka Operator 0.23.0. However, this API version is not supported by AMQ Streams 1.6.3. If you have the following environment, your Jaeger services will not be upgraded, and you cannot create new Jaeger services or modify existing Jaeger services:

  • Jaeger Operator channel: 1.17.x stable or 1.20.x stable

  • AMQ Streams Operator channel: amq-streams-1.6.x

    To resolve this issue, switch the subscription channel for your AMQ Streams Operator to either amq-streams-1.7.x or stable.

• OSSM-4197 Previously, if you deployed a v2.2 or v2.1 of the 'ServiceMeshControlPlane' resource, the /etc/cni/multus/net.d/ directory was not created. As a result, the istio-cni pod failed to become ready, and the istio-cni pods log contained the following message:

  $ error   Installer exits with open /host/etc/cni/multus/net.d/v2-2-istio-cni.kubeconfig.tmp.841118073: no such file or directory

  Now, if you deploy a v2.2 or v2.1 of the 'ServiceMeshControlPlane' resource, the /etc/cni/multus/net.d/ directory is created, and the istio-cni pod becomes ready.

• OSSM-3993 Previously, Kiali only supported OpenShift OAuth via a proxy on the standard HTTPS port of 443. Now, Kiali supports OpenShift OAuth over a non-standard HTTPS port. To enable the port, you must set the spec.server.web_port field to the proxy’s non-standard HTTPS port in the Kiali CR.
• OSSM-3936 Previously, the values for the injection_label_rev and injection_label_name attributes were hardcoded. This prevented custom configurations from taking effect in the Kiali Custom Resource Definition (CRD). Now, the attribute values are not hardcoded. You can customize the values for the injection_label_rev and injection_label_name attributes in the spec.istio_labels specification.
• OSSM-3644 Previously, the federation egress-gateway received the wrong update of network gateway endpoints, causing extra endpoint entries. Now, the federation-egress gateway has been updated on the server side so it receives the correct network gateway endpoints.
• OSSM-3595 Previously, the istio-cni plugin sometimes failed on RHEL because SELinux did not allow the utility iptables-restore to open files in the /tmp directory. Now, SELinux passes iptables-restore via stdin input stream instead of via a file.
• OSSM-3586 Previously, Istio proxies were slow to start when Google Cloud Platform (GCP) metadata servers were not available. When you upgrade to Istio 1.14.6, Istio proxies start as expected on GCP, even if metadata servers are not available.
• OSSM-3025 Istiod sometimes fails to become ready. Sometimes, when a mesh contained many member namespaces, the Istiod pod did not become ready due to a deadlock within Istiod. The deadlock is now resolved and the pod now starts as expected.
• OSSM-2493 Default nodeSelector and tolerations in SMCP not passed to Kiali. The nodeSelector and tolerations you add to SMCP.spec.runtime.defaults are now passed to the Kiali resource.
• OSSM-2492 Default tolerations in SMCP not passed to Jaeger. The nodeSelector and tolerations you add to SMCP.spec.runtime.defaults are now passed to the Jaeger resource.
• OSSM-2374 If you deleted one of the ServiceMeshMember resources, then the Service Mesh operator deleted the ServiceMeshMemberRoll. While this is expected behavior when you delete the last ServiceMeshMember, the operator should not delete the ServiceMeshMemberRoll if it contains any members in addition to the one that was deleted. This issue is now fixed and the operator only deletes the ServiceMeshMemberRoll when the last ServiceMeshMember resource is deleted.

• OSSM-2373 Error trying to get OAuth metadata when logging in. To fetch the cluster version, the system:anonymous account is used. With the cluster’s default bundled ClusterRoles and ClusterRoleBinding, the anonymous account can fetch the version correctly. If the system:anonymous account loses its privileges to fetch the cluster version, OpenShift authentication becomes unusable.

  This is fixed by using the Kiali SA to fetch the cluster version. This also allows for improved security on the cluster.

• OSSM-2371 Despite Kiali being configured as "view-only," a user can change the proxy logging level via the Workload details' Logs tab’s kebab menu. This issue has been fixed so the options under "Set Proxy Log Level" are disabled when Kiali is configured as "view-only."
• OSSM-2344 Restarting Istiod causes Kiali to flood CRI-O with port-forward requests. This issue occurred when Kiali could not connect to Istiod and Kiali simultaneously issued a large number of requests to istiod. Kiali now limits the number of requests it sends to istiod.
• OSSM-2335 Dragging the mouse pointer over the Traces scatterchart plot sometimes caused the Kiali console to stop responding due to concurrent backend requests.

• OSSM-2221 Previously, gateway injection in the ServiceMeshControlPlane namespace was not possible because the ignore-namespace label was applied to the namespace by default.

  When creating a v2.4 control plane, the namespace no longer has the ignore-namespace label applied, and gateway injection is possible.

  In the following example, the oc label command removes the ignore-namespace label from a namespace in an existing deployment:

  $ oc label namespace <istio_system> maistra.io/ignore-namespace-

  In the example above, <istio_system> represents the name of the ServiceMeshControlPlane namespace.

• OSSM-2053 Using Red Hat OpenShift Service Mesh Operator 2.2 or 2.3, during SMCP reconciliation, the SMMR controller removed the member namespaces from SMMR.status.configuredMembers. This caused the services in the member namespaces to become unavailable for a few moments.

  Using Red Hat OpenShift Service Mesh Operator 2.2 or 2.3, the SMMR controller no longer removes the namespaces from SMMR.status.configuredMembers. Instead, the controller adds the namespaces to SMMR.status.pendingMembers to indicate that they are not up-to-date. During reconciliation, as each namespace synchronizes with the SMCP, the namespace is automatically removed from SMMR.status.pendingMembers.

• OSSM-1962 Use EndpointSlices in federation controller. The federation controller now uses EndpointSlices, which improves scalability and performance in large deployments. The PILOT_USE_ENDPOINT_SLICE flag is enabled by default. Disabling the flag prevents use of federation deployments.
• OSSM-1668 A new field spec.security.jwksResolverCA was added to the Version 2.1 SMCP but was missing in the 2.2.0 and 2.2.1 releases. When upgrading from an Operator version where this field was present to an Operator version that was missing this field, the .spec.security.jwksResolverCA field was not available in the SMCP.
• OSSM-1325 istiod pod crashes and displays the following error message: fatal error: concurrent map iteration and map write.

• OSSM-1211 Configuring Federated service meshes for failover does not work as expected.

  The Istiod pilot log displays the following error: envoy connection [C289] TLS error: 337047686:SSL routines:tls_process_server_certificate:certificate verify failed

• OSSM-1099 The Kiali console displayed the message Sorry, there was a problem. Try a refresh or navigate to a different page.
• OSSM-1074 Pod annotations defined in SMCP are not injected in the pods.
• OSSM-999 Kiali retention did not work as expected. Calendar times were greyed out in the dashboard graph.
• OSSM-797 Kiali Operator pod generates CreateContainerConfigError while installing or updating the operator.
• OSSM-722 Namespace starting with kube is hidden from Kiali.
• OSSM-569 There is no CPU memory limit for the Prometheus istio-proxy container. The Prometheus istio-proxy sidecar now uses the resource limits defined in spec.proxy.runtime.container.
• OSSM-535 Support validationMessages in SMCP. The ValidationMessages field in the Service Mesh Control Plane can now be set to True. This writes a log for the status of the resources, which can be helpful when troubleshooting problems.
• OSSM-449 VirtualService and Service causes an error "Only unique values for domains are permitted. Duplicate entry of domain."
• OSSM-419 Namespaces with similar names will all show in Kiali namespace list, even though namespaces may not be defined in Service Mesh Member Role.
• OSSM-296 When adding health configuration to the Kiali custom resource (CR) is it not being replicated to the Kiali configmap.
• OSSM-291 In the Kiali console, on the Applications, Services, and Workloads pages, the "Remove Label from Filters" function is not working.
• OSSM-289 In the Kiali console, on the Service Details pages for the 'istio-ingressgateway' and 'jaeger-query' services there are no Traces being displayed. The traces exist in Jaeger.
• OSSM-287 In the Kiali console there are no traces being displayed on the Graph Service.

• OSSM-285 When trying to access the Kiali console, receive the following error message "Error trying to get OAuth Metadata".

  Workaround: Restart the Kiali pod.

• MAISTRA-2735 The resources that the Service Mesh Operator deletes when reconciling the SMCP changed in Red Hat OpenShift Service Mesh version 2.1. Previously, the Operator deleted a resource with the following labels:

  • maistra.io/owner
  • app.kubernetes.io/version

  Now, the Operator ignores resources that does not also include the app.kubernetes.io/managed-by=maistra-istio-operator label. If you create your own resources, you should not add the app.kubernetes.io/managed-by=maistra-istio-operator label to them.

• MAISTRA-2687 Red Hat OpenShift Service Mesh 2.1 federation gateway does not send the full certificate chain when using external certificates. The Service Mesh federation egress gateway only sends the client certificate. Because the federation ingress gateway only knows about the root certificate, it cannot verify the client certificate unless you add the root certificate to the federation import ConfigMap.
• MAISTRA-2635 Replace deprecated Kubernetes API. To remain compatible with OpenShift Container Platform 4.8, the apiextensions.k8s.io/v1beta1 API was deprecated as of Red Hat OpenShift Service Mesh 2.0.8.
• MAISTRA-2631 The WASM feature is not working because podman is failing due to nsenter binary not being present. Red Hat OpenShift Service Mesh generates the following error message: Error: error configuring CNI network plugin exec: "nsenter": executable file not found in $PATH. The container image now contains nsenter and WASM works as expected.
• MAISTRA-2534 When istiod attempted to fetch the JWKS for an issuer specified in a JWT rule, the issuer service responded with a 502. This prevented the proxy container from becoming ready and caused deployments to hang. The fix for the community bug has been included in the Service Mesh 2.0.7 release.

• MAISTRA-2411 When the Operator creates a new ingress gateway using spec.gateways.additionaIngress in the ServiceMeshControlPlane, Operator is not creating a NetworkPolicy for the additional ingress gateway like it does for the default istio-ingressgateway. This is causing a 503 response from the route of the new gateway.

  Workaround: Manually create the NetworkPolicy in the <istio-system> namespace.

• MAISTRA-2401 CVE-2021-3586 servicemesh-operator: NetworkPolicy resources incorrectly specified ports for ingress resources. The NetworkPolicy resources installed for Red Hat OpenShift Service Mesh did not properly specify which ports could be accessed. This allowed access to all ports on these resources from any pod. Network policies applied to the following resources are affected:

  • Galley
  • Grafana
  • Istiod
  • Jaeger
  • Kiali
  • Prometheus
  • Sidecar injector
• MAISTRA-2378 When the cluster is configured to use OpenShift SDN with ovs-multitenant and the mesh contains a large number of namespaces (200+), the OpenShift Container Platform networking plugin is unable to configure the namespaces quickly. Service Mesh times out causing namespaces to be continuously dropped from the service mesh and then reenlisted.
• MAISTRA-2370 Handle tombstones in listerInformer. The updated cache codebase was not handling tombstones when translating the events from the namespace caches to the aggregated cache, leading to a panic in the go routine.

• MAISTRA-2117 Add optional ConfigMap mount to operator. The CSV now contains an optional ConfigMap volume mount, which mounts the smcp-templates ConfigMap if it exists. If the smcp-templates ConfigMap does not exist, the mounted directory is empty. When you create the ConfigMap, the directory is populated with the entries from the ConfigMap and can be referenced in SMCP.spec.profiles. No restart of the Service Mesh operator is required.

  Customers using the 2.0 operator with a modified CSV to mount the smcp-templates ConfigMap can upgrade to Red Hat OpenShift Service Mesh 2.1. After upgrading, you can continue using an existing ConfigMap, and the profiles it contains, without editing the CSV. Customers that previously used ConfigMap with a different name will either have to rename the ConfigMap or update the CSV after upgrading.

• MAISTRA-2010 AuthorizationPolicy does not support request.regex.headers field. The validatingwebhook rejects any AuthorizationPolicy with the field, and even if you disable that, Pilot tries to validate it using the same code, and it does not work.

• MAISTRA-1979 Migration to 2.0 The conversion webhook drops the following important fields when converting SMCP.status from v2 to v1:

  • conditions
  • components
  • observedGeneration

  • annotations

    Upgrading the operator to 2.0 might break client tools that read the SMCP status using the maistra.io/v1 version of the resource.

    This also causes the READY and STATUS columns to be empty when you run oc get servicemeshcontrolplanes.v1.maistra.io.

• MAISTRA-1947 Technology Preview Updates to ServiceMeshExtensions are not applied.

  Workaround: Remove and recreate the ServiceMeshExtensions.

• MAISTRA-1983 Migration to 2.0 Upgrading to 2.0.0 with an existing invalid ServiceMeshControlPlane cannot easily be repaired. The invalid items in the ServiceMeshControlPlane resource caused an unrecoverable error. The fix makes the errors recoverable. You can delete the invalid resource and replace it with a new one or edit the resource to fix the errors. For more information about editing your resource, see [Configuring the Red Hat OpenShift Service Mesh installation].
• MAISTRA-1502 As a result of CVEs fixes in version 1.0.10, the Istio dashboards are not available from the Home Dashboard menu in Grafana. To access the Istio dashboards, click the Dashboard menu in the navigation panel and select the Manage tab.
• MAISTRA-1399 Red Hat OpenShift Service Mesh no longer prevents you from installing unsupported CNI protocols. The supported network configurations has not changed.
• MAISTRA-1089 Migration to 2.0 Gateways created in a non-control plane namespace are automatically deleted. After removing the gateway definition from the SMCP spec, you need to manually delete these resources.

• MAISTRA-858 The following Envoy log messages describing deprecated options and configurations associated with Istio 1.1.x are expected:

  • [2019-06-03 07:03:28.943][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon.
  • [2019-08-12 22:12:59.001][13][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Listener.use_original_dst' from file lds.proto. This configuration will be removed from Envoy soon.

• MAISTRA-806 Evicted Istio Operator Pod causes mesh and CNI not to deploy.

  Workaround: If the istio-operator pod is evicted while deploying the control pane, delete the evicted istio-operator pod.

• MAISTRA-681 When the Service Mesh control plane has many namespaces, it can lead to performance issues.
• MAISTRA-193 Unexpected console info messages are visible when health checking is enabled for citadel.
• Bugzilla 1821432 The toggle controls in OpenShift Container Platform Custom Resource details page does not update the CR correctly. UI Toggle controls in the Service Mesh Control Plane (SMCP) Overview page in the OpenShift Container Platform web console sometimes updates the wrong field in the resource. To update a SMCP, edit the YAML content directly or update the resource from the command line instead of clicking the toggle controls.

• OSSM-1910 Because of an issue introduced in version 2.6, TLS connections could not be established with OpenShift Container Platform Service Mesh. This update resolves the issue by changing the service port names to match conventions used by OpenShift Container Platform Service Mesh and Istio.
• OBSDA-208 Before this update, the default 200m CPU and 256Mi memory resource limits could cause distributed tracing data collection to restart continuously on large clusters. This update resolves the issue by removing these resource limits.
• OBSDA-222 Before this update, spans could be dropped in the OpenShift Container Platform distributed tracing platform. To help prevent this issue from occurring, this release updates version dependencies.

• TRACING-2337 Jaeger is logging a repetitive warning message in the Jaeger logs similar to the following:

  {"level":"warn","ts":1642438880.918793,"caller":"channelz/logging.go:62","msg":"[core]grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"transport: http2Server.HandleStreams received bogus greeting from client: \\\"\\\\x16\\\\x03\\\\x01\\\\x02\\\\x00\\\\x01\\\\x00\\\\x01\\\\xfc\\\\x03\\\\x03vw\\\\x1a\\\\xc9T\\\\xe7\\\\xdaCj\\\\xb7\\\\x8dK\\\\xa6\\\"\"","system":"grpc","grpc_log":true}

  This issue was resolved by exposing only the HTTP(S) port of the query service, and not the gRPC port.

• TRACING-2009 The Jaeger Operator has been updated to include support for the Strimzi Kafka Operator 0.23.0.
• TRACING-1907 The Jaeger agent sidecar injection was failing due to missing config maps in the application namespace. The config maps were getting automatically deleted due to an incorrect OwnerReference field setting and as a result, the application pods were not moving past the "ContainerCreating" stage. The incorrect settings have been removed.
• TRACING-1725 Follow-up to TRACING-1631. Additional fix to ensure that Elasticsearch certificates are properly reconciled when there are multiple Jaeger production instances, using same name but within different namespaces. See also BZ-1918920.
• TRACING-1631 Multiple Jaeger production instances, using same name but within different namespaces, causing Elasticsearch certificate issue. When multiple service meshes were installed, all of the Jaeger Elasticsearch instances had the same Elasticsearch secret instead of individual secrets, which prevented the OpenShift Elasticsearch Operator from communicating with all of the Elasticsearch clusters.
• TRACING-1300 Failed connection between Agent and Collector when using Istio sidecar. An update of the Jaeger Operator enabled TLS communication by default between a Jaeger sidecar agent and the Jaeger Collector.
• TRACING-1208 Authentication "500 Internal Error" when accessing Jaeger UI. When trying to authenticate to the UI using OAuth, I get a 500 error because oauth-proxy sidecar doesn’t trust the custom CA bundle defined at installation time with the additionalTrustBundle.
• TRACING-1166 It is not currently possible to use the Jaeger streaming strategy within a disconnected environment. When a Kafka cluster is being provisioned, it results in a error: Failed to pull image registry.redhat.io/amq7/amq-streams-kafka-24-rhel7@sha256:f9ceca004f1b7dccb3b82d9a8027961f9fe4104e0ed69752c0bdd8078b4a1076.
• TRACING-809 Jaeger Ingester is incompatible with Kafka 2.3. When there are two or more instances of the Jaeger Ingester and enough traffic it will continuously generate rebalancing messages in the logs. This is due to a regression in Kafka 2.3 that was fixed in Kafka 2.3.1. For more information, see Jaegertracing-1819.

• BZ-1918920/LOG-1619 The Elasticsearch pods does not get restarted automatically after an update.

  Workaround: Restart the pods manually.

• Kiali application (back end) – This component runs in the container application platform and communicates with the service mesh components, retrieves and processes data, and exposes this data to the console. The Kiali application does not need storage. When deploying the application to a cluster, configurations are set in ConfigMaps and secrets.
• Kiali console (front end) – The Kiali console is a web application. The Kiali application serves the Kiali console, which then queries the back end for data to present it to the user.

• Red Hat Service Mesh (Istio) - Istio is a Kiali requirement. Istio is the component that provides and controls the service mesh. Although Kiali and Istio can be installed separately, Kiali depends on Istio and will not work if it is not present. Kiali needs to retrieve Istio data and configurations, which are exposed through Prometheus and the cluster API.
• Prometheus - A dedicated Prometheus instance is included as part of the Red Hat OpenShift Service Mesh installation. When Istio telemetry is enabled, metrics data are stored in Prometheus. Kiali uses this Prometheus data to determine the mesh topology, display metrics, calculate health, show possible problems, and so on. Kiali communicates directly with Prometheus and assumes the data schema used by Istio Telemetry. Prometheus is an Istio dependency and a hard dependency for Kiali, and many of Kiali’s features will not work without Prometheus.
• Cluster API - Kiali uses the API of the OpenShift Container Platform (cluster API) to fetch and resolve service mesh configurations. Kiali queries the cluster API to retrieve, for example, definitions for namespaces, services, deployments, pods, and other entities. Kiali also makes queries to resolve relationships between the different cluster entities. The cluster API is also queried to retrieve Istio configurations like virtual services, destination rules, route rules, gateways, quotas, and so on.
• Jaeger - Jaeger is optional, but is installed by default as part of the Red Hat OpenShift Service Mesh installation. When you install the distributed tracing platform as part of the default Red Hat OpenShift Service Mesh installation, the Kiali console includes a tab to display distributed tracing data. Note that tracing data will not be available if you disable Istio’s distributed tracing feature. Also note that user must have access to the namespace where the Service Mesh control plane is installed to view tracing data.
• Grafana - Grafana is optional, but is installed by default as part of the Red Hat OpenShift Service Mesh installation. When available, the metrics pages of Kiali display links to direct the user to the same metric in Grafana. Note that user must have access to the namespace where the Service Mesh control plane is installed to view links to the Grafana dashboard and view Grafana data.

• Health – Quickly identify issues with applications, services, or workloads.
• Topology – Visualize how your applications, services, or workloads communicate via the Kiali graph.
• Metrics – Predefined metrics dashboards let you chart service mesh and application performance for Go, Node.js. Quarkus, Spring Boot, Thorntail and Vert.x. You can also create your own custom dashboards.
• Tracing – Integration with Jaeger lets you follow the path of a request through various microservices that make up an application.
• Validations – Perform advanced validations on the most common Istio objects (Destination Rules, Service Entries, Virtual Services, and so on).
• Configuration – Optional ability to create, update and delete Istio routing configuration using wizards or directly in the YAML editor in the Kiali Console.

• Monitor distributed transactions
• Optimize performance and latency
• Perform root cause analysis

• Red Hat OpenShift distributed tracing platform - This component is based on the open source Jaeger project.
• Red Hat OpenShift distributed tracing data collection - This component is based on the open source OpenTelemetry project.

• Red Hat OpenShift distributed tracing platform - This component is based on the open source Jaeger project.

  • Client (Jaeger client, Tracer, Reporter, instrumented application, client libraries)- The distributed tracing platform clients are language-specific implementations of the OpenTracing API. They can be used to instrument applications for distributed tracing either manually or with a variety of existing open source frameworks, such as Camel (Fuse), Spring Boot (RHOAR), MicroProfile (RHOAR/Thorntail), Wildfly (EAP), and many more, that are already integrated with OpenTracing.
  • Agent (Jaeger agent, Server Queue, Processor Workers) - The distributed tracing platform agent is a network daemon that listens for spans sent over User Datagram Protocol (UDP), which it batches and sends to the Collector. The agent is meant to be placed on the same host as the instrumented application. This is typically accomplished by having a sidecar in container environments such as Kubernetes.
  • Jaeger Collector (Collector, Queue, Workers) - Similar to the Jaeger agent, the Jaeger Collector receives spans and places them in an internal queue for processing. This allows the Jaeger Collector to return immediately to the client/agent instead of waiting for the span to make its way to the storage.
  • Storage (Data Store) - Collectors require a persistent storage backend. Red Hat OpenShift distributed tracing platform has a pluggable mechanism for span storage. Note that for this release, the only supported storage is Elasticsearch.
  • Query (Query Service) - Query is a service that retrieves traces from storage.
  • Ingester (Ingester Service) - Red Hat OpenShift distributed tracing can use Apache Kafka as a buffer between the Collector and the actual Elasticsearch backing storage. Ingester is a service that reads data from Kafka and writes to the Elasticsearch storage backend.
  • Jaeger Console – With the Red Hat OpenShift distributed tracing platform user interface, you can visualize your distributed tracing data. On the Search page, you can find traces and explore details of the spans that make up an individual trace.

• Red Hat OpenShift distributed tracing data collection - This component is based on the open source OpenTelemetry project.

  • OpenTelemetry Collector - The OpenTelemetry Collector is a vendor-agnostic way to receive, process, and export telemetry data. The OpenTelemetry Collector supports open-source observability data formats, for example, Jaeger and Prometheus, sending to one or more open-source or commercial back-ends. The Collector is the default location instrumentation libraries export their telemetry data.

• Integration with Kiali – When properly configured, you can view distributed tracing data from the Kiali console.
• High scalability – The distributed tracing back end is designed to have no single points of failure and to scale with the business needs.
• Distributed Context Propagation – Enables you to connect data from different components together to create a complete end-to-end trace.
• Backwards compatibility with Zipkin – Red Hat OpenShift distributed tracing has APIs that enable it to be used as a drop-in replacement for Zipkin, but Red Hat is not supporting Zipkin compatibility in this release.

• A maistra-version label has been added to all resources.
• All Ingress resources have been converted to OpenShift Route resources.
• Grafana, distributed tracing (Jaeger), and Kiali are enabled by default and exposed through OpenShift routes.
• Godebug has been removed from all templates
• The istio-multi ServiceAccount and ClusterRoleBinding have been removed, as well as the istio-reader ClusterRole.

• spec.gateways.enabled=false to disable both ingress and egress gateways.
• spec.gateways.ingress.enabled=false to disable ingress gateways.
• spec.gateways.egress.enabled=false to disable egress gateways.

• NetworkPolicy: Red Hat OpenShift Service Mesh creates a NetworkPolicy resource in each member project allowing ingress to all pods from the other members and the control plane. If you remove a member from Service Mesh, this NetworkPolicy resource is deleted from the project.

  Note

  This also restricts ingress to only member projects. If you require ingress from non-member projects, you need to create a NetworkPolicy to allow that traffic through.

• Multitenant: Red Hat OpenShift Service Mesh joins the NetNamespace for each member project to the NetNamespace of the control plane project (the equivalent of running oc adm pod-network join-projects --to control-plane-project member-project). If you remove a member from the Service Mesh, its NetNamespace is isolated from the control plane (the equivalent of running oc adm pod-network isolate-projects member-project).
• Subnet: No additional configuration is performed.

• ServiceMeshPolicy replaces MeshPolicy for configuration of control-plane-wide authentication policies. This must be created in the same project as the control plane.
• ServicemeshRbacConfig replaces ClusterRbacConfig for configuration of control-plane-wide role based access control. This must be created in the same project as the control plane.

• Red Hat OpenShift Container Platform version 4.10 or later.
• Red Hat OpenShift Dedicated version 4.
• Azure Red Hat OpenShift (ARO) version 4.
• Red Hat OpenShift Service on AWS (ROSA).

• OpenShift Online is not supported for Red Hat OpenShift Service Mesh.
• Red Hat OpenShift Service Mesh does not support the management of microservices outside the cluster where Service Mesh is running.

• OpenShift-SDN
• OVN-Kubernetes is available on all supported versions of OpenShift Container Platform.
• Third-Party Container Network Interface (CNI) plugins that have been certified on OpenShift Container Platform and passed Service Mesh conformance testing. See Certified OpenShift CNI Plug-ins for more information.

• This release of Red Hat OpenShift Service Mesh is only available on OpenShift Container Platform x86_64, IBM Z, and IBM Power.

  • IBM Z is only supported on OpenShift Container Platform 4.10 and later.
  • IBM Power is only supported on OpenShift Container Platform 4.10 and later.
• Configurations where all Service Mesh components are contained within a single OpenShift Container Platform cluster.
• Configurations that do not integrate external services such as virtual machines.
• Red Hat OpenShift Service Mesh does not support EnvoyFilter configuration except where explicitly documented.

• The Kiali console is only supported on the two most recent releases of the Google Chrome, Microsoft Edge, Mozilla Firefox, or Apple Safari browsers.
• The openshift authentication strategy is the only supported authentication configuration when Kiali is deployed with Red Hat OpenShift Service Mesh (OSSM). The openshift strategy controls access based on the individual’s role-based access control (RBAC) roles of the OpenShift Container Platform.

• Jaeger agent as a sidecar is the only supported configuration for Jaeger. Jaeger as a daemonset is not supported for multitenant installations or OpenShift Dedicated.

• 3scale WebAssembly is the only provided WebAssembly module. You can create custom WebAssembly modules.

Prerequisites

• The Red Hat OpenShift Service Mesh Operator must be installed.
• An account with the cluster-admin role.

Prerequisites

• The Red Hat OpenShift Service Mesh Operator must be installed.
• Access to the OpenShift CLI (oc).

Prerequisites

• You have installed the Red Hat OpenShift Service Mesh Operator.
• You are logged in as a user with the cluster-admin role. If you use Red Hat OpenShift Dedicated, you are logged in as a user with the dedicated-admin role.

Prerequisites

• You have installed the Red Hat OpenShift Service Mesh Operator.
• You are logged in as a user with the cluster-admin role. If you use Red Hat OpenShift Dedicated, you are logged in as a user with the dedicated-admin role.

Prerequisites

• You have installed the Red Hat OpenShift Service Mesh Operator.
• You are logged in as a user with the cluster-admin role. If you use Red Hat OpenShift Dedicated, you are logged in as a user with the dedicated-admin role.

Prerequisites

• You have installed the Red Hat OpenShift Service Mesh Operator.
• You are logged in as a user with the cluster-admin role. If you use Red Hat OpenShift Dedicated, you are logged in as a user with the dedicated-admin role.

Procedure

• Confirm that the nodes associated with Istiod, Ingress Gateway, and Egress Gateway pods are infrastructure nodes:

  $ oc -n istio-system get pods -owide

Prerequisites

• The Red Hat OpenShift Service Mesh Operator is installed.
• You are logged in using an account with the cluster-admin role, or if you use Red Hat OpenShift Dedicated with the dedicated-admin role.

Prerequisites

• The Red Hat OpenShift Service Mesh Operator is installed.
• You have access to the OpenShift CLI (oc).

• ['**'] (all namespaces)
• default
• codeready-*
• openshift-*
• redhat-*

Prerequisites

• An installed, verified Red Hat OpenShift Service Mesh Operator.
• List of existing projects to add to the service mesh.

Prerequisites

• An installed, verified Red Hat OpenShift Service Mesh Operator.
• List of projects to add to the service mesh.
• Access to the OpenShift CLI (oc).

Prerequisites

• An installed, verified Red Hat OpenShift Service Mesh Operator.
• An existing ServiceMeshMemberRoll resource.
• The name of the project with the ServiceMeshMemberRoll resource.
• The names of the projects you want to add or remove from the mesh.

Prerequisites

• An installed, verified Red Hat OpenShift Service Mesh Operator.
• An existing ServiceMeshMemberRoll resource.
• The name of the project with the ServiceMeshMemberRoll resource.
• The names of the projects you want to add or remove from the mesh.
• Access to the OpenShift CLI (oc).

Prerequisites

• You have installed the Red Hat OpenShift Service Mesh Operator.
• You know the name of the ServiceMeshControlPlane resource and the name of the project that the resource belongs to.
• You know the name of the project you want to add to the mesh.
• A service mesh administrator must explicitly grant access to the service mesh. Administrators can grant users permissions to access the mesh by assigning them the mesh-user Role using a RoleBinding or ClusterRoleBinding. For more information, see Creating the Red Hat OpenShift Service Mesh members.

Prerequisites

• You have installed the Red Hat OpenShift Service Mesh Operator.
• You know the name of the ServiceMeshControlPlane resource and the name of the project it belongs to.
• You know the name of the project you want to add to the mesh.
• A service mesh administrator must explicitly grant access to the service mesh. Administrators can grant users permissions to access the mesh by assigning them the mesh-user Role using a RoleBinding or ClusterRoleBinding. For more information, see Creating the Red Hat OpenShift Service Mesh members.

Prerequisites

• You have installed the Red Hat OpenShift Service Mesh Operator.
• The deployment has an existing ServiceMeshMemberRoll resource.
• You are logged in as a user with the cluster-admin role. If you use Red Hat OpenShift Dedicated, you are logged in as a user with the dedicated-admin role.

Prerequisites

• You have installed the Red Hat OpenShift Service Mesh Operator.
• The deployment has an existing ServiceMeshMemberRoll resource.
• You are logged in as a user with the cluster-admin role. If you use Red Hat OpenShift Dedicated, you are logged in as a user with the dedicated-admin role.

Prerequisites:

• OpenShift Container Platform 4.1 or higher installed.
• Red Hat OpenShift Service Mesh 2.4.2 installed.
• Access to the OpenShift CLI (oc).
• An account with the cluster-admin role.

Prerequisites

• Red Hat OpenShift Service Mesh installed.
• Complete the steps for installing the Bookinfo sample app.

Prerequisites

• OpenShift Container Platform 4.1 or higher installed.
• Red Hat OpenShift Service Mesh 2.4.2 installed.
• Access to the OpenShift CLI (oc).

• Patch updates - Patch upgrades are managed by the Operator Lifecycle Manager (OLM); they happen automatically when you update your Operators.
• Minor upgrades - Minor upgrades require both updating to the most recent Red Hat OpenShift Service Mesh Operator version and manually modifying the spec.version value in your ServiceMeshControlPlane resources.
• Major upgrades - Major upgrades require both updating to the most recent Red Hat OpenShift Service Mesh Operator version and manually modifying the spec.version value in your ServiceMeshControlPlane resources. Because major upgrades can contain changes that are not backwards compatible, additional manual changes might be required.

• Operator version - The most current Operator version is 2.4.2. The Operator version number only indicates the version of the currently installed Operator. Because the Red Hat OpenShift Service Mesh Operator supports multiple versions of the Service Mesh control plane, the version of the Operator does not determine the version of your deployed ServiceMeshControlPlane resources.

  Important

  Upgrading to the latest Operator version automatically applies patch updates, but does not automatically upgrade your Service Mesh control plane to the latest minor version.

• ServiceMeshControlPlane version - The ServiceMeshControlPlane version determines what version of Red Hat OpenShift Service Mesh you are using. The value of the spec.version field in the ServiceMeshControlPlane resource controls the architecture and configuration settings that are used to install and deploy Red Hat OpenShift Service Mesh. When you create the Service Mesh control plane you can set the version in one of two ways:

  • To configure in the Form View, select the version from the Control Plane Version menu.
  • To configure in the YAML View, set the value for spec.version in the YAML file.

• Red Hat OpenShift Service Mesh does not support the use of EnvoyFilter configuration except where explicitly documented. This is due to tight coupling with the underlying Envoy APIs, meaning that backward compatibility cannot be maintained. If you are using Envoy Filters, and the configuration generated by Istio has changed due to the lastest version of Envoy introduced by upgrading your ServiceMeshControlPlane, that has the potential to break any EnvoyFilter you may have implemented.
• OSSM-1505 ServiceMeshExtension does not work with OpenShift Container Platform version 4.11. Because ServiceMeshExtension has been deprecated in Red Hat OpenShift Service Mesh 2.2, this known issue will not be fixed and you must migrate your extensions to WasmPluging
• OSSM-1396 If a gateway resource contains the spec.externalIPs setting, rather than being recreated when the ServiceMeshControlPlane is updated, the gateway is removed and never recreated.

• OSSM-1052 When configuring a Service ExternalIP for the ingressgateway in the Service Mesh control plane, the service is not created. The schema for the SMCP is missing the parameter for the service.

  Workaround: Disable the gateway creation in the SMCP spec and manage the gateway deployment entirely manually (including Service, Role and RoleBinding).

• Support for Istio OpenShift Routing (IOR) has been deprecated. IOR functionality is still enabled, but it will be removed in a future release.

• The following cipher suites are no longer supported, and were removed from the list of ciphers used in client and server side TLS negotiations.

  • ECDHE-ECDSA-AES128-SHA
  • ECDHE-RSA-AES128-SHA
  • AES128-GCM-SHA256
  • AES128-SHA
  • ECDHE-ECDSA-AES256-SHA
  • ECDHE-RSA-AES256-SHA
  • AES256-GCM-SHA384

  • AES256-SHA

    Applications that require access to services that use one of these cipher suites will fail to connect when the proxy initiates a TLS connection.

• This release requires use of the WasmPlugin API. Support for the ServiceMeshExtension API, which was deprecated in 2.2, has now been removed. If you attempt to upgrade while using the ServiceMeshExtension API, then the upgrade fails.

• The istio-node DaemonSet is renamed to istio-cni-node to match the name in upstream Istio.
• Istio 1.10 updated Envoy to send traffic to the application container using eth0 rather than lo by default.
• This release adds support for the WasmPlugin API and deprecates the ServiceMeshExtension API.

Behavioral changes

• AuthorizationPolicy updates:

  • With the PROXY protocol, if you’re using ipBlocks and notIpBlocks to specify remote IP addresses, update the configuration to use remoteIpBlocks and notRemoteIpBlocks instead.
  • Added support for nested JSON Web Token (JWT) claims.

• EnvoyFilter breaking changes>

  • Must use typed_config
  • xDS v2 is no longer supported
  • Deprecated filter names
• Older versions of proxies may report 503 status codes when receiving 1xx or 204 status codes from newer proxies.

Prerequisites

• You are running OpenShift Container Platform 4.9 or later.
• You have the latest Red Hat OpenShift Service Mesh Operator.

Prerequisites

• You must upgrade to OpenShift Container Platform 4.7. before you upgrade to Red Hat OpenShift Service Mesh 2.0.
• You must have Red Hat OpenShift Service Mesh version 2.0 operator. If you selected the automatic upgrade path, the operator automatically downloads the latest information. However, there are steps you must take to use the features in Red Hat OpenShift Service Mesh version 2.0.

Prerequisites

• An installed, verified Service Mesh Operator.
• An account with the cluster-admin role. If you use Red Hat OpenShift Dedicated, you must have an account with the dedicated-admin role.
• Location of the Operator deployment.
• Access to the OpenShift CLI (oc).

• Traffic into the service mesh must always go through the ingress-gateway for Istio to work properly.
• Deploy services external to the service mesh in separate namespaces that are not in any service mesh.
• Non-mesh services that need to be deployed within a service mesh enlisted namespace should label their deployments maistra.io/expose-route: "true", which ensures OpenShift Container Platform routes to these services still work.

Prerequisites

• The namespace must be a member of the mesh by defining it in the ServiceMeshMemberRoll or by creating a ServiceMeshMember resource.

Additional resources

• Route-specific annotations.