1.6. Customizing the Red Hat OpenShift Service Mesh installation

1.6.1. Prerequisites

An account with the cluster-admin role.
Completed the Preparing to install Red Hat OpenShift Service Mesh process.
Have installed the operators.

1.6.2. Red Hat OpenShift Service Mesh custom resources

注意

The istio-system project is used as an example throughout the Service Mesh documentation, but you can use other projects as necessary.

A custom resource allows you to extend the API in an Red Hat OpenShift Service Mesh project or cluster. When you deploy Service Mesh it creates a default ServiceMeshControlPlane that you can modify to change the project parameters.

The Service Mesh operator extends the API by adding the ServiceMeshControlPlane resource type, which enables you to create ServiceMeshControlPlane objects within projects. By creating a ServiceMeshControlPlane object, you instruct the Operator to install a Service Mesh control plane into the project, configured with the parameters you set in the ServiceMeshControlPlane object.

This example ServiceMeshControlPlane definition contains all of the supported parameters and deploys Red Hat OpenShift Service Mesh 1.1.16 images based on Red Hat Enterprise Linux (RHEL).

重要

The 3scale Istio Adapter is deployed and configured in the custom resource file. It also requires a working 3scale account (SaaS or On-Premises).

Example istio-installation.yaml

apiVersion: maistra.io/v1
kind: ServiceMeshControlPlane
metadata:
  name: basic-install
spec:

  istio:
    global:
      proxy:
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 128Mi

    gateways:
      istio-egressgateway:
        autoscaleEnabled: false
      istio-ingressgateway:
        autoscaleEnabled: false
        ior_enabled: false

    mixer:
      policy:
        autoscaleEnabled: false

      telemetry:
        autoscaleEnabled: false
        resources:
          requests:
            cpu: 100m
            memory: 1G
          limits:
            cpu: 500m
            memory: 4G

    pilot:
      autoscaleEnabled: false
      traceSampling: 100

    kiali:
      enabled: true

    grafana:
      enabled: true

    tracing:
      enabled: true
      jaeger:
        template: all-in-one

1.6.3. ServiceMeshControlPlane parameters

The following examples illustrate use of the ServiceMeshControlPlane parameters and the tables provide additional information about supported parameters.

重要

The resources you configure for Red Hat OpenShift Service Mesh with these parameters, including CPUs, memory, and the number of pods, are based on the configuration of your OpenShift cluster. Configure these parameters based on the available resources in your current cluster configuration.

1.6.3.1. Istio global example

Here is an example that illustrates the Istio global parameters for the ServiceMeshControlPlane and a description of the available parameters with appropriate values.

注意

In order for the 3scale Istio Adapter to work, disablePolicyChecks must be false.

Example global parameters

  istio:
    global:
      tag: 1.1.0
      hub: registry.redhat.io/openshift-service-mesh/
      proxy:
        resources:
          requests:
            cpu: 10m
            memory: 128Mi
          limits:
      mtls:
        enabled: false
      disablePolicyChecks: true
      policyCheckFailOpen: false
      imagePullSecrets:
        - MyPullSecret

表 1.3. Global parameters
Parameter	Description	Values	Default value
`disablePolicyChecks`	This parameter enables/disables policy checks.	`true`/`false`	`true`
`policyCheckFailOpen`	This parameter indicates whether traffic is allowed to pass through to the Envoy sidecar when the Mixer policy service cannot be reached.	`true`/`false`	`false`
`tag`	The tag that the Operator uses to pull the Istio images.	A valid container image tag.	`1.1.0`
`hub`	The hub that the Operator uses to pull Istio images.	A valid image repository.	`maistra/` or `registry.redhat.io/openshift-service-mesh/`
`mtls`	This parameter controls whether to enable/disable Mutual Transport Layer Security (mTLS) between services by default.	`true`/`false`	`false`
`imagePullSecrets`	If access to the registry providing the Istio images is secure, list an imagePullSecret here.	redhat-registry-pullsecret OR quay-pullsecret	None

These parameters are specific to the proxy subset of global parameters.

表 1.4. Proxy parameters
Type	Parameter	Description	Values	Default value
Requests	`cpu`	The amount of CPU resources requested for Envoy proxy.	CPU resources, specified in cores or millicores (for example, 200m, 0.5, 1) based on your environment’s configuration.	`10m`
	`memory`	The amount of memory requested for Envoy proxy	Available memory in bytes(for example, 200Ki, 50Mi, 5Gi) based on your environment’s configuration.	`128Mi`
Limits	`cpu`	The maximum amount of CPU resources requested for Envoy proxy.	CPU resources, specified in cores or millicores (for example, 200m, 0.5, 1) based on your environment’s configuration.	`2000m`
	`memory`	The maximum amount of memory Envoy proxy is permitted to use.	Available memory in bytes (for example, 200Ki, 50Mi, 5Gi) based on your environment’s configuration.	`1024Mi`

1.6.3.2. Istio gateway configuration

Here is an example that illustrates the Istio gateway parameters for the ServiceMeshControlPlane and a description of the available parameters with appropriate values.

Example gateway parameters

  gateways:
       istio-egressgateway:
         autoscaleEnabled: false
         autoscaleMin: 1
         autoscaleMax: 5
       istio-ingressgateway:
         autoscaleEnabled: false
         autoscaleMin: 1
         autoscaleMax: 5
         ior_enabled: true

表 1.5. Istio Gateway parameters
Type	Parameter	Description	Values	Default value
`istio-egressgateway`	`autoscaleEnabled`	This parameter enables/disables autoscaling.	`true`/`false`	`true`
	`autoscaleMin`	The minimum number of pods to deploy for the egress gateway based on the `autoscaleEnabled` setting.	A valid number of allocatable pods based on your environment’s configuration.	`1`
	`autoscaleMax`	The maximum number of pods to deploy for the egress gateway based on the `autoscaleEnabled` setting.	A valid number of allocatable pods based on your environment’s configuration.	`5`
`istio-ingressgateway`	`autoscaleEnabled`	This parameter enables/disables autoscaling.	`true`/`false`	`true`
	`autoscaleMin`	The minimum number of pods to deploy for the ingress gateway based on the `autoscaleEnabled` setting.	A valid number of allocatable pods based on your environment’s configuration.	`1`
	`autoscaleMax`	The maximum number of pods to deploy for the ingress gateway based on the `autoscaleEnabled` setting.	A valid number of allocatable pods based on your environment’s configuration.	`5`
	`ior_enabled`	Controls whether Automatic Route Creation is enabled.	`true`/`false`	`false`

1.6.3.3. Automatic route creation

OpenShift routes for Istio Gateways are automatically managed in Red Hat OpenShift Service Mesh. Every time an Istio Gateway is created, updated or deleted inside the service mesh, an OpenShift route is created, updated or deleted.

1.6.3.3.1. Enabling Automatic Route Creation

A Red Hat OpenShift Service Mesh control plane component called Istio OpenShift Routing (IOR) synchronizes the gateway route. Enable IOR as part of the control plane deployment.

If the Gateway contains a TLS section, the OpenShift Route will be configured to support TLS.

In the ServiceMeshControlPlane resource, add the ior_enabled parameter and set it to true. For example, see the following resource snippet:

spec:
  istio:
    gateways:
     istio-egressgateway:
       autoscaleEnabled: false
       autoscaleMin: 1
       autoscaleMax: 5
     istio-ingressgateway:
       autoscaleEnabled: false
       autoscaleMin: 1
       autoscaleMax: 5
       ior_enabled: true

1.6.3.3.2. Subdomains

Red Hat OpenShift Service Mesh creates the route with the subdomain, but OpenShift Container Platform must be configured to enable it. Subdomains, for example *.domain.com, are supported but not by default.

If the following gateway is created:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway1
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - www.bookinfo.com
    - bookinfo.example.com

Then, the following OpenShift Routes are created automatically. You can check that the routes are created with the following command.

$ oc -n <your-control-plane-namespace> get routes

Expected output

NAME           HOST/PORT             PATH  SERVICES               PORT  TERMINATION   WILDCARD
gateway1-lvlfn bookinfo.example.com        istio-ingressgateway   <all>               None
gateway1-scqhv www.bookinfo.com            istio-ingressgateway   <all>               None

If the gateway is deleted, Red Hat OpenShift Service Mesh deletes the routes. However, routes created manually are never modified by Red Hat OpenShift Service Mesh.

Cluster administrators can refer to Using wildcard routes for instructions on how to enable subdomains.

1.6.3.4. Istio Mixer configuration

Here is an example that illustrates the Mixer parameters for the ServiceMeshControlPlane and a description of the available parameters with appropriate values.

Example mixer parameters

mixer:
  enabled: true
  policy:
    autoscaleEnabled: false
  telemetry:
    autoscaleEnabled: false
    resources:
    requests:
      cpu: 10m
      memory: 128Mi
      limits:

表 1.6. Istio Mixer policy parameters
Parameter	Description	Values	Default value
`enabled`	This parameter enables/disables Mixer.	`true`/`false`	`true`
`autoscaleEnabled`	This parameter enables/disables autoscaling. Disable this for small environments.	`true`/`false`	`true`
`autoscaleMin`	The minimum number of pods to deploy based on the `autoscaleEnabled` setting.	A valid number of allocatable pods based on your environment’s configuration.	`1`
`autoscaleMax`	The maximum number of pods to deploy based on the `autoscaleEnabled` setting.	A valid number of allocatable pods based on your environment’s configuration.	`5`

表 1.7. Istio Mixer telemetry parameters
Type	Parameter	Description	Values	Default
Requests	`cpu`	The percentage of CPU resources requested for Mixer telemetry.	CPU resources in millicores based on your environment’s configuration.	`10m`
	`memory`	The amount of memory requested for Mixer telemetry.	Available memory in bytes (for example, 200Ki, 50Mi, 5Gi) based on your environment’s configuration.	`128Mi`
Limits	`cpu`	The maximum percentage of CPU resources Mixer telemetry is permitted to use.	CPU resources in millicores based on your environment’s configuration.	`4800m`
	`memory`	The maximum amount of memory Mixer telemetry is permitted to use.	Available memory in bytes (for example, 200Ki, 50Mi, 5Gi) based on your environment’s configuration.	`4G`

1.6.3.5. Istio Pilot configuration

Here is an example that illustrates the Istio Pilot parameters for the ServiceMeshControlPlane and a description of the available parameters with appropriate values.

Example pilot parameters

  pilot:
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    autoscaleEnabled: false
    traceSampling: 100

表 1.8. Istio Pilot parameters
Parameter	Description	Values	Default value
`cpu`	The percentage of CPU resources requested for Pilot.	CPU resources in millicores based on your environment’s configuration.	`10m`
`memory`	The amount of memory requested for Pilot.	Available memory in bytes (for example, 200Ki, 50Mi, 5Gi) based on your environment’s configuration.	`128Mi`
`autoscaleEnabled`	This parameter enables/disables autoscaling. Disable this for small environments.	`true`/`false`	`true`
`traceSampling`	This value controls how often random sampling occurs. Note: Increase for development or testing.	A valid percentage.	`1.0`

1.6.4. Configuring Kiali

When the Service Mesh Operator creates the ServiceMeshControlPlane it also processes the Kiali resource. The Kiali Operator then uses this object when creating Kiali instances.

The default Kiali parameters specified in the ServiceMeshControlPlane are as follows:

Example Kiali parameters

apiVersion: maistra.io/v1
kind: ServiceMeshControlPlane
spec:
    kiali:
      enabled: true
      dashboard:
        viewOnlyMode: false
      ingress:
        enabled: true

表 1.9. Kiali parameters
Parameter	Description	Values	Default value
enabled	This parameter enables/disables Kiali. Kiali is enabled by default.	`true`/`false`	`true`
dashboard viewOnlyMode	This parameter enables/disables view-only mode for the Kiali console. When view-only mode is enabled, users cannot use the console to make changes to the Service Mesh.	`true`/`false`	`false`
ingress enabled	This parameter enables/disables ingress for Kiali.	`true`/`false`	`true`

1.6.4.1. Configuring Kiali for Grafana

When you install Kiali and Grafana as part of Red Hat OpenShift Service Mesh the Operator configures the following by default:

Grafana is enabled as an external service for Kiali
Grafana authorization for the Kiali console
Grafana URL for the Kiali console

Kiali can automatically detect the Grafana URL. However if you have a custom Grafana installation that is not easily auto-detectable by Kiali, you must update the URL value in the ServiceMeshControlPlane resource.

Additional Grafana parameters

spec:
  kiali:
    enabled: true
    dashboard:
      viewOnlyMode: false
      grafanaURL:  "https://grafana-istio-system.127.0.0.1.nip.io"
    ingress:
      enabled: true

1.6.4.2. Configuring Kiali for Jaeger

When you install Kiali and Jaeger as part of Red Hat OpenShift Service Mesh the Operator configures the following by default:

Jaeger is enabled as an external service for Kiali
Jaeger authorization for the Kiali console
Jaeger URL for the Kiali console

Kiali can automatically detect the Jaeger URL. However if you have a custom Jaeger installation that is not easily auto-detectable by Kiali, you must update the URL value in the ServiceMeshControlPlane resource.

Additional Jaeger parameters

spec:
  kiali:
    enabled: true
    dashboard:
      viewOnlyMode: false
      jaegerURL: "http://jaeger-query-istio-system.127.0.0.1.nip.io"
    ingress:
      enabled: true

1.6.5. Configuring Jaeger

When the Service Mesh Operator creates the ServiceMeshControlPlane resource it also creates the Jaeger resource. The Jaeger Operator then uses this object when creating Jaeger instances.

The default Jaeger parameters specified in the ServiceMeshControlPlane are as follows:

Default all-in-one Jaeger parameters

  apiVersion: maistra.io/v1
  kind: ServiceMeshControlPlane
  spec:
    istio:
      tracing:
        enabled: true
        jaeger:
          template: all-in-one

表 1.10. Jaeger parameters
Parameter	Description	Values	Default value
tracing enabled	This parameter enables/disables tracing in Service Mesh. Jaeger is installed by default.	`true`/`false`	`true`
jaeger template	This parameter specifies which Jaeger deployment strategy to use.	`all-in-one`- For development, testing, demonstrations, and proof of concept. `production-elasticsearch` - For production use.	`all-in-one`

注意

The default template in the ServiceMeshControlPlane resource is the all-in-one deployment strategy which uses in-memory storage. For production, the only supported storage option is Elasticsearch, therefore you must configure the ServiceMeshControlPlane to request the production-elasticsearch template when you deploy Service Mesh within a production environment.

1.6.5.1. Configuring Elasticsearch

The default Jaeger deployment strategy uses the all-in-one template so that the installation can be completed using minimal resources. However, because the all-in-one template uses in-memory storage, it is only recommended for development, demo, or testing purposes and should NOT be used for production environments.

If you are deploying Service Mesh and Jaeger in a production environment you must change the template to the production-elasticsearch template, which uses Elasticsearch for Jaeger’s storage needs.

Elasticsearch is a memory intensive application. The initial set of nodes specified in the default OpenShift Container Platform installation may not be large enough to support the Elasticsearch cluster. You should modify the default Elasticsearch configuration to match your use case and the resources you have requested for your OpenShift Container Platform installation. You can adjust both the CPU and memory limits for each component by modifying the resources block with valid CPU and memory values. Additional nodes must be added to the cluster if you want to run with the recommended amount (or more) of memory. Ensure that you do not exceed the resources requested for your OpenShift Container Platform installation.

Default "production" Jaeger parameters with Elasticsearch

  apiVersion: maistra.io/v1
  kind: ServiceMeshControlPlane
  spec:
    istio:
      tracing:
      enabled: true
      ingress:
        enabled: true
      jaeger:
        template: production-elasticsearch
        elasticsearch:
          nodeCount: 3
          redundancyPolicy:
          resources:
            requests:
              cpu: "1"
              memory: "16Gi"
            limits:
              cpu: "1"
              memory: "16Gi"

表 1.11. Elasticsearch parameters
Parameter	Description	Values	Default Value	Examples
tracing: enabled	This parameter enables/disables tracing in Service Mesh. Jaeger is installed by default.	`true`/`false`	`true`
ingress: enabled	This parameter enables/disables ingress for Jaeger.	`true`/`false`	`true`
jaeger template	This parameter specifies which Jaeger deployment strategy to use.	`all-in-one`/`production-elasticsearch`	`all-in-one`
elasticsearch: nodeCount	Number of Elasticsearch nodes to create.	Integer value.	1	Proof of concept = 1, Minimum deployment =3
requests: cpu	Number of central processing units for requests, based on your environment’s configuration.	Specified in cores or millicores (for example, 200m, 0.5, 1).	1Gi	Proof of concept = 500m, Minimum deployment =1
requests: memory	Available memory for requests, based on your environment’s configuration.	Specified in bytes (for example, 200Ki, 50Mi, 5Gi).	500m	Proof of concept = 1Gi, Minimum deployment = 16Gi*
limits: cpu	Limit on number of central processing units, based on your environment’s configuration.	Specified in cores or millicores (for example, 200m, 0.5, 1).		Proof of concept = 500m, Minimum deployment =1
limits: memory	Available memory limit based on your environment’s configuration.	Specified in bytes (for example, 200Ki, 50Mi, 5Gi).		Proof of concept = 1Gi, Minimum deployment = 16Gi*
	* Each Elasticsearch node can operate with a lower memory setting though this is not recommended for production deployments. For production use, you should have no less than 16Gi allocated to each pod by default, but preferably allocate as much as you can, up to 64Gi per pod.

Procedure

Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.
Navigate to Operators Installed Operators.
Click the Red Hat OpenShift Service Mesh Operator.
Click the Istio Service Mesh Control Plane tab.
Click the name of your control plane file, for example, basic-install.
Click the YAML tab.
Edit the Jaeger parameters, replacing the default all-in-one template with parameters for the production-elasticsearch template, modified for your use case. Ensure that the indentation is correct.
Click Save.
Click Reload. OpenShift Container Platform redeploys Jaeger and creates the Elasticsearch resources based on the specified parameters.

1.6.5.2. Configuring the Elasticsearch index cleaner job

When the Service Mesh Operator creates the ServiceMeshControlPlane it also creates the custom resource (CR) for Jaeger. The Jaeger operator then uses this CR when creating Jaeger instances.

When using Elasticsearch storage, by default a job is created to clean old traces from it. To configure the options for this job, you edit the Jaeger custom resource (CR), to customize it for your use case. The relevant options are listed below.

  apiVersion: jaegertracing.io/v1
  kind: Jaeger
  spec:
    strategy: production
    storage:
      type: elasticsearch
      esIndexCleaner:
        enabled: false
        numberOfDays: 7
        schedule: "55 23 * * *"

表 1.12. Elasticsearch index cleaner parameters
Parameter	Values	Description
enabled	true/ false	Enable or disable the index cleaner job.
numberOfDays	integer value	Number of days to wait before deleting an index.
schedule	"55 23 * * *"	Cron expression for the job to run

For more information about configuring Elasticsearch with OpenShift Container Platform, see Configuring the log store.

1.6.6. 3scale configuration

Here is an example that illustrates the 3scale Istio Adapter parameters for the Red Hat OpenShift Service Mesh custom resource and a description of the available parameters with appropriate values.

Example 3scale parameters

threeScale:
  enabled: false
  PARAM_THREESCALE_LISTEN_ADDR: 3333
  PARAM_THREESCALE_LOG_LEVEL: info
  PARAM_THREESCALE_LOG_JSON: true
  PARAM_THREESCALE_LOG_GRPC: false
  PARAM_THREESCALE_REPORT_METRICS: true
  PARAM_THREESCALE_METRICS_PORT: 8080
  PARAM_THREESCALE_CACHE_TTL_SECONDS: 300
  PARAM_THREESCALE_CACHE_REFRESH_SECONDS: 180
  PARAM_THREESCALE_CACHE_ENTRIES_MAX: 1000
  PARAM_THREESCALE_CACHE_REFRESH_RETRIES: 1
  PARAM_THREESCALE_ALLOW_INSECURE_CONN: false
  PARAM_THREESCALE_CLIENT_TIMEOUT_SECONDS: 10
  PARAM_THREESCALE_GRPC_CONN_MAX_SECONDS: 60

表 1.13. 3scale parameters
Parameter	Description	Values	Default value
`enabled`	Whether to use the 3scale adapter	`true`/`false`	`false`
`PARAM_THREESCALE_LISTEN_ADDR`	Sets the listen address for the gRPC server	Valid port number	`3333`
`PARAM_THREESCALE_LOG_LEVEL`	Sets the minimum log output level.	`debug`, `info`, `warn`, `error`, or `none`	`info`
`PARAM_THREESCALE_LOG_JSON`	Controls whether the log is formatted as JSON	`true`/`false`	`true`
`PARAM_THREESCALE_LOG_GRPC`	Controls whether the log contains gRPC info	`true`/`false`	`true`
`PARAM_THREESCALE_REPORT_METRICS`	Controls whether 3scale system and backend metrics are collected and reported to Prometheus	`true`/`false`	`true`
`PARAM_THREESCALE_METRICS_PORT`	Sets the port that the 3scale `/metrics` endpoint can be scrapped from	Valid port number	`8080`
`PARAM_THREESCALE_CACHE_TTL_SECONDS`	Time period, in seconds, to wait before purging expired items from the cache	Time period in seconds	`300`
`PARAM_THREESCALE_CACHE_REFRESH_SECONDS`	Time period before expiry when cache elements are attempted to be refreshed	Time period in seconds	`180`
`PARAM_THREESCALE_CACHE_ENTRIES_MAX`	Max number of items that can be stored in the cache at any time. Set to `0` to disable caching	Valid number	`1000`
`PARAM_THREESCALE_CACHE_REFRESH_RETRIES`	The number of times unreachable hosts are retried during a cache update loop	Valid number	`1`
`PARAM_THREESCALE_ALLOW_INSECURE_CONN`	Allow to skip certificate verification when calling `3scale` APIs. Enabling this is not recommended.	`true`/`false`	`false`
`PARAM_THREESCALE_CLIENT_TIMEOUT_SECONDS`	Sets the number of seconds to wait before terminating requests to 3scale System and Backend	Time period in seconds	`10`
`PARAM_THREESCALE_GRPC_CONN_MAX_SECONDS`	Sets the maximum amount of seconds (+/-10% jitter) a connection may exist before it is closed	Time period in seconds	60

1.6.7. Next steps

Prepare to deploy applications on Red Hat OpenShift Service Mesh.