Chapter 7. Setting up Data Grid services

7.1. Service types

Services are stateful applications, based on the Data Grid Server image, that provide flexible and robust in-memory data storage. When you create Data Grid clusters you specify either DataGrid or Cache as the service type with the spec.service.type field.

DataGrid service type: Deploy Data Grid clusters with full configuration and capabilities.
Cache service type: Deploy Data Grid clusters with minimal configuration.

Red Hat recommends recommends the DataGrid service type for clusters because it lets you:

Back up data across global clusters with cross-site replication.
Create caches with any valid configuration.
Add file-based cache stores to save data in a persistent volume.
Query values across caches using the Data Grid Query API.
Use advanced Data Grid features and capabilities.

Important

The Cache service type was designed to provide a convenient way to create a low-latency data store with minimal configuration. Additional development on the Infinispan CRD has shown that the Cache CR offers a better approach to achieving this goal, ultimately giving users more choice and less deployment overhead. For this reason, the Cache service type is planned for removal in the next version of the Infinispan CRD and is no longer under active development.

The DataGrid service type continues to benefit from new features and improved tooling to automate complex operations such as cluster upgrades and data migration.

7.2. Creating Data Grid service pods

To use custom cache definitions along with Data Grid capabilities such as cross-site replication, create clusters of Data Grid service pods.

Procedure

Create an Infinispan CR that sets spec.service.type: DataGrid and configures any other Data Grid service resources.
```
apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan
spec:
  replicas: 2
  service:
    type: DataGrid
```
Important
You cannot change the spec.service.type field after you create pods. To change the service type, you must delete the existing pods and create new ones.
Apply your Infinispan CR to create the cluster.

7.2.1. Data Grid service CR

This topic describes the Infinispan CR for Data Grid service pods.

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan
  annotations:
    infinispan.org/monitoring: 'true'
spec:
  replicas: 6
  upgrades:
    type: Shutdown
  service:
    type: DataGrid
    container:
      storage: 2Gi
      # The ephemeralStorage and storageClassName fields are mutually exclusive.
      ephemeralStorage: false
      storageClassName: my-storage-class
    sites:
      local:
      name: azure
      expose:
        type: LoadBalancer
      locations:
      - name: azure
        url: openshift://api.azure.host:6443
        secretName: azure-token
      - name: aws
        clusterName: infinispan
        namespace: rhdg-namespace
        url: openshift://api.aws.host:6443
        secretName: aws-token
  security:
    endpointSecretName: endpoint-identities
    endpointEncryption:
        type: Secret
        certSecretName: tls-secret
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "2000m:1000m"
    memory: "2Gi:1Gi"
  logging:
    categories:
      org.infinispan: debug
      org.jgroups: debug
      org.jgroups.protocols.TCP: error
      org.jgroups.protocols.relay.RELAY2: error
  expose:
    type: LoadBalancer
  configMapName: "my-cluster-config"
  configListener:
    enabled: true
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: infinispan-pod
              clusterName: infinispan
              infinispan_cr: infinispan
          topologyKey: "kubernetes.io/hostname"

Field	Description
`metadata.name`	Names your Data Grid cluster.
`metadata.annotations.infinispan.org/monitoring`	Automatically creates a `ServiceMonitor` for your cluster.
`spec.replicas`	Specifies the number of pods in your cluster.
`spec.upgrades.type`	Controls how Data Grid Operator upgrades your Data Grid cluster when new versions become available.
`spec.service.type`	Configures the type Data Grid service. A value of `DataGrid` creates a cluster with Data Grid service pods.
`spec.service.container`	Configures the storage resources for Data Grid service pods.
`spec.service.sites`	Configures cross-site replication.
`spec.security.endpointSecretName`	Specifies an authentication secret that contains Data Grid user credentials.
`spec.security.endpointEncryption`	Specifies TLS certificates and keystores to encrypt client connections.
`spec.container`	Specifies JVM, CPU, and memory resources for Data Grid pods.
`spec.logging`	Configures Data Grid logging categories.
`spec.expose`	Controls how Data Grid endpoints are exposed on the network.
`spec.configMapName`	Specifies a `ConfigMap` that contains Data Grid configuration.
`spec.configListener`	Creates a `listener` pod in each Data Grid cluster that allows Data Grid Operator to reconcile server-side modifications with Data Grid resources such as the `Cache` CR. The `listener` pod consumes minimal resources and is enabled by default. Setting a value of `false` removes the `listener` pod and disables bi-directional reconciliation. You should do this only if you do not need declarative Kubernetes representations of Data Grid resources created through the Data Grid Console, CLI, or client applications.
`spec.affinity`	Configures anti-affinity strategies that guarantee Data Grid availability.

7.3. Allocating storage resources

You can allocate storage for Data Grid service pods but not Cache service pods.

By default, Data Grid Operator allocates 1Gi for the persistent volume claim. However you should adjust the amount of storage available to Data Grid service pods so that Data Grid can preserve cluster state during shutdown.

Important

If available container storage is less than the amount of available memory, data loss can occur.

Procedure

Allocate storage resources with the spec.service.container.storage field.
Configure either the ephemeralStorage field or the storageClassName field as required.
Note
These fields are mutually exclusive. Add only one of them to your Infinispan CR.
Apply the changes.

Ephemeral storage

spec:
  service:
    type: DataGrid
    container:
      storage: 2Gi
      ephemeralStorage: true

Name of a StorageClass object

spec:
  service:
    type: DataGrid
    container:
      storage: 2Gi
      storageClassName: my-storage-class

Field	Description
`spec.service.container.storage`	Specifies the amount of storage for Data Grid service pods.
`spec.service.container.ephemeralStorage`	Defines whether storage is ephemeral or permanent. Set the value to `true` to use ephemeral storage, which means all data in storage is deleted when clusters shut down or restart. The default value is `false`, which means storage is permanent.
`spec.service.container.storageClassName`	Specifies the name of a `StorageClass` object to use for the persistent volume claim (PVC). If you include this field, you must specify an existing storage class as the value. If you do not include this field, the persistent volume claim uses the storage class that has the `storageclass.kubernetes.io/is-default-class` annotation set to `true`.

7.3.1. Persistent volume claims

Data Grid Operator creates a persistent volume claim (PVC) and mounts container storage at:
/opt/infinispan/server/data

Caches

When you create caches, Data Grid permanently stores their configuration so your caches are available after cluster restarts. This applies to both Cache service and Data Grid service pods.

Data

Data is always volatile in clusters of Cache service pods. When you shutdown the cluster, you permanently lose the data.

Use a file-based cache store, by adding the <file-store/> element to your Data Grid cache configuration, if you want Data Grid service pods to persist data during cluster shutdown.

7.4. Allocating CPU and memory

Allocate CPU and memory resources to Data Grid pods with the Infinispan CR.

Note

Data Grid Operator requests 1Gi of memory from the OpenShift scheduler when creating Data Grid pods. CPU requests are unbounded by default.

Procedure

Allocate the number of CPU units with the spec.container.cpu field.
Allocate the amount of memory, in bytes, with the spec.container.memory field.
The cpu and memory fields have values in the format of <limit>:<requests>. For example, cpu: "2000m:1000m" limits pods to a maximum of 2000m of CPU and requests 1000m of CPU for each pod at startup. Specifying a single value sets both the limit and request.
Apply your Infinispan CR.
If your cluster is running, Data Grid Operator restarts the Data Grid pods so changes take effect.

spec:
  container:
    cpu: "2000m:1000m"
    memory: "2Gi:1Gi"

7.5. Setting JVM options

Pass additional JVM options to Data Grid pods at startup.

Procedure

Configure JVM options with the spec.container filed in your Infinispan CR.
Apply your Infinispan CR.
If your cluster is running, Data Grid Operator restarts the Data Grid pods so changes take effect.

JVM options

spec:
  container:
    extraJvmOpts: "-<option>=<value>"
    routerExtraJvmOpts: "-<option>=<value>"
    cliExtraJvmOpts: "-<option>=<value>"

Field	Description
`spec.container.extraJvmOpts`	Specifies additional JVM options for the Data Grid Server.
`spec.container.routerExtraJvmOpts`	Specifies additional JVM options for the Gossip router.
`spec.container.cliExtraJvmOpts`	Specifies additional JVM options for the Data Grid CLI.

7.6. Disabling FIPS mode in your `Infinispan` CR

The Red Hat OpenShift Container Platform can use certain Federal Information Processing Standards (FIPS) components that ensure OpenShift clusters meet the requirements of a FIPS compliance audit. This might cause issues when you want your Data Grid instance to run on any OpenShift cluster that has FIPS mode enabled. Data Grid 8.3 does not support FIPS mode, so you must disable FIPS mode in your Infinispan CR.

After you disable FIPS mode in your Infinispan CR, any component that uses a JVM, such as Data Grid Server, Data Grid CLI, or Gossip router, ignores FIPS mode. This happens, because the JVM no longer loads FIPS-related cryptographic libraries on implementation startup.

Note

You need to explicitly disable FIPS mode in your Infinispan CR configuration only if FIPS mode is enabled for an OpenShift cluster.

Prerequisites

Created Infinispan CR on OpenShift, so that your Data Grid Operator can interact with OpenShift.

Procedure

Configure JVM options with the spec.container field in your Infinispan CR.

spec:
  container:
    extraJvmOpts: "-Dcom.redhat.fips=false"
    cliExtraJvmOpts: "-Dcom.redhat.fips=false"
    routerExtraJvmOpts: "-Dcom.redhat.fips=false"

Apply your Infinispan CR.

Additional resources

7.7. Adjusting log levels

Change levels for different Data Grid logging categories when you need to debug issues. You can also adjust log levels to reduce the number of messages for certain categories to minimize the use of container resources.

Procedure

Configure Data Grid logging with the spec.logging.categories field in your Infinispan CR.

spec:
  logging:
    categories:
      org.infinispan: debug
      org.jgroups: debug

Apply the changes.
Retrieve logs from Data Grid pods as required.
```
oc logs -f $POD_NAME
```

7.7.1. Logging reference

Find information about log categories and levels.

Table 7.1. Log categories
Root category	Description	Default level
`org.infinispan`	Data Grid messages	`info`
`org.jgroups`	Cluster transport messages	`info`

Table 7.2. Log levels
Log level	Description
`trace`	Provides detailed information about running state of applications. This is the most verbose log level.
`debug`	Indicates the progress of individual requests or activities.
`info`	Indicates overall progress of applications, including lifecycle events.
`warn`	Indicates circumstances that can lead to error or degrade performance.
`error`	Indicates error conditions that might prevent operations or activities from being successful but do not prevent applications from running.

Garbage collection (GC) messages

Data Grid Operator does not log GC messages by default. You can direct GC messages to stdout with the following JVM options:

extraJvmOpts: "-Xlog:gc*:stdout:time,level,tags"

7.8. Creating Cache service pods

Create Data Grid clusters with Cache service pods for a volatile, low-latency data store with minimal configuration.

Important

Cache service pods provide volatile storage only, which means you lose all data when you modify your Infinispan CR or update the version of your Data Grid cluster.

Procedure

Create an Infinispan CR that sets spec.service.type: Cache and configures any other Cache service resources.

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan
spec:
  replicas: 2
  service:
    type: Cache

Apply your Infinispan CR to create the cluster.

7.8.1. Cache service CR

This topic describes the Infinispan CR for Cache service pods.

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan
  annotations:
    infinispan.org/monitoring: 'true'
spec:
  replicas: 2
  upgrades:
    type: Shutdown
  service:
    type: Cache
    replicationFactor: 2
  autoscale:
    maxMemUsagePercent: 70
    maxReplicas: 5
    minMemUsagePercent: 30
    minReplicas: 2
  security:
    endpointSecretName: endpoint-identities
    endpointEncryption:
        type: Secret
        certSecretName: tls-secret
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "2000m:1000m"
    memory: "2Gi:1Gi"
  logging:
    categories:
      org.infinispan: trace
      org.jgroups: trace
  expose:
    type: LoadBalancer
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: infinispan-pod
              clusterName: infinispan
              infinispan_cr: infinispan
          topologyKey: "kubernetes.io/hostname"

Field	Description
`metadata.name`	Names your Data Grid cluster.
`metadata.annotations.infinispan.org/monitoring`	Automatically creates a `ServiceMonitor` for your cluster.
`spec.replicas`	Specifies the number of pods in your cluster. If you enable autoscaling capabilities, this field specifies the initial number of pods.
`spec.upgrades.type`	Controls how Data Grid Operator upgrades your Data Grid cluster when new versions become available.
`spec.service.type`	Configures the type Data Grid service. A value of `Cache` creates a cluster with Cache service pods.
`spec.service.replicationFactor`	Sets the number of copies for each entry across the cluster. The default for Cache service pods is two, which replicates each cache entry to avoid data loss.
`spec.autoscale`	Enables and configures automatic scaling.
`spec.security.endpointSecretName`	Specifies an authentication secret that contains Data Grid user credentials.
`spec.security.endpointEncryption`	Specifies TLS certificates and keystores to encrypt client connections.
`spec.container`	Specifies JVM, CPU, and memory resources for Data Grid pods.
`spec.logging`	Configures Data Grid logging categories.
`spec.expose`	Controls how Data Grid endpoints are exposed on the network.
`spec.affinity`	Configures anti-affinity strategies that guarantee Data Grid availability.

7.9. Automatic scaling

Data Grid Operator can monitor the default cache on Cache service pods to automatically scale clusters up or down, by creating or deleting pods based on memory usage.

Important

Automatic scaling is available for clusters of Cache service pods only. Data Grid Operator does not perform automatic scaling for clusters of Data Grid service pods.

When you enable automatic scaling, you define memory usage thresholds that let Data Grid Operator determine when it needs to create or delete pods. Data Grid Operator monitors statistics for the default cache and, when memory usage reaches the configured thresholds, scales your clusters up or down.

Maximum threshold

This threshold sets an upper boundary for the amount of memory that pods in your cluster can use before scaling up or performing eviction. When Data Grid Operator detects that any node reaches the maximum amount of memory that you configure, it creates a new node if possible. If Data Grid Operator cannot create a new node then it performs eviction when memory usage reaches 100 percent.

Minimum threshold

This threshold sets a lower boundary for memory usage across your Data Grid cluster. When Data Grid Operator detects that memory usage falls below the minimum, it shuts down pods.

Default cache only

Autoscaling capabilities work with the default cache only. If you plan to add other caches to your cluster, you should not include the autoscale field in your Infinispan CR. In this case you should use eviction to control the size of the data container on each node.

7.9.1. Configuring automatic scaling

If you create clusters with Cache service pods, you can configure Data Grid Operator to automatically scale clusters.

Procedure

Add the spec.autoscale resource to your Infinispan CR to enable automatic scaling.
Note
Set a value of true for the autoscale.disabled field to disable automatic scaling.

Configure thresholds for automatic scaling with the following fields:

Field	Description
`spec.autoscale.maxMemUsagePercent`	Specifies a maximum threshold, as a percentage, for memory usage on each node.
`spec.autoscale.maxReplicas`	Specifies the maximum number of Cache service pods for the cluster.
`spec.autoscale.minMemUsagePercent`	Specifies a minimum threshold, as a percentage, for cluster memory usage.
`spec.autoscale.minReplicas`	Specifies the minimum number of Cache service pods for the cluster.

For example, add the following to your Infinispan CR:

spec:
  service:
    type: Cache
  autoscale:
    disabled: false
    maxMemUsagePercent: 70
    maxReplicas: 5
    minMemUsagePercent: 30
    minReplicas: 2

Apply the changes.

7.10. Adding labels and annotations to Data Grid resources

Attach key/value labels and annotations to pods and services that Data Grid Operator creates and manages. Labels help you identify relationships between objects to better organize and monitor Data Grid resources. Annotations are arbitrary non-identifying metadata for client applications or deployment and management tooling.

Note

Red Hat subscription labels are automatically applied to Data Grid resources.

Procedure

Open your Infinispan CR for editing.
Attach labels and annotations to Data Grid resources in the metadata.annotations section.
- Define values for annotations directly in the metadata.annotations section.
- Define values for labels with the metadata.labels field.
Apply your Infinispan CR.

Custom annotations

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  annotations:
    infinispan.org/targetAnnotations: service-annotation1, service-annotation2
    infinispan.org/podTargetAnnotations: pod-annotation1, pod-annotation2
    service-annotation1: value
    service-annotation2: value
    pod-annotation1: value
    pod-annotation2: value

Custom labels

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  annotations:
    infinispan.org/targetLabels: service-label1, service-label2
    infinispan.org/podTargetLabels: pod-label1, pod-label2
  labels:
    service-label1: value
    service-label2: value
    pod-label1: value
    pod-label2: value
    # The operator does not attach these labels to resources.
    my-label: my-value
    environment: development

7.11. Adding labels and annotations with environment variables

Set environment variables for Data Grid Operator to add labels and annotations that automatically propagate to all Data Grid pods and services.

Procedure

Add labels and annotations to your Data Grid Operator subscription with the spec.config.env field in one of the following ways:

Use the oc edit subscription command.

oc edit subscription {subscription_name} -n openshift-operators

Use the Red Hat OpenShift Console.
1. Navigate to Operators > Installed Operators > Subscription.
2. Select Edit Subscription from the Actions menu.

Labels and annotations with environment variables

spec:
  config:
    env:
      - name: INFINISPAN_OPERATOR_TARGET_LABELS
        value: |
         {"service-label1":"value",
         service-label1":"value"}
      - name: INFINISPAN_OPERATOR_POD_TARGET_LABELS
        value: |
         {"pod-label1":"value",
         "pod-label2":"value"}
      - name: INFINISPAN_OPERATOR_TARGET_ANNOTATIONS
        value: |
         {"service-annotation1":"value",
         "service-annotation2":"value"}
      - name: INFINISPAN_OPERATOR_POD_TARGET_ANNOTATIONS
        value: |
         {"pod-annotation1":"value",
         "pod-annotation2":"value"}

7.1. Service types

7.2. Creating Data Grid service pods

7.2.1. Data Grid service CR

7.3. Allocating storage resources

7.3.1. Persistent volume claims

7.4. Allocating CPU and memory

7.5. Setting JVM options

7.6. Disabling FIPS mode in your `Infinispan` CR

7.7. Adjusting log levels

7.7.1. Logging reference

7.8. Creating Cache service pods

7.8.1. Cache service CR

7.9. Automatic scaling

7.9.1. Configuring automatic scaling

7.10. Adding labels and annotations to Data Grid resources

7.11. Adding labels and annotations with environment variables

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links

Chapter 7. Setting up Data Grid services

7.1. Service types

7.2. Creating Data Grid service pods

7.2.1. Data Grid service CR

7.3. Allocating storage resources

7.3.1. Persistent volume claims

7.4. Allocating CPU and memory

7.5. Setting JVM options

7.6. Disabling FIPS mode in your Infinispan CR

7.7. Adjusting log levels

7.7.1. Logging reference

7.8. Creating Cache service pods

7.8.1. Cache service CR

7.9. Automatic scaling

7.9.1. Configuring automatic scaling

7.10. Adding labels and annotations to Data Grid resources

7.11. Adding labels and annotations with environment variables

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links

7.6. Disabling FIPS mode in your `Infinispan` CR