Data Grid Operator Guide
Create Data Grid clusters on OpenShift
Abstract
Red Hat Data Grid
Data Grid is a high-performance, distributed in-memory data store.
- Schemaless data structure
- Flexibility to store different objects as key-value pairs.
- Grid-based data storage
- Designed to distribute and replicate data across clusters.
- Elastic scaling
- Dynamically adjust the number of nodes to meet demand without service disruption.
- Data interoperability
- Store, retrieve, and query data in the grid from different endpoints.
Data Grid documentation
Documentation for Data Grid is available on the Red Hat customer portal.
Data Grid downloads
Access the Data Grid Software Downloads on the Red Hat customer portal.
You must have a Red Hat account to access and download Data Grid software.
Making open source more inclusive
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Chapter 1. Installing Data Grid Operator
Install Data Grid Operator into a OpenShift namespace to create and manage Data Grid clusters.
1.1. Installing Data Grid Operator on Red Hat OpenShift
Create subscriptions to Data Grid Operator on OpenShift so you can install different Data Grid versions and receive automatic updates.
Automatic updates apply to Data Grid Operator first and then for each Data Grid node. Data Grid Operator updates clusters one node at a time, gracefully shutting down each node and then bringing it back online with the updated version before going on to the next node.
Prerequisites
- Access to OperatorHub running on OpenShift. Some OpenShift environments, such as OpenShift Container Platform, can require administrator credentials.
- Have an OpenShift project for Data Grid Operator if you plan to install it into a specific namespace.
Procedure
- Log in to the OpenShift Web Console.
- Navigate to OperatorHub.
- Find and select Data Grid Operator.
- Select Install and continue to Create Operator Subscription.
Specify options for your subscription.
- Installation Mode
- You can install Data Grid Operator into a Specific namespace or All namespaces.
- Update Channel
- Get updates for Data Grid Operator 8.2.x.
- Approval Strategies
- Automatically install updates from the 8.2.x channel or require approval before installation.
- Select Subscribe to install Data Grid Operator.
- Navigate to Installed Operators to verify the Data Grid Operator installation.
1.2. Installing Data Grid Operator from the command line
As an alternative to installing Data Grid Operator through the OperatorHub on OpenShift, use the oc
client to create subscriptions.
Prerequisites
-
Have an
oc
client.
Procedure
Set up projects.
- Create a project for Data Grid Operator.
If you want Data Grid Operator to control a specific Data Grid cluster only, create a project for that cluster.
$ oc new-project ${INSTALL_NAMESPACE} 1 $ oc new-project ${WATCH_NAMESPACE} 2
Create an
OperatorGroup
resource.Control all Data Grid clusters
$ oc apply -f - << EOF apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: datagrid namespace: ${INSTALL_NAMESPACE} EOF
Control a specific Data Grid cluster
$ oc apply -f - << EOF apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: datagrid namespace: ${INSTALL_NAMESPACE} spec: targetNamespaces: - ${WATCH_NAMESPACE} EOF
Create a subscription for Data Grid Operator.
$ oc apply -f - << EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: datagrid-operator namespace: ${INSTALL_NAMESPACE} spec: channel: 8.2.x installPlanApproval: Automatic name: datagrid source: redhat-operators sourceNamespace: openshift-marketplace EOF
NoteIf you want to manually approve updates from the 8.2.x channel, change the value of the
spec.installPlanApproval
field toManual
.Verify the installation.
$ oc get pods -n ${INSTALL_NAMESPACE} NAME READY STATUS infinispan-operator-<id> 1/1 Running
1.3. Data Grid cluster upgrades
Data Grid Operator can automatically upgrade Data Grid clusters when new versions become available. You can also perform upgrades manually if you prefer to control when they occur.
Upgrade notifications
If you upgrade Data Grid clusters manually and have upgraded the channel for your Data Grid Operator subscription from 8.1.x to 8.2.x you should apply the upgrade for the latest Data Grid 8.2.x version as soon as possible to avoid potential data loss that can result from an issue in 8.2.0. For more information, see Data Grid Operator 8.2 Release Notes.
Chapter 2. Getting started with Infinispan CR
After you install Data Grid Operator, learn how to create Data Grid clusters on OpenShift.
2.1. Infinispan custom resource (CR)
Data Grid Operator adds a new Custom Resource (CR) of type Infinispan
that lets you handle Data Grid clusters as complex units on OpenShift.
Data Grid Operator watches for Infinispan
Custom Resources (CR) that you use to instantiate and configure Data Grid clusters and manage OpenShift resources, such as StatefulSets and Services. In this way, the Infinispan
CR is your primary interface to Data Grid on OpenShift.
The minimal Infinispan
CR is as follows:
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 2
Field | Description |
---|---|
|
Declares the version of the |
|
Declares the |
| Specifies a name for your Data Grid cluster. |
| Specifies the number of pods in your Data Grid cluster. |
2.2. Creating Data Grid clusters
Use Data Grid Operator to create clusters of two or more Data Grid pods.
Prerequisites
- Install Data Grid Operator.
-
Have an
oc
client.
Procedure
Specify the number of Data Grid pods in the cluster with
spec.replicas
in yourInfinispan
CR.For example, create a
cr_minimal.yaml
file as follows:$ cat > cr_minimal.yaml<<EOF apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 2 EOF
Apply your
Infinispan
CR.$ oc apply -f cr_minimal.yaml
Watch Data Grid Operator create the Data Grid pods.
$ oc get pods -w NAME READY STATUS RESTARTS AGE example-infinispan-1 0/1 ContainerCreating 0 4s example-infinispan-2 0/1 ContainerCreating 0 4s example-infinispan-3 0/1 ContainerCreating 0 5s infinispan-operator-0 1/1 Running 0 3m example-infinispan-3 1/1 Running 0 8s example-infinispan-2 1/1 Running 0 8s example-infinispan-1 1/1 Running 0 8s
Next Steps
Try changing the value of replicas:
and watching Data Grid Operator scale the cluster up or down.
2.3. Verifying Data Grid clusters
Check that Data Grid pods have successfully formed clusters.
Procedure
Retrieve the
Infinispan
CR for Data Grid Operator.$ oc get infinispan -o yaml
The response indicates that Data Grid pods have received clustered views, as in the following example:
conditions: - message: 'View: [example-infinispan-0, example-infinispan-1]' status: "True" type: wellFormed
Do the following for automated scripts:
$ oc wait --for condition=wellFormed --timeout=240s infinispan/example-infinispan
Alternatively, you can retrieve cluster view from logs as follows:
$ oc logs example-infinispan-0 | grep ISPN000094 INFO [org.infinispan.CLUSTER] (MSC service thread 1-2) \ ISPN000094: Received new cluster view for channel infinispan: \ [example-infinispan-0|0] (1) [example-infinispan-0] INFO [org.infinispan.CLUSTER] (jgroups-3,example-infinispan-0) \ ISPN000094: Received new cluster view for channel infinispan: \ [example-infinispan-0|1] (2) [example-infinispan-0, example-infinispan-1]
2.4. Stopping and starting Data Grid clusters
Stop and start Data Grid pods in a graceful, ordered fashion to correctly preserve cluster state.
Clusters of Data Grid service pods must restart with the same number of pods that existed before shutdown. This allows Data Grid to restore the distribution of data across the cluster. After Data Grid Operator fully restarts the cluster you can safely add and remove pods.
Procedure
Change the
spec.replicas
field to0
to stop the Data Grid cluster.spec: replicas: 0
Ensure you have the correct number of pods before you restart the cluster.
$ oc get infinispan example-infinispan -o=jsonpath='{.status.replicasWantedAtRestart}'
Change the
spec.replicas
field to the same number of pods to restart the Data Grid cluster.spec: replicas: 6
Chapter 3. Setting up Data Grid services
Use Data Grid Operator to create clusters of either Cache service or Data Grid service pods.
If you do not specify a value for the spec.service.type
field, Data Grid Operator creates Cache service pods by default.
You cannot change the spec.service.type
field after you create pods. To change the service type, you must delete the existing pods and create new ones.
3.1. Service types
Services are stateful applications, based on the Data Grid Server image, that provide flexible and robust in-memory data storage.
3.1.1. Data Grid service
Deploy clusters of Data Grid service pods if you want to:
- Back up data across global clusters with cross-site replication.
- Create caches with any valid configuration.
- Add file-based cache stores to save data in a persistent volume.
- Query values across caches using the Data Grid Query API.
- Use advanced Data Grid features and capabilities.
3.1.2. Cache service
Deploy clusters of Cache service pods if you want a low-latency data store with minimal configuration.
Cache service pods provide volatile storage only, which means you lose all data when you modify your Infinispan
CR or update the version of your Data Grid cluster. However, if you only want to quickly provide applications with high-performance caching without the overhead of configuration then you can use Cache service pods to:
- Automatically scale to meet capacity when data storage demands go up or down.
- Synchronously distribute data to ensure consistency.
- Replicates each entry in the cache across the cluster.
- Store cache entries off-heap and use eviction for JVM efficiency.
- Ensure data consistency with a default partition handling configuration.
Red Hat recommends that you deploy Data Grid service pods instead of Cache service pods.
Cache service is planned for removal in the next version of the Infinispan
CRD. Data Grid service remains under active development and will continue to benefit from new features and improved tooling to automate complex operations such as upgrading clusters and migrating data.
3.2. Creating Data Grid service pods
To use custom cache definitions along with Data Grid capabilities such as cross-site replication, create clusters of Data Grid service pods.
Procedure
Create an
Infinispan
CR that setsspec.service.type: DataGrid
and configures any other Data Grid service resources.apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 2 service: type: DataGrid
-
Apply your
Infinispan
CR to create the cluster.
3.2.1. Data Grid service CR
This topic describes the Infinispan
CR for Data Grid service pods.
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan annotations: infinispan.org/monitoring: 'true' spec: replicas: 6 service: type: DataGrid container: storage: 2Gi ephemeralStorage: false storageClassName: my-storage-class sites: local: name: azure expose: type: LoadBalancer locations: - name: azure url: openshift://api.azure.host:6443 secretName: azure-token - name: aws clusterName: example-infinispan namespace: rhdg-namespace url: openshift://api.aws.host:6443 secretName: aws-token security: endpointSecretName: endpoint-identities endpointEncryption: type: Secret certSecretName: tls-secret container: extraJvmOpts: "-XX:NativeMemoryTracking=summary" cpu: "1000m" memory: 1Gi logging: categories: org.infinispan: debug org.jgroups: debug org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error expose: type: LoadBalancer affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: infinispan-pod clusterName: example-infinispan infinispan_cr: example-infinispan topologyKey: "kubernetes.io/hostname"
Field | Description |
---|---|
| Names your Data Grid cluster. |
|
Automatically creates a |
| Specifies the number of pods in your cluster. |
|
Configures the type Data Grid service. A value of |
| Configures the storage resources for Data Grid service pods. |
| Configures cross-site replication. |
| Specifies an authentication secret that contains Data Grid user credentials. |
| Specifies TLS certificates and keystores to encrypt client connections. |
| Specifies JVM, CPU, and memory resources for Data Grid pods. |
| Configures Data Grid logging categories. |
| Controls how Data Grid endpoints are exposed on the network. |
| Configures anti-affinity strategies that guarantee Data Grid availability. |
3.3. Creating Cache service pods
Create Data Grid clusters with Cache service pods for a volatile, low-latency data store with minimal configuration.
Procedure
Create an
Infinispan
CR that setsspec.service.type: Cache
and configures any other Cache service resources.apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 2 service: type: Cache
-
Apply your
Infinispan
CR to create the cluster.
3.3.1. Cache service CR
This topic describes the Infinispan
CR for Cache service pods.
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan annotations: infinispan.org/monitoring: 'true' spec: replicas: 2 service: type: Cache replicationFactor: 2 autoscale: maxMemUsagePercent: 70 maxReplicas: 5 minMemUsagePercent: 30 minReplicas: 2 security: endpointSecretName: endpoint-identities endpointEncryption: type: Secret certSecretName: tls-secret container: extraJvmOpts: "-XX:NativeMemoryTracking=summary" cpu: "2000m" memory: 1Gi logging: categories: org.infinispan: trace org.jgroups: trace expose: type: LoadBalancer affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: infinispan-pod clusterName: example-infinispan infinispan_cr: example-infinispan topologyKey: "kubernetes.io/hostname"
Field | Description |
---|---|
| Names your Data Grid cluster. |
|
Automatically creates a |
| Specifies the number of pods in your cluster. If you enable autoscaling capabilities, this field specifies the initial number of pods. |
|
Configures the type Data Grid service. A value of |
| Sets the number of copies for each entry across the cluster. The default for Cache service pods is two, which replicates each cache entry to avoid data loss. |
| Enables and configures automatic scaling. |
| Specifies an authentication secret that contains Data Grid user credentials. |
| Specifies TLS certificates and keystores to encrypt client connections. |
| Specifies JVM, CPU, and memory resources for Data Grid pods. |
| Configures Data Grid logging categories. |
| Controls how Data Grid endpoints are exposed on the network. |
| Configures anti-affinity strategies that guarantee Data Grid availability. |
3.4. Automatic scaling
Data Grid Operator can monitor the default cache on Cache service pods to automatically scale clusters up or down, by creating or deleting pods based on memory usage.
Automatic scaling is available for clusters of Cache service pods only. Data Grid Operator does not perform automatic scaling for clusters of Data Grid service pods.
When you enable automatic scaling, you define memory usage thresholds that let Data Grid Operator determine when it needs to create or delete pods. Data Grid Operator monitors statistics for the default cache and, when memory usage reaches the configured thresholds, scales your clusters up or down.
Maximum threshold
This threshold sets an upper boundary for the amount of memory that pods in your cluster can use before scaling up or performing eviction. When Data Grid Operator detects that any node reaches the maximum amount of memory that you configure, it creates a new node if possible. If Data Grid Operator cannot create a new node then it performs eviction when memory usage reaches 100 percent.
Minimum threshold
This threshold sets a lower boundary for memory usage across your Data Grid cluster. When Data Grid Operator detects that memory usage falls below the minimum, it shuts down pods.
Default cache only
Autoscaling capabilities work with the default cache only. If you plan to add other caches to your cluster, you should not include the autoscale
field in your Infinispan
CR. In this case you should use eviction to control the size of the data container on each node.
3.4.1. Configuring automatic scaling
If you create clusters with Cache service pods, you can configure Data Grid Operator to automatically scale clusters.
Procedure
Add the
spec.autoscale
resource to yourInfinispan
CR to enable automatic scaling.NoteSet a value of
true
for theautoscale.disabled
field to disable automatic scaling.Configure thresholds for automatic scaling with the following fields:
Field Description spec.autoscale.maxMemUsagePercent
Specifies a maximum threshold, as a percentage, for memory usage on each node.
spec.autoscale.maxReplicas
Specifies the maximum number of Cache service pods for the cluster.
spec.autoscale.minMemUsagePercent
Specifies a minimum threshold, as a percentage, for cluster memory usage.
spec.autoscale.minReplicas
Specifies the minimum number of Cache service pods for the cluster.
For example, add the following to your
Infinispan
CR:spec: service: type: Cache autoscale: disabled: false maxMemUsagePercent: 70 maxReplicas: 5 minMemUsagePercent: 30 minReplicas: 2
- Apply the changes.
3.5. Allocating storage resources
You can allocate storage for Data Grid service pods but not Cache service pods.
By default, Data Grid Operator allocates 1Gi
for the persistent volume claim. However you should adjust the amount of storage available to Data Grid service pods so that Data Grid can preserve cluster state during shutdown.
If available container storage is less than the amount of available memory, data loss can occur.
Procedure
-
Allocate storage resources with the
spec.service.container.storage
field. Optionally configure the
ephemeralStorage
andstorageClassName
fields as required.spec: service: type: DataGrid container: storage: 2Gi ephemeralStorage: false storageClassName: my-storage-class
- Apply the changes.
Field | Description |
---|---|
| Specifies the amount of storage for Data Grid service pods. |
|
Defines whether storage is ephemeral or permanent. Set the value to |
|
Specifies the name of a |
3.5.1. Persistent volume claims
Data Grid Operator creates a persistent volume claim (PVC) and mounts container storage at:/opt/infinispan/server/data
Caches
When you create caches, Data Grid permanently stores their configuration so your caches are available after cluster restarts. This applies to both Cache service and Data Grid service pods.
Data
Data is always volatile in clusters of Cache service pods. When you shutdown the cluster, you permanently lose the data.
Use a file-based cache store, by adding the <file-store/>
element to your Data Grid cache configuration, if you want Data Grid service pods to persist data during cluster shutdown.
3.6. JVM, CPU, and memory
You can set JVM options in Infinispan
CR as well as CPU and memory allocation.
spec: container: extraJvmOpts: "-XX:NativeMemoryTracking=summary" cpu: "1000m" memory: 1Gi
Field | Description |
---|---|
| Specifies JVM options. |
| Allocates host CPU resources to Data Grid pods, measured in CPU units. |
| Allocates host memory to Data Grid pods, measured in bytes. |
When Data Grid Operator creates Data Grid clusters, it uses spec.container.cpu
and spec.container.memory
to:
-
Ensure that OpenShift has sufficient capacity to run the Data Grid node. By default Data Grid Operator requests 512Mi of
memory
and 0.5cpu
from the OpenShift scheduler. -
Constrain node resource usage. Data Grid Operator sets the values of
cpu
andmemory
as resource limits.
3.7. Adjusting log levels
Change levels for different Data Grid logging categories when you need to debug issues. You can also adjust log levels to reduce the number of messages for certain categories to minimize the use of container resources.
Procedure
Configure Data Grid logging with the
spec.logging.categories
field in yourInfinispan
CR.spec: logging: categories: org.infinispan: debug org.jgroups: debug
- Apply the changes.
Retrieve logs from Data Grid pods as required.
$ oc logs -f $POD_NAME
3.7.1. Logging reference
Find information about log categories and levels.
Root category | Description | Default level |
---|---|---|
| Data Grid messages |
|
| Cluster transport messages |
|
Log level | Description |
---|---|
| Provides detailed information about running state of applications. This is the most verbose log level. |
| Indicates the progress of individual requests or activities. |
| Indicates overall progress of applications, including lifecycle events. |
| Indicates circumstances that can lead to error or degrade performance. |
| Indicates error conditions that might prevent operations or activities from being successful but do not prevent applications from running. |
Garbage collection (GC) messages
Data Grid Operator does not log GC messages by default. You can direct GC messages to stdout
with the following JVM options:
extraJvmOpts: "-Xlog:gc*:stdout:time,level,tags"
3.8. Adding labels to Data Grid resources
Attach key/value labels to pods and services that Data Grid Operator creates and manages. These labels help you identify relationships between objects to better organize and monitor Data Grid resources.
Red Hat subscription labels are automatically applied to Data Grid pods.
Procedure
-
Open your
Infinispan
CR for editing. -
Add any labels that you want Data Grid Operator to attach to resources with
metadata.annotations
. Add values for your labels with
metadata.labels
.-
Specify labels that you want to attach to services with the
metadata.annotations.infinispan.org/targetLabels
field. -
Specify labels that you want to attach to pods with the
metadata.annotations.infinispan.org/podTargetLabels
field. Define values for your labels with the
metadata.labels
fields.apiVersion: infinispan.org/v1 kind: Infinispan metadata: annotations: infinispan.org/targetLabels: svc-label1, svc-label2 infinispan.org/podTargetLabels: pod-label1, pod-label2 labels: svc-label1: svc-value1 svc-label2: svc-value2 pod-label1: pod-value1 pod-label2: pod-value2 # The operator does not attach these labels to resources. my-label: my-value environment: development
-
Specify labels that you want to attach to services with the
-
Apply your
Infinispan
CR.
Additional resources
Chapter 4. Configuring authentication
Application users need credentials to access Data Grid clusters. You can use default, generated credentials or add your own.
4.1. Default credentials
Data Grid Operator generates base64-encoded default credentials stored in an authentication secrets named
Username | Secret name | Description |
---|---|---|
|
| Credentials for the default application user. |
|
| Credentials that Data Grid Operator uses to interact with Data Grid resources. |
4.2. Retrieving credentials
Get credentials from authentication secrets to access Data Grid clusters.
Procedure
Retrieve credentials from authentication secrets.
$ oc get secret example-infinispan-generated-secret
Base64-decode credentials.
$ oc get secret example-infinispan-generated-secret \ -o jsonpath="{.data.identities\.yaml}" | base64 --decode credentials: - username: developer password: dIRs5cAAsHIeeRIL
4.3. Adding custom user credentials
Configure access to Data Grid cluster endpoints with custom credentials.
Modifying spec.security.endpointSecretName
triggers a cluster restart.
Procedure
Create an
identities.yaml
file with the credentials that you want to add.credentials: - username: myfirstusername password: changeme-one - username: mysecondusername password: changeme-two
Create an authentication secret from
identities.yaml
.$ oc create secret generic --from-file=identities.yaml connect-secret
Specify the authentication secret with
spec.security.endpointSecretName
in yourInfinispan
CR and then apply the changes.spec: security: endpointSecretName: connect-secret
4.4. Changing the operator password
You can change the password for the operator
user if you do not want to use the automatically generated password.
Procedure
Update the
password
key in theexample-infinispan-generated-operator-secret
secret as follows:oc patch secret example-infinispan-generated-operator-secret -p='{"stringData":{"password": "supersecretoperatorpassword"}}'
NoteYou should update only the
password
key in thegenerated-operator-secret
secret. When you update the password, Data Grid Operator automatically refreshes other keys in that secret.
4.5. Disabling user authentication
Allow users to access Data Grid clusters and manipulate data without providing credentials.
Do not disable authentication if endpoints are accessible from outside the OpenShift cluster via spec.expose.type
. You should disable authentication for development environments only.
Procedure
Set
false
as the value for thespec.security.endpointAuthentication
field in yourInfinispan
CR.spec: security: endpointAuthentication: false
- Apply the changes.
Chapter 5. Configuring client certificate authentication
Add client trust stores to your project and configure Data Grid to allow connections only from clients that present valid certificates. This increases security of your deployment by ensuring that clients are trusted by a public certificate authority (CA).
5.1. Client certificate authentication
Client certificate authentication restricts in-bound connections based on the certificates that clients present.
You can configure Data Grid to use trust stores with either of the following strategies:
Validate
To validate client certificates, Data Grid requires a trust store that contains any part of the certificate chain for the signing authority, typically the root CA certificate. Any client that presents a certificate signed by the CA can connect to Data Grid.
If you use the Validate
strategy for verifying client certificates, you must also configure clients to provide valid Data Grid credentials if you enable authentication.
Authenticate
Requires a trust store that contains all public client certificates in addition to the root CA certificate. Only clients that present a signed certificate can connect to Data Grid.
If you use the Authenticate
strategy for verifying client certificates, you must ensure that certificates contain valid Data Grid credentials as part of the distinguished name (DN).
5.2. Enabling client certificate authentication
To enable client certificate authentication, you configure Data Grid to use trust stores with either the Validate
or Authenticate
strategy.
Procedure
Set either
Validate
orAuthenticate
as the value for thespec.security.endpointEncryption.clientCert
field in yourInfinispan
CR.NoteThe default value is
None
.Specify the secret that contains the client trust store with the
spec.security.endpointEncryption.clientCertSecretName
field.By default Data Grid Operator expects a trust store secret named
<cluster-name>-client-cert-secret
.NoteThe secret must be unique to each
Infinispan
CR instance in the OpenShift cluster. When you delete theInfinispan
CR, OpenShift also automatically deletes the associated secret.spec: security: endpointEncryption: type: Service certSecretName: tls-secret clientCert: Validate clientCertSecretName: example-infinispan-client-cert-secret
- Apply the changes.
Next steps
Provide Data Grid Operator with a trust store that contains all client certificates. Alternatively you can provide certificates in PEM format and let Data Grid generate a client trust store.
5.3. Providing client truststores
If you have a trust store that contains the required certificates you can make it available to Data Grid Operator.
Data Grid supports trust stores in PKCS12
format only.
Procedure
Specify the name of the secret that contains the client trust store as the value of the
metadata.name
field.NoteThe name must match the value of the
spec.security.endpointEncryption.clientCertSecretName
field.-
Provide the password for the trust store with the
stringData.truststore-password
field. Specify the trust store with the
data.truststore.p12
field.apiVersion: v1 kind: Secret metadata: name: example-infinispan-client-cert-secret type: Opaque stringData: truststore-password: changme data: truststore.p12: "<base64_encoded_PKCS12_trust_store>"
- Apply the changes.
5.4. Providing client certificates
Data Grid Operator can generate a trust store from certificates in PEM format.
Procedure
Specify the name of the secret that contains the client trust store as the value of the
metadata.name
field.NoteThe name must match the value of the
spec.security.endpointEncryption.clientCertSecretName
field.-
Specify the signing certificate, or CA certificate bundle, as the value of the
data.trust.ca
field. If you use the
Authenticate
strategy to verify client identities, add the certificate for each client that can connect to Data Grid endpoints with thedata.trust.cert.<name>
field.NoteData Grid Operator uses the
<name>
value as the alias for the certificate when it generates the trust store.Optionally provide a password for the trust store with the
stringData.truststore-password
field.If you do not provide one, Data Grid Operator sets "password" as the trust store password.
apiVersion: v1 kind: Secret metadata: name: example-infinispan-client-cert-secret type: Opaque stringData: truststore-password: changme data: trust.ca: "<base64_encoded_CA_certificate>" trust.cert.client1: "<base64_encoded_client_certificate>" trust.cert.client2: "<base64_encoded_client_certificate>"
- Apply the changes.
Chapter 6. Configuring encryption
Encrypt connections between clients and Data Grid pods with Red Hat OpenShift service certificates or custom TLS certificates.
6.1. Encryption with Red Hat OpenShift service certificates
Data Grid Operator automatically generates TLS certificates that are signed by the Red Hat OpenShift service CA. Data Grid Operator then stores the certificates and keys in a secret so you can retrieve them and use with remote clients.
If the Red Hat OpenShift service CA is available, Data Grid Operator adds the following spec.security.endpointEncryption
configuration to the Infinispan
CR:
spec: security: endpointEncryption: type: Service certServiceName: service.beta.openshift.io certSecretName: example-infinispan-cert-secret
Field | Description |
---|---|
| Specifies the service that provides TLS certificates. |
|
Specifies a secret with a service certificate and key in PEM format. Defaults to |
Service certificates use the internal DNS name of the Data Grid cluster as the common name (CN), for example:
Subject: CN = example-infinispan.mynamespace.svc
For this reason, service certificates can be fully trusted only inside OpenShift. If you want to encrypt connections with clients running outside OpenShift, you should use custom TLS certificates.
Service certificates are valid for one year and are automatically replaced before they expire.
6.2. Retrieving TLS certificates
Get TLS certificates from encryption secrets to create client trust stores.
Procedure
Retrieve
tls.crt
from encryption secrets as follows:$ oc get secret example-infinispan-cert-secret \ -o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.crt
6.3. Disabling encryption
You can disable encryption so clients do not need TLS certificates to establish connections with Data Grid.
Do not disable encryption if endpoints are accessible from outside the OpenShift cluster via spec.expose.type
. You should disable encryption for development environments only.
Procedure
Set
None
as the value for thespec.security.endpointEncryption.type
field in yourInfinispan
CR.spec: security: endpointEncryption: type: None
- Apply the changes.
6.4. Using custom TLS certificates
Use custom PKCS12 keystore or TLS certificate/key pairs to encrypt connections between clients and Data Grid clusters.
Prerequisites
Create either a keystore or certificate secret.
NoteThe secret must be unique to each
Infinispan
CR instance in the OpenShift cluster. When you delete theInfinispan
CR, OpenShift also automatically deletes the associated secret.
Procedure
Add the encryption secret to your OpenShift namespace, for example:
$ oc apply -f tls_secret.yaml
Specify the encryption secret with the
spec.security.endpointEncryption.certSecretName
field in yourInfinispan
CR.spec: security: endpointEncryption: type: Secret certSecretName: tls-secret
- Apply the changes.
6.4.1. Custom encryption secrets
This topic describes resources for custom encryption secrets.
Keystore secrets
apiVersion: v1 kind: Secret metadata: name: tls-secret type: Opaque stringData: alias: server password: changeme data: keystore.p12: "MIIKDgIBAzCCCdQGCSqGSIb3DQEHA..."
Field | Description |
---|---|
| Specifies an alias for the keystore. |
| Specifies the keystore password. |
| Adds a base64-encoded keystore. |
Certificate secrets
apiVersion: v1 kind: Secret metadata: name: tls-secret type: Opaque data: tls.key: "LS0tLS1CRUdJTiBQUk ..." tls.crt: "LS0tLS1CRUdJTiBDRVl ..."
Field | Description |
---|---|
| Adds a base64-encoded TLS key. |
| Adds a base64-encoded TLS certificate. |
Chapter 7. Configuring user roles and permissions
Secure access to Data Grid services by configuring role-based access control (RBAC) for users. This requires you to assign roles to users so that they have permission to access caches and Data Grid resources.
7.1. Enabling security authorization
By default authorization is disabled to ensure backwards compatibility with Infinispan
CR instances. Complete the following procedure to enable authorization and use role-based access control (RBAC) for Data Grid users.
Procedure
Set
true
as the value for thespec.security.authorization.enabled
field in yourInfinispan
CR.spec: security: authorization: enabled: true
- Apply the changes.
7.2. User roles and permissions
Data Grid Operator provides a set of default roles that are associated with different permissions.
Role | Permissions | Description |
---|---|---|
| ALL | Superuser with all permissions including control of the Cache Manager lifecycle. |
| ALL_READ, ALL_WRITE, LISTEN, EXEC, MONITOR, CREATE |
Can create and delete Data Grid resources in addition to |
| ALL_READ, ALL_WRITE, LISTEN, EXEC, MONITOR |
Has read and write access to Data Grid resources in addition to |
| ALL_READ, MONITOR |
Has read access to Data Grid resources in addition to |
| MONITOR | Can view statistics for Data Grid clusters. |
Data Grid Operator credentials
Data Grid Operator generates credentials that it uses to authenticate with Data Grid clusters to perform internal operations. By default Data Grid Operator credentials are automatically assigned the admin
role when you enable security authorization.
Additional resources
- How security authorization works (Data Grid Security Guide).
7.3. Assigning roles and permissions to users
Assign users with roles that control whether users are authorized to access Data Grid cluster resources. Roles can have different permission levels, from read-only to unrestricted access.
Users gain authorization implicitly. For example, "admin" gets admin
permissions automatically. A user named "deployer" has the deployer
role automatically, and so on.
Procedure
Create an
identities.yaml
file that assigns roles to users.credentials: - username: admin password: changeme - username: my-user-1 password: changeme roles: - admin - username: my-user-2 password: changeme roles: - monitor
Create an authentication secret from
identities.yaml
.If necessary, delete the existing secret first.
$ oc delete secret connect-secret --ignore-not-found $ oc create secret generic --from-file=identities.yaml connect-secret
Specify the authentication secret with
spec.security.endpointSecretName
in yourInfinispan
CR and then apply the changes.spec: security: endpointSecretName: connect-secret
7.4. Adding custom roles and permissions
You can define custom roles with different combinations of permissions.
Procedure
-
Open your
Infinispan
CR for editing. Specify custom roles and their associated permissions with the
spec.security.authorization.roles
field.spec: security: authorization: enabled: true roles: - name: my-role-1 permissions: - ALL - name: my-role-2 permissions: - READ - WRITE
- Apply the changes.
Chapter 8. Configuring network access to Data Grid
Expose Data Grid clusters so you can access Data Grid Console, the Data Grid command line interface (CLI), REST API, and Hot Rod endpoint.
8.1. Getting the service for internal connections
By default, Data Grid Operator creates a service that provides access to Data Grid clusters from clients running on OpenShift.
This internal service has the same name as your Data Grid cluster, for example:
metadata: name: example-infinispan
Procedure
Check that the internal service is available as follows:
$ oc get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) example-infinispan ClusterIP 192.0.2.0 <none> 11222/TCP
8.2. Exposing Data Grid through load balancers
Use a load balancer service to make Data Grid clusters available to clients running outside OpenShift.
To access Data Grid with unencrypted Hot Rod client connections you must use a load balancer service.
Procedure
-
Include
spec.expose
in yourInfinispan
CR. -
Specify
LoadBalancer
as the service type with thespec.expose.type
field. Optionally specify the network port where the service is exposed with the
spec.expose.port
field. The default port is7900
.spec: expose: type: LoadBalancer port: 65535
- Apply the changes.
Verify that the
-external
service is available.$ oc get services | grep external NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) example-infinispan-external LoadBalancer 192.0.2.24 hostname.com 11222/TCP
8.3. Exposing Data Grid through node ports
Use a node port service to expose Data Grid clusters on the network.
Procedure
-
Include
spec.expose
in yourInfinispan
CR. -
Specify
NodePort
as the service type with thespec.expose.type
field. Configure the port where Data Grid is exposed with the
spec.expose.nodePort
field.spec: expose: type: NodePort nodePort: 30000
- Apply the changes.
Verify that the
-external
service is available.$ oc get services | grep external NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) example-infinispan-external NodePort 192.0.2.24 <none> 11222:30000/TCP
8.4. Exposing Data Grid through routes
Use an OpenShift Route with passthrough encryption to make Data Grid clusters available on the network.
Procedure
-
Include
spec.expose
in yourInfinispan
CR. -
Specify
Route
as the service type with thespec.expose.type
field. Optionally add a hostname with the
spec.expose.host
field.spec: expose: type: Route host: www.example.org
- Apply the changes.
Verify that the route is available.
$ oc get routes NAME CLASS HOSTS ADDRESS PORTS AGE example-infinispan <none> * 443 73s
Route ports
When you create a route, it exposes a port on the network that accepts client connections and redirects traffic to Data Grid services that listen on port 11222
.
The port where the route is available depends on whether you use encryption or not.
Port | Description |
---|---|
| Encryption is disabled. |
| Encryption is enabled. |
8.5. Network services
Reference information for network services that Data Grid Operator creates and manages.
8.5.1. Internal service
- Allow Data Grid pods to discover each other and form clusters.
- Provide access to Data Grid endpoints from clients in the same OpenShift namespace.
Service | Port | Protocol | Description |
---|---|---|---|
|
| TCP | Internal access to Data Grid endpoints |
|
| TCP | Cluster discovery |
8.5.2. External service
Provides access to Data Grid endpoints from clients outside OpenShift or in different namespaces.
You must create the external service with Data Grid Operator. It is not available by default.
Service | Port | Protocol | Description |
---|---|---|---|
|
| TCP | External access to Data Grid endpoints. |
8.5.3. Cross-site service
Allows Data Grid to back up data between clusters in different locations.
Service | Port | Protocol | Description |
---|---|---|---|
|
| TCP | JGroups RELAY2 channel for cross-site communication. |
Chapter 9. Monitoring Data Grid services
Data Grid exposes metrics that can be used by Prometheus and Grafana for monitoring and visualizing the cluster state.
This documentation explains how to set up monitoring on OpenShift Container Platform. If you’re working with community Prometheus deployments, you might find these instructions useful as a general guide. However you should refer to the Prometheus documentation for installation and usage instructions.
See the Prometheus Operator documentation.
9.1. Creating a Prometheus service monitor
Data Grid Operator automatically creates a Prometheus ServiceMonitor
that scrapes metrics from your Data Grid cluster.
Procedure
Enable monitoring for user-defined projects on OpenShift Container Platform.
When the Operator detects an Infinispan
CR with the monitoring annotation set to true
, which is the default, Data Grid Operator does the following:
-
Creates a
ServiceMonitor
named<cluster_name>-monitor
. Adds the
infinispan.org/monitoring: 'true'
annotation to yourInfinispan
CR metadata, if the value is not already explicitly set:apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan annotations: infinispan.org/monitoring: 'true'
To authenticate with Data Grid, Prometheus uses the operator
credentials.
Verification
You can check that Prometheus is scraping Data Grid metrics as follows:
- In the OpenShift Web Console, select the </> Developer perspective and then select Monitoring.
- Open the Dashboard tab for the namespace where your Data Grid cluster runs.
Open the Metrics tab and confirm that you can query Data Grid metrics such as:
vendor_cache_manager_default_cluster_size
Additional resources
9.1.1. Disabling the Prometheus service monitor
You can disable the ServiceMonitor
if you do not want Prometheus to scrape metrics for your Data Grid cluster.
Procedure
Set
'false'
as the value for theinfinispan.org/monitoring
annotation in yourInfinispan
CR.apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan annotations: infinispan.org/monitoring: 'false'
- Apply the changes.
9.2. Installing the Grafana Operator
To support various needs, Data Grid Operator integrates with the community version of the Grafana Operator to create dashboards for Data Grid services.
Until Grafana is integrated with OpenShift user workload monitoring, the only option is to rely on the community version. You can install the Grafana Operator on OpenShift from the OperatorHub and should create a subscription for the alpha channel.
However, as is the policy for all Community Operators, Red Hat does not certify the Grafana Operator and does not provide support for it in combination with Data Grid. When you install the Grafana Operator you are prompted to acknowledge a warning about the community version before you can continue.
9.3. Creating Grafana data sources
Create a GrafanaDatasource
CR so you can visualize Data Grid metrics in Grafana dashboards.
Prerequisites
-
Have an
oc
client. -
Have
cluster-admin
access to OpenShift Container Platform. - Enable monitoring for user-defined projects on OpenShift Container Platform.
-
Install the Grafana Operator from the alpha channel and create a
Grafana
CR.
Procedure
Create a
ServiceAccount
that lets Grafana read Data Grid metrics from Prometheus.apiVersion: v1 kind: ServiceAccount metadata: name: infinispan-monitoring
Apply the
ServiceAccount
.$ oc apply -f service-account.yaml
Grant
cluster-monitoring-view
permissions to theServiceAccount
.$ oc adm policy add-cluster-role-to-user cluster-monitoring-view -z infinispan-monitoring
Create a Grafana data source.
Retrieve the token for the
ServiceAccount
.$ oc serviceaccounts get-token infinispan-monitoring eyJhbGciOiJSUzI1NiIsImtpZCI6Imc4O...
Define a
GrafanaDataSource
that includes the token in thespec.datasources.secureJsonData.httpHeaderValue1
field, as in the following example:apiVersion: integreatly.org/v1alpha1 kind: GrafanaDataSource metadata: name: grafanadatasource spec: name: datasource.yaml datasources: - access: proxy editable: true isDefault: true jsonData: httpHeaderName1: Authorization timeInterval: 5s tlsSkipVerify: true name: Prometheus secureJsonData: httpHeaderValue1: >- Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Imc4O... type: prometheus url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
Apply the
GrafanaDataSource
.$ oc apply -f grafana-datasource.yaml
Next steps
Enable Grafana dashboards with the Data Grid Operator configuration properties.
9.4. Configuring Data Grid dashboards
Data Grid Operator provides global configuration properties that let you configure Grafana dashboards for Data Grid clusters.
You can modify global configuration properties while Data Grid Operator is running.
Prerequisites
- Data Grid Operator must watch the namespace where the Grafana Operator is running.
Procedure
Create a
ConfigMap
namedinfinispan-operator-config
in the Data Grid Operator namespace.apiVersion: v1 kind: ConfigMap metadata: name: infinispan-operator-config data: grafana.dashboard.namespace: example-infinispan grafana.dashboard.name: infinispan grafana.dashboard.monitoring.key: middleware
Specify the namespace of your Data Grid cluster with the
data.grafana.dashboard.namespace
property.NoteDeleting the value for this property removes the dashboard. Changing the value moves the dashboard to that namespace.
-
Specify a name for the dashboard with the
data.grafana.dashboard.name
property. -
If necessary, specify a monitoring key with the
data.grafana.dashboard.monitoring.key
property. Create
infinispan-operator-config
or update the configuration.$ oc apply -f infinispan-operator-config.yaml
Open the Grafana UI, which is available at:
$ oc get routes grafana-route -o jsonpath=https://"{.spec.host}"
Chapter 10. Setting up cross-site replication
Ensure service availability with Data Grid Operator by configuring cross-site replication to back up data between Data Grid clusters.
10.1. Using Data Grid Operator to manage cross-site connections
Data Grid Operator in one data center can discover a Data Grid cluster that Data Grid Operator manages in another data center. This discovery allows Data Grid to automatically form cross-site views and create global clusters.
The following illustration provides an example in which Data Grid Operator manages a Data Grid cluster at a data center in New York City, NYC. At another data center in London, LON, Data Grid Operator also manages a Data Grid cluster.
Data Grid Operator uses the Kubernetes API to establish a secure connection between the OpenShift Container Platform clusters in NYC and LON. Data Grid Operator then creates a cross-site replication service so Data Grid clusters can back up data across locations.
Data Grid Operator in each OpenShift cluster must have network access to the remote Kubernetes API.
When you configure automatic connections, Data Grid clusters do not start running until Data Grid Operator discovers all backup locations in the configuration.
Each Data Grid cluster has one site master node that coordinates all backup requests. Data Grid Operator identifies the site master node so that all traffic through the cross-site replication service goes to the site master.
If the current site master node goes offline then a new node becomes site master. Data Grid Operator automatically finds the new site master node and updates the cross-site replication service to forward backup requests to it.
10.1.1. Creating service account tokens
Generate service account tokens on each OpenShift cluster that acts as a backup location. Clusters use these tokens to authenticate with each other so Data Grid Operator can create a cross-site replication service.
Procedure
- Log in to an OpenShift cluster.
Create a service account.
For example, create a service account at LON:
$ oc create sa lon serviceaccount/lon created
Add the view role to the service account with the following command:
$ oc policy add-role-to-user view system:serviceaccount:<namespace>:lon
If you use a node port service to expose Data Grid clusters on the network, you must also add the
cluster-reader
role to the service account:$ oc adm policy add-cluster-role-to-user cluster-reader -z <service-account-name> -n <namespace>
- Repeat the preceding steps on your other OpenShift clusters.
Additional resources
10.1.2. Exchanging service account tokens
After you create service account tokens on your OpenShift clusters, you add them to secrets on each backup location. For example, at LON you add the service account token for NYC. At NYC you add the service account token for LON.
Prerequisites
Get tokens from each service account.
Use the following command or get the token from the OpenShift Web Console:
$ oc sa get-token lon eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
Procedure
- Log in to an OpenShift cluster.
Add the service account token for a backup location with the following command:
$ oc create secret generic <token-name> --from-literal=token=<token>
For example, log in to the OpenShift cluster at NYC and create a
lon-token
secret as follows:$ oc create secret generic lon-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
- Repeat the preceding steps on your other OpenShift clusters.
10.1.3. Configuring Data Grid Operator to handle cross-site connections
Configure Data Grid Operator to establish cross-site views with Data Grid clusters.
Prerequisites
- Create secrets that contain service account tokens for each backup location.
Procedure
-
Create an
Infinispan
CR for each Data Grid cluster. -
Specify the name of the local site with
spec.service.sites.local.name
. -
Set the value of the
spec.service.sites.local.expose.type
field to eitherNodePort
orLoadBalancer
. Optionally configure ports with the following fields:
-
spec.service.sites.local.expose.nodePort
if you useNodePort
. -
spec.service.sites.local.expose.port
if you useLoadBalancer
.
-
-
Provide the name, URL, and secret for each Data Grid cluster that acts as a backup location with
spec.service.sites.locations
. If Data Grid cluster names or namespaces at the remote site do not match the local site, specify those values with the
clusterName
andnamespace
fields.The following are example
Infinispan
CR definitions for LON and NYC:LON
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 3 service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer port: 65535 locations: - name: NYC clusterName: <nyc_cluster_name> namespace: <nyc_cluster_namespace> url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443 secretName: nyc-token logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
NYC
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: nyc-cluster spec: replicas: 2 service: type: DataGrid sites: local: name: NYC expose: type: LoadBalancer port: 65535 locations: - name: LON clusterName: example-infinispan namespace: rhdg-namespace url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443 secretName: lon-token logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
ImportantBe sure to adjust logging categories in your
Infinispan
CR to decrease log levels for JGroups TCP and RELAY2 protocols. This prevents a large number of log files from uses container storage.spec: logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
-
Configure your
Infinispan
CRs with any other Data Grid service resources and then apply the changes. Verify that Data Grid clusters form a cross-site view.
Retrieve the
Infinispan
CR.$ oc get infinispan -o yaml
-
Check for the
type: CrossSiteViewFormed
condition.
Next steps
If your clusters have formed a cross-site view, you can start adding backup locations to caches.
10.1.4. Resources for managed cross-site connections
This topic describes resources for cross-site connections that Data Grid Operator manages.
spec: service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer locations: - name: NYC clusterName: <nyc_cluster_name> namespace: <nyc_cluster_namespace> url: openshift://api.site-b.devcluster.openshift.com:6443 secretName: nyc-token
Field | Description |
---|---|
| Data Grid supports cross-site replication with Data Grid service clusters only. |
| Names the local site where a Data Grid cluster runs. |
|
Specifies the network service for cross-site replication. Data Grid clusters use this service to communicate and perform backup operations. You can set the value to |
|
Specifies a static port within the default range of |
|
Specifies the network port for the service if you expose Data Grid through a |
| Provides connection information for all backup locations. |
|
Specifies a backup location that matches |
| Specifies the URL of the Kubernetes API for the backup location. |
| Specifies the secret that contains the service account token for the backup site. |
| Specifies the cluster name at the backup location if it is different to the cluster name at the local site. |
| Specifies the namespace of the Data Grid cluster at the backup location if it does not match the namespace at the local site. |
10.2. Manually connecting Data Grid clusters
You can specify static network connection details to perform cross-site replication with Data Grid clusters running outside OpenShift. Manual cross-site connections are necessary in any scenario where access to the Kubernetes API is not available outside the OpenShift cluster where Data Grid runs.
You can use both automatic and manual connections for Data Grid clusters in the same Infinispan
CR. However, you must ensure that Data Grid clusters establish connections in the same way at each site.
Prerequisites
Manually connecting Data Grid clusters to form cross-site views requires predictable network locations for Data Grid services.
You need to know the network locations before they are created, which requires you to:
- Have the host names and ports for each Data Grid cluster that you plan to configure as a backup location.
-
Have the host name of the
<cluster-name>-site
service for any remote Data Grid cluster that is running on OpenShift.
You must use the<cluster-name>-site
service to form a cross-site view between a cluster that Data Grid Operator manages and any other cluster.
Procedure
-
Create an
Infinispan
CR for each Data Grid cluster. -
Specify the name of the local site with
spec.service.sites.local.name
. -
Set the value of the
spec.service.sites.local.expose.type
field to eitherNodePort
orLoadBalancer
. Optionally configure ports with the following fields:
-
spec.service.sites.local.expose.nodePort
if you useNodePort
. -
spec.service.sites.local.expose.port
if you useLoadBalancer
.
-
Provide the name and static URL for each Data Grid cluster that acts as a backup location with
spec.service.sites.locations
, for example:LON
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 3 service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer port: 65535 locations: - name: NYC url: infinispan+xsite://infinispan-nyc.myhost.com:7900 logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
NYC
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 2 service: type: DataGrid sites: local: name: NYC expose: type: LoadBalancer port: 65535 locations: - name: LON url: infinispan+xsite://infinispan-lon.myhost.com logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
ImportantBe sure to adjust logging categories in your
Infinispan
CR to decrease log levels for JGroups TCP and RELAY2 protocols. This prevents a large number of log files from uses container storage.spec: logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
-
Configure your
Infinispan
CRs with any other Data Grid service resources and then apply the changes. Verify that Data Grid clusters form a cross-site view.
Retrieve the
Infinispan
CR.$ oc get infinispan -o yaml
-
Check for the
type: CrossSiteViewFormed
condition.
Next steps
If your clusters have formed a cross-site view, you can start adding backup locations to caches.
10.2.1. Resources for manual cross-site connections
This topic describes resources for cross-site connections that you maintain manually.
spec: service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer port: 65535 locations: - name: NYC url: infinispan+xsite://infinispan-nyc.myhost.com:7900
Field | Description |
---|---|
| Data Grid supports cross-site replication with Data Grid service clusters only. |
| Names the local site where a Data Grid cluster runs. |
|
Specifies the network service for cross-site replication. Data Grid clusters use this service to communicate and perform backup operations. You can set the value to |
|
Specifies a static port within the default range of |
|
Specifies the network port for the service if you expose Data Grid through a |
| Provides connection information for all backup locations. |
|
Specifies a backup location that matches |
|
Specifies the static URL for the backup location in the format of |
10.3. Configuring sites in the same OpenShift cluster
For evaluation and demonstration purposes, you can configure Data Grid to back up between nodes in the same OpenShift cluster.
Procedure
-
Create an
Infinispan
CR for each Data Grid cluster. -
Specify the name of the local site with
spec.service.sites.local.name
. -
Set
ClusterIP
as the value of thespec.service.sites.local.expose.type
field. -
Provide the name of the Data Grid cluster that acts as a backup location with
spec.service.sites.locations.clusterName
. If both Data Grid clusters have the same name, specify the namespace of the backup location with
spec.service.sites.locations.namespace
.apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-clustera spec: replicas: 1 expose: type: LoadBalancer service: type: DataGrid sites: local: name: SiteA expose: type: ClusterIP locations: - name: SiteB clusterName: example-clusterb namespace: cluster-namespace
-
Configure your
Infinispan
CRs with any other Data Grid service resources and then apply the changes. Verify that Data Grid clusters form a cross-site view.
Retrieve the
Infinispan
CR.$ oc get infinispan -o yaml
-
Check for the
type: CrossSiteViewFormed
condition.
Chapter 11. Guaranteeing availability with anti-affinity
Kubernetes includes anti-affinity capabilities that protect workloads from single points of failure.
11.1. Anti-affinity strategies
Each Data Grid node in a cluster runs in a pod that runs on an OpenShift node in a cluster. Each Red Hat OpenShift node runs on a physical host system. Anti-affinity works by distributing Data Grid nodes across OpenShift nodes, ensuring that your Data Grid clusters remain available even if hardware failures occur.
Data Grid Operator offers two anti-affinity strategies:
kubernetes.io/hostname
- Data Grid replica pods are scheduled on different OpenShift nodes.
topology.kubernetes.io/zone
- Data Grid replica pods are scheduled across multiple zones.
Fault tolerance
Anti-affinity strategies guarantee cluster availability in different ways.
The equations in the following section apply only if the number of OpenShift nodes or zones is greater than the number of Data Grid nodes.
Scheduling pods on different OpenShift nodes
Provides tolerance of x
node failures for the following types of cache:
-
Replicated:
x = spec.replicas - 1
-
Distributed:
x = num_owners - 1
Scheduling pods across multiple zones
Provides tolerance of x
zone failures when x
zones exist for the following types of cache:
-
Replicated:
x = spec.replicas - 1
-
Distributed:
x = num_owners - 1
spec.replicas
- Defines the number of pods in each Data Grid cluster.
num_owners
- Is the cache configuration attribute that defines the number of replicas for each entry in the cache.
11.2. Configuring anti-affinity
Specify where OpenShift schedules pods for your Data Grid clusters to ensure availability.
Procedure
-
Add the
spec.affinity
block to yourInfinispan
CR. - Configure anti-affinity strategies as necessary.
-
Apply your
Infinispan
CR.
11.2.1. Anti-affinity strategy configurations
Configure anti-affinity strategies in your Infinispan
CR to control where OpenShift schedules Data Grid replica pods.
Topology keys | Description |
---|---|
| Schedules Data Grid replica pods across multiple zones. |
| Schedules Data Grid replica pods on different OpenShift nodes. |
Schedule pods on different OpenShift nodes
The following is the anti-affinity strategy that Data Grid Operator uses if you do not configure the spec.affinity
field in your Infinispan
CR:
spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: infinispan-pod clusterName: <cluster_name> infinispan_cr: <cluster_name> topologyKey: "kubernetes.io/hostname"
Requiring different nodes
In the following example, OpenShift does not schedule Data Grid pods if different nodes are not available:
spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: infinispan-pod clusterName: <cluster_name> infinispan_cr: <cluster_name> topologyKey: "topology.kubernetes.io/hostname"
To ensure that you can schedule Data Grid replica pods on different OpenShift nodes, the number of OpenShift nodes available must be greater than the value of spec.replicas
.
Schedule pods across multiple OpenShift zones
The following example prefers multiple zones when scheduling pods but schedules Data Grid replica pods on different OpenShift nodes if it is not possible to schedule across zones:
spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: infinispan-pod clusterName: <cluster_name> infinispan_cr: <cluster_name> topologyKey: "topology.kubernetes.io/zone" - weight: 90 podAffinityTerm: labelSelector: matchLabels: app: infinispan-pod clusterName: <cluster_name> infinispan_cr: <cluster_name> topologyKey: "kubernetes.io/hostname"
Requiring multiple zones
The following example uses the zone strategy only when scheduling Data Grid replica pods:
spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: infinispan-pod clusterName: <cluster_name> infinispan_cr: <cluster_name> topologyKey: "topology.kubernetes.io/zone"
Chapter 12. Creating caches with Data Grid Operator
Use Cache
CRs to add cache configuration with Data Grid Operator and control how Data Grid stores your data.
Creating caches with Data Grid Operator is available as a technology preview.
12.1. Technology Previews
Technology Preview features or capabilities are not supported with Red Hat production service-level agreements (SLAs) and might not be functionally complete.
Red Hat does not recommend using Technology Preview features or capabilities for production. These features provide early access to upcoming product features, which enables you to test functionality and provide feedback during the development process.
For more information, see Red Hat Technology Preview Features Support Scope.
12.2. Data Grid caches
Cache configuration defines the characteristics and features of the data store and must be valid with the Data Grid schema. Data Grid recommends creating standalone files in XML or JSON format that define your cache configuration. You should separate Data Grid configuration from application code for easier validation and to avoid the situation where you need to maintain XML snippets in Java or some other client language.
To create caches with Data Grid clusters running on OpenShift, you should:
-
Use
Cache
CR as the mechanism for creating caches through the OpenShift front end. -
Use
Batch
CR to create multiple caches at a time from standalone configuration files. - Access Data Grid Console and create caches in XML or JSON format.
You can use Hot Rod or HTTP clients but Data Grid recommends Cache
CR or Batch
CR unless your specific use case requires programmatic remote cache creation.
12.3. Cache CRs
Find out details for configuring Data Grid caches with Cache
CR.
When using Cache
CRs, the following rules apply:
-
Cache
CRs apply to Data Grid service pods only. -
You can create a single cache for each
Cache
CR. -
If your
Cache
CR contains both a template and an XML configuration, Data Grid Operator uses the template. - If you edit caches in the OpenShift Web Console, the changes are reflected through the user interface but do not take effect on the Data Grid cluster. You cannot edit caches. To change cache configuration, you must first delete the cache through the console or CLI and then re-create the cache.
-
Deleting
Cache
CRs in the OpenShift Web Console does not remove caches from Data Grid clusters. You must delete caches through the console or CLI.
In previous versions, you need to add credentials to a secret so that Data Grid Operator can access your cluster when creating caches.
That is no longer necessary. Data Grid Operator uses the operator user and corresponding password to perform cache operations.
12.4. Creating caches from XML
Complete the following steps to create caches on Data Grid service clusters using valid infinispan.xml
configuration.
Procedure
Create a
Cache
CR that contains an XML cache configuration.-
Specify a name for the
Cache
CR with themetadata.name
field. -
Specify the target Data Grid cluster with the
spec.clusterName
field. Name your cache with the
spec.name
field.NoteThe
name
attribute in the XML configuration is ignored. Only thespec.name
field applies to the resulting cache.Add an XML cache configuration with the
spec.template
field.apiVersion: infinispan.org/v2alpha1 kind: Cache metadata: name: mycachedefinition spec: clusterName: example-infinispan name: mycache template: <distributed-cache name="mycache" mode="SYNC"><persistence><file-store/></persistence></distributed-cache>
-
Specify a name for the
Apply the
Cache
CR, for example:$ oc apply -f mycache.yaml cache.infinispan.org/mycachedefinition created
12.5. Creating caches from templates
Complete the following steps to create caches on Data Grid service clusters using cache templates.
Prerequisites
-
Identify the cache template you want to use for your cache.
You can find a list of available templates in Data Grid Console.
Procedure
Create a
Cache
CR that specifies the name of a template to use.-
Specify a name for the
Cache
CR with themetadata.name
field. -
Specify the target Data Grid cluster with the
spec.clusterName
field. -
Name your cache with the
spec.name
field. Specify a cache template with the
spec.template
field.The following example creates a cache named "mycache" from the
org.infinispan.DIST_SYNC
cache template:apiVersion: infinispan.org/v2alpha1 kind: Cache metadata: name: mycachedefinition spec: clusterName: example-infinispan name: mycache templateName: org.infinispan.DIST_SYNC
-
Specify a name for the
Apply the
Cache
CR, for example:$ oc apply -f mycache.yaml cache.infinispan.org/mycachedefinition created
12.6. Adding backup locations to caches
When you configure Data Grid clusters to perform cross-site replication, you can add backup locations to your cache configurations.
Procedure
Create cache configurations that name remote sites as backup locations.
Data Grid replicates data based on cache names. For this reason, site names in your cache configurations must match site names,
spec.service.sites.local.name
, in yourInfinispan
CRs.Configure backup locations to go offline automatically with the
take-offline
element.-
Set the amount of time, in milliseconds, before backup locations go offline with the
min-wait
attribute.
-
Set the amount of time, in milliseconds, before backup locations go offline with the
- Define any other valid cache configuration.
Add backup locations to the named cache on all sites in the global cluster.
For example, if you add LON as a backup for NYC you should add NYC as a backup for LON.
The following configuration examples show backup locations for caches:
NYC
<distributed-cache name="customers"> <encoding media-type="application/x-protostream"/> <backups> <backup site="LON" strategy="SYNC"> <take-offline min-wait="120000"/> </backup> </backups> </distributed-cache>
LON
<replicated-cache name="customers"> <encoding media-type="application/x-protostream"/> <backups> <backup site="NYC" strategy="ASYNC" > <take-offline min-wait="120000"/> </backup> </backups> </replicated-cache>
Additional resources
12.6.1. Performance considerations with taking backup locations offline
Backup locations can automatically go offline when remote sites become unavailable. This prevents pods from attempting to replicate data to offline backup locations, which can have a performance impact on your cluster because it results in error.
You can configure how long to wait before backup locations go offline. A good rule of thumb is one or two minutes. However, you should test different wait periods and evaluate their performance impacts to determine the correct value for your deployment.
For instance when OpenShift terminates the site master pod, that backup location becomes unavailable for a short period of time until Data Grid Operator elects a new site master. In this case, if the minimum wait time is not long enough then the backup locations go offline. You then need to bring those backup locations online and perform state transfer operations to ensure the data is in sync.
Likewise, if the minimum wait time is too long, node CPU usage increases from failed backup attempts which can lead to performance degradation.
12.7. Adding persistent cache stores
You can add persistent cache stores to Data Grid service pods to save data to the persistent volume.
Data Grid creates a Single File cache store, .dat
file, in the /opt/infinispan/server/data
directory.
Procedure
Add the
<file-store/>
element to thepersistence
configuration in your Data Grid cache, as in the following example:<distributed-cache name="persistent-cache" mode="SYNC"> <encoding media-type="application/x-protostream"/> <persistence> <file-store/> </persistence> </distributed-cache>
Chapter 13. Running batch operations
Data Grid Operator provides a Batch
CR that lets you create Data Grid resources in bulk. Batch
CR uses the Data Grid command line interface (CLI) in batch mode to carry out sequences of operations.
Modifying a Batch
CR instance has no effect. Batch operations are "one-time" events that modify Data Grid resources. To update .spec
fields for the CR, or when a batch operation fails, you must create a new instance of the Batch
CR.
13.1. Running inline batch operations
Include your batch operations directly in a Batch
CR if they do not require separate configuration artifacts.
Procedure
Create a
Batch
CR.-
Specify the name of the Data Grid cluster where you want the batch operations to run as the value of the
spec.cluster
field. Add each CLI command to run on a line in the
spec.config
field.apiVersion: infinispan.org/v2alpha1 kind: Batch metadata: name: mybatch spec: cluster: example-infinispan config: | create cache --template=org.infinispan.DIST_SYNC mycache put --cache=mycache hello world put --cache=mycache hola mundo
-
Specify the name of the Data Grid cluster where you want the batch operations to run as the value of the
Apply your
Batch
CR.$ oc apply -f mybatch.yaml
-
Check the
status.Phase
field in theBatch
CR to verify the operations completed successfully.
13.2. Creating ConfigMaps for batch operations
Create a ConfigMap
so that additional files, such as Data Grid cache configuration, are available for batch operations.
Prerequisites
For demonstration purposes, you should add some configuration artifacts to your host filesystem before you start the procedure:
Create a
/tmp/mybatch
directory where you can add some files.$ mkdir -p /tmp/mybatch
Create a Data Grid cache configuration.
$ cat > /tmp/mybatch/mycache.xml<<EOF <distributed-cache name="mycache" mode="SYNC"> <encoding media-type="application/x-protostream"/> <memory max-count="1000000" when-full="REMOVE"/> </distributed-cache> EOF
Procedure
Create a
batch
file that contains all commands you want to run.For example, the following
batch
file creates a cache named "mycache" and adds two entries to it:create cache mycache --file=/etc/batch/mycache.xml put --cache=mycache hello world put --cache=mycache hola mundo
ImportantThe
ConfigMap
is mounted in Data Grid pods at/etc/batch
. You must prepend all--file=
directives in your batch operations with that path.Ensure all configuration artifacts that your batch operations require are in the same directory as the
batch
file.$ ls /tmp/mybatch batch mycache.xml
Create a
ConfigMap
from the directory.$ oc create configmap mybatch-config-map --from-file=/tmp/mybatch
13.3. Running batch operations with ConfigMaps
Run batch operations that include configuration artifacts.
Prerequisites
-
Create a
ConfigMap
that contains any files your batch operations require.
Procedure
-
Create a
Batch
CR that specifies the name of a Data Grid cluster as the value of thespec.cluster
field. Set the name of the
ConfigMap
that contains yourbatch
file and configuration artifacts with thespec.configMap
field.$ cat > mybatch.yaml<<EOF apiVersion: infinispan.org/v2alpha1 kind: Batch metadata: name: mybatch spec: cluster: example-infinispan configMap: mybatch-config-map EOF
Apply your
Batch
CR.$ oc apply -f mybatch.yaml
-
Check the
status.Phase
field in theBatch
CR to verify the operations completed successfully.
13.4. Batch status messages
Verify and troubleshoot batch operations with the status.Phase
field in the Batch
CR.
Phase | Description |
---|---|
| All batch operations have completed successfully. |
| Batch operations are queued and resources are initializing. |
| Batch operations are ready to start. |
| Batch operations are in progress. |
| One or more batch operations were not successful. |
Failed operations
Batch operations are not atomic. If a command in a batch script fails, it does not affect the other operations or cause them to rollback.
If your batch operations have any server or syntax errors, you can view log messages in the Batch
CR in the status.Reason
field.
13.5. Example batch operations
Use these example batch operations as starting points for creating and modifying Data Grid resources with the Batch
CR.
You can pass configuration files to Data Grid Operator only via a ConfigMap
.
The ConfigMap
is mounted in Data Grid pods at /etc/batch
so you must prepend all --file=
directives with that path.
13.5.1. Caches
- Create multiple caches from configuration files.
echo "creating caches..." create cache sessions --file=/etc/batch/infinispan-prod-sessions.xml create cache tokens --file=/etc/batch/infinispan-prod-tokens.xml create cache people --file=/etc/batch/infinispan-prod-people.xml create cache books --file=/etc/batch/infinispan-prod-books.xml create cache authors --file=/etc/batch/infinispan-prod-authors.xml echo "list caches in the cluster" ls caches
- Create a template from a file and then create caches from the template.
echo "creating caches..." create cache mytemplate --file=/etc/batch/mycache.xml create cache sessions --template=mytemplate create cache tokens --template=mytemplate echo "list caches in the cluster" ls caches
13.5.2. Counters
Use the Batch
CR to create multiple counters that can increment and decrement to record the count of objects.
You can use counters to generate identifiers, act as rate limiters, or track the number of times a resource is accessed.
echo "creating counters..." create counter --concurrency-level=1 --initial-value=5 --storage=PERSISTENT --type=weak mycounter1 create counter --initial-value=3 --storage=PERSISTENT --type=strong mycounter2 create counter --initial-value=13 --storage=PERSISTENT --type=strong --upper-bound=10 mycounter3 echo "list counters in the cluster" ls counters
13.5.3. Protobuf schema
Register Protobuf schema to query values in caches. Protobuf schema (.proto
files) provide metadata about custom entities and controls field indexing.
echo "creating schema..." schema --upload=person.proto person.proto schema --upload=book.proto book.proto schema --upload=author.proto book.proto echo "list Protobuf schema" ls schemas
13.5.4. Tasks
Upload tasks that implement org.infinispan.tasks.ServerTask
or scripts that are compatible with the javax.script
scripting API.
echo "creating tasks..." task upload --file=/etc/batch/myfirstscript.js myfirstscript task upload --file=/etc/batch/mysecondscript.js mysecondscript task upload --file=/etc/batch/mythirdscript.js mythirdscript echo "list tasks" ls tasks
Additional resources
Chapter 14. Backing up and restoring Data Grid clusters
Data Grid Operator lets you back up and restore Data Grid cluster state for disaster recovery and to migrate Data Grid resources between clusters.
14.1. Backup and Restore CRs
Backup
and Restore
CRs save in-memory data at runtime so you can easily recreate Data Grid clusters.
Applying a Backup
or Restore
CR creates a new pod that joins the Data Grid cluster as a zero-capacity member, which means it does not require cluster rebalancing or state transfer to join.
For backup operations, the pod iterates over cache entries and other resources and creates an archive, a .zip
file, in the /opt/infinispan/backups
directory on the persistent volume (PV).
Performing backups does not significantly impact performance because the other pods in the Data Grid cluster only need to respond to the backup pod as it iterates over cache entries.
For restore operations, the pod retrieves Data Grid resources from the archive on the PV and applies them to the Data Grid cluster.
When either the backup or restore operation completes, the pod leaves the cluster and is terminated.
Reconciliation
Data Grid Operator does not reconcile Backup
and Restore
CRs which mean that backup and restore operations are "one-time" events.
Modifying an existing Backup
or Restore
CR instance does not perform an operation or have any effect. If you want to update .spec
fields, you must create a new instance of the Backup
or Restore
CR.
14.2. Backing up Data Grid clusters
Create a backup file that stores Data Grid cluster state to a persistent volume.
Prerequisites
-
Create an
Infinispan
CR withspec.service.type: DataGrid
. Ensure there are no active client connections to the Data Grid cluster.
Data Grid backups do not provide snapshot isolation and data modifications are not written to the archive after the cache is backed up.
To archive the exact state of the cluster, you should always disconnect any clients before you back it up.
Procedure
-
Name the
Backup
CR with themetadata.name
field. -
Specify the Data Grid cluster to backup with the
spec.cluster
field. Configure the persistent volume claim (PVC) that adds the backup archive to the persistent volume (PV) with the
spec.volume.storage
andspec.volume.storage.storageClassName
fields.apiVersion: infinispan.org/v2alpha1 kind: Backup metadata: name: my-backup spec: cluster: source-cluster volume: storage: 1Gi storageClassName: my-storage-class
Optionally include
spec.resources
fields to specify which Data Grid resources you want to back up.If you do not include any
spec.resources
fields, theBackup
CR creates an archive that contains all Data Grid resources. If you do specifyspec.resources
fields, theBackup
CR creates an archive that contains those resources only.spec: ... resources: templates: - distributed-sync-prod - distributed-sync-dev caches: - cache-one - cache-two counters: - counter-name protoSchemas: - authors.proto - books.proto tasks: - wordStream.js
You can also use the
*
wildcard character as in the following example:spec: ... resources: caches: - "*" protoSchemas: - "*"
Apply your
Backup
CR.$ oc apply -f my-backup.yaml
Verification
Check that the
status.phase
field has a status ofSucceeded
in theBackup
CR and that Data Grid logs have the following message:ISPN005044: Backup file created 'my-backup.zip'
Run the following command to check that the backup is successfully created:
$ oc describe Backup my-backup
14.3. Restoring Data Grid clusters
Restore Data Grid cluster state from a backup archive.
Prerequisites
-
Create a
Backup
CR on a source cluster. Create a target Data Grid cluster of Data Grid service pods.
NoteIf you restore an existing cache, the operation overwrites the data in the cache but not the cache configuration.
For example, you back up a distributed cache named
mycache
on the source cluster. You then restoremycache
on a target cluster where it already exists as a replicated cache. In this case, the data from the source cluster is restored andmycache
continues to have a replicated configuration on the target cluster.Ensure there are no active client connections to the target Data Grid cluster you want to restore.
Cache entries that you restore from a backup can overwrite more recent cache entries.
For example, a client performs acache.put(k=2)
operation and you then restore a backup that containsk=1
.
Procedure
-
Name the
Restore
CR with themetadata.name
field. -
Specify a
Backup
CR to use with thespec.backup
field. Specify the Data Grid cluster to restore with the
spec.cluster
field.apiVersion: infinispan.org/v2alpha1 kind: Restore metadata: name: my-restore spec: backup: my-backup cluster: target-cluster
Optionally add the
spec.resources
field to restore specific resources only.spec: ... resources: templates: - distributed-sync-prod - distributed-sync-dev caches: - cache-one - cache-two counters: - counter-name protoSchemas: - authors.proto - books.proto tasks: - wordStream.js
Apply your
Restore
CR.$ oc apply -f my-restore.yaml
Verification
Check that the
status.phase
field has a status ofSucceeded
in theRestore
CR and that Data Grid logs have the following message:ISPN005045: Restore 'my-backup' complete
You should then open the Data Grid Console or establish a CLI connection to verify data and Data Grid resources are restored as expected.
14.4. Backup and restore status
Backup
and Restore
CRs include a status.phase
field that provides the status for each phase of the operation.
Status | Description |
---|---|
| The system has accepted the request and the controller is preparing the underlying resources to create the pod. |
| The controller has prepared all underlying resources successfully. |
| The pod is created and the operation is in progress on the Data Grid cluster. |
| The operation has completed successfully on the Data Grid cluster and the pod is terminated. |
| The operation did not successfully complete and the pod is terminated. |
| The controller cannot obtain the status of the pod or determine the state of the operation. This condition typically indicates a temporary communication error with the pod. |
14.4.1. Handling failed backup and restore operations
If the status.phase
field of the Backup
or Restore
CR is Failed
, you should examine pod logs to determine the root cause before you attempt the operation again.
Procedure
Examine the logs for the pod that performed the failed operation.
Pods are terminated but remain available until you delete the
Backup
orRestore
CR.$ oc logs <backup|restore_pod_name>
- Resolve any error conditions or other causes of failure as indicated by the pod logs.
-
Create a new instance of the
Backup
orRestore
CR and attempt the operation again.
Chapter 15. Deploying custom code to Data Grid
Add custom code, such as scripts and event listeners, to your Data Grid clusters.
Before you can deploy custom code to Data Grid clusters, you need to make it available. To do this you can copy artifacts from a persistent volume (PV), download artifacts from an HTTP or FTP server, or use both methods.
15.1. Copying code artifacts to Data Grid clusters
Adding your artifacts to a persistent volume (PV) and then copy them to Data Grid pods.
This procedure explains how to use a temporary pod that mounts a persistent volume claim (PVC) that:
- Lets you add code artifacts to the PV (perform a write operation).
- Allows Data Grid pods to load code artifacts from the PV (perform a read operation).
To perform these read and write operations, you need certain PV access modes. However, support for different PVC access modes is platform dependent.
It is beyond the scope of this document to provide instructions for creating PVCs with different platforms. For simplicity, the following procedure shows a PVC with the ReadWriteMany
access mode.
In some cases only the ReadOnlyMany
or ReadWriteOnce
access modes are available. You can use a combination of those access modes by reclaiming and reusing PVCs with the same spec.volumeName
.
Using ReadWriteOnce
access mode results in all Data Grid pods in a cluster being scheduled on the same OpenShift node.
Procedure
Change to the namespace for your Data Grid cluster.
$ oc project rhdg-namespace
Create a PVC for your custom code artifacts, for example:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: datagrid-libs spec: accessModes: - ReadWriteMany resources: requests: storage: 100Mi
Apply your PVC.
$ oc apply -f datagrid-libs.yaml
Create a pod that mounts the PVC, for example:
apiVersion: v1 kind: Pod metadata: name: datagrid-libs-pod spec: securityContext: fsGroup: 2000 volumes: - name: lib-pv-storage persistentVolumeClaim: claimName: datagrid-libs containers: - name: lib-pv-container image: registry.redhat.io/datagrid/datagrid-8-rhel8:8.2 volumeMounts: - mountPath: /tmp/libs name: lib-pv-storage
Add the pod to the Data Grid namespace and wait for it to be ready.
$ oc apply -f datagrid-libs-pod.yaml $ oc wait --for=condition=ready --timeout=2m pod/datagrid-libs-pod
Copy your code artifacts to the pod so that they are loaded into the PVC.
For example to copy code artifacts from a local
libs
directory, do the following:$ oc cp --no-preserve=true libs datagrid-libs-pod:/tmp/
Delete the pod.
$ oc delete pod datagrid-libs-pod
Specify the persistent volume with
spec.dependencies.volumeClaimName
in yourInfinispan
CR and then apply the changes.apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 2 dependencies: volumeClaimName: datagrid-libs service: type: DataGrid
If you update your custom code on the persistent volume, you must restart the Data Grid cluster so it can load the changes.
Additional resources
15.2. Downloading code artifacts
Add your artifacts to an HTTP or FTP server so that Data Grid Operator downloads them to the {lib_path}
directory on each Data Grid node.
When downloading files, Data Grid Operator can automatically detect the file type. Data Grid Operator also extracts archived files, such as zip
or tgz
, to the filesystem after the download completes.
Each time Data Grid Operator creates a Data Grid node it downloads the artifacts to the node. The download also occurs when Data Grid Operator recreates pods after terminating them.
Prerequisites
- Host your code artifacts on an HTTP or FTP server.
Procedure
Add the
spec.dependencies.artifacts
field to yourInfinispan
CR.-
Specify the location of the file to download via
HTTP
orFTP
as the value of thespec.dependencies.artifacts.url
field. Optionally specify a checksum to verify the integrity of the download with the
spec.dependencies.artifacts.hash
field.The
hash
field requires a value is in the format of<algorithm>:<checksum>
where<algorithm>
issha1|sha224|sha256|sha384|sha512|md5
.Set the file type, if necessary, with the
spec.dependencies.artifacts.type
field.You should explicitly set the file type if it is not included in the URL or if the file type is actually different to the extension in the URL.
NoteIf you set
type: file
, Data Grid Operator downloads the file as-is without extracting it to the filesystem.apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 2 dependencies: artifacts: - url: http://example.com:8080/path hash: sha256:596408848b56b5a23096baa110cd8b633c9a9aef2edd6b38943ade5b4edcd686 type: zip service: type: DataGrid
-
Specify the location of the file to download via
- Apply the changes.
Chapter 16. Sending cloud events from Data Grid clusters
Configure Data Grid as a Knative source by sending CloudEvents
to Apache Kafka topics.
Sending cloud events with Red Hat OpenShift Serverless is available as a technology preview.
16.1. Technology Previews
Technology Preview features or capabilities are not supported with Red Hat production service-level agreements (SLAs) and might not be functionally complete.
Red Hat does not recommend using Technology Preview features or capabilities for production. These features provide early access to upcoming product features, which enables you to test functionality and provide feedback during the development process.
For more information, see Red Hat Technology Preview Features Support Scope.
16.2. Cloud events
You can send CloudEvents
from Data Grid clusters when entries in caches are created, updated, removed, or expired.
Data Grid sends structured events to Kafka in JSON format, as in the following example:
{ "specversion": "1.0", "source": "/infinispan/<cluster_name>/<cache_name>", "type": "org.infinispan.entry.created", "time": "<timestamp>", "subject": "<key-name>", "id": "key-name:CommandInvocation:node-name:0", "data": { "property": "value" } }
Field | Description |
---|---|
|
Prefixes events for Data Grid cache entries with |
| Entry value. |
| Entry key, converted to string. |
| Generated identifier for the event. |
16.3. Enabling cloud events
Configure Data Grid to send CloudEvents
.
Prerequisites
- Set up an Kafka cluster that listens for Data Grid topics.
Procedure
Add
spec.cloudEvents
to yourInfinispan
CR.-
Configure the number of acknowledgements with the
spec.cloudEvents.acks
field. Values are "0", "1", or "all". -
List Kafka servers to which Data Grid sends events with the
spec.cloudEvents.bootstrapServers
field. Specify the Kafka topic for Data Grid events with the
spec.cloudEvents.cacheEntriesTopic
field.spec: cloudEvents: acks: "1" bootstrapServers: my-cluster-kafka-bootstrap_1.<namespace_1>.svc:9092,my-cluster-kafka-bootstrap_2.<namespace_2>.svc:9092 cacheEntriesTopic: target-topic
-
Configure the number of acknowledgements with the
- Apply your changes.
Chapter 17. Establishing remote client connections
Connect to Data Grid clusters from the Data Grid Console, Command Line Interface (CLI), and remote clients.
17.1. Client connection details
Before you can connect to Data Grid, you need to retrieve the following pieces of information:
- Service hostname
- Port
- Authentication credentials, if required
- TLS certificate, if you use encryption
Service hostnames
The service hostname depends on how you expose Data Grid on the network or if your clients are running on OpenShift.
For clients running on OpenShift, you can use the name of the internal service that Data Grid Operator creates.
For clients running outside OpenShift, the service hostname is the location URL if you use a load balancer. For a node port service, the service hostname is the node host name. For a route, the service hostname is either a custom hostname or a system-defined hostname.
Ports
Client connections on OpenShift and through load balancers use port 11222
.
Node port services use a port in the range of 30000
to 60000
. Routes use either port 80
(unencrypted) or 443
(encrypted).
Additional resources
17.2. Data Grid caches
Cache configuration defines the characteristics and features of the data store and must be valid with the Data Grid schema. Data Grid recommends creating standalone files in XML or JSON format that define your cache configuration. You should separate Data Grid configuration from application code for easier validation and to avoid the situation where you need to maintain XML snippets in Java or some other client language.
To create caches with Data Grid clusters running on OpenShift, you should:
-
Use
Cache
CR as the mechanism for creating caches through the OpenShift front end. -
Use
Batch
CR to create multiple caches at a time from standalone configuration files. - Access Data Grid Console and create caches in XML or JSON format.
You can use Hot Rod or HTTP clients but Data Grid recommends Cache
CR or Batch
CR unless your specific use case requires programmatic remote cache creation.
17.3. Connecting the Data Grid CLI
Use the command line interface (CLI) to connect to your Data Grid cluster and perform administrative operations.
Prerequisites
- Download a CLI distribution so you can connect to Data Grid clusters on OpenShift.
The Data Grid CLI is available with the server distribution or as a native executable.
Follow the instructions in Getting Started with Data Grid Server for information on downloading and installing the CLI as part of the server distribution. For the native CLI, you should follow the installation instructions in the README file that is included in the ZIP download.
It is possible to open a remote shell to a Data Grid node and access the CLI.
$ oc rsh example-infinispan-0
However using the CLI in this way consumes memory allocated to the container, which can lead to out of memory exceptions.
Procedure
Create a CLI connection to your Data Grid cluster.
Using the server distribution
$ bin/cli.sh -c https://$SERVICE_HOSTNAME:$PORT --trustall
Using the native CLI
$ ./redhat-datagrid-cli -c https://$SERVICE_HOSTNAME:$PORT --trustall
Replace
$SERVICE_HOSTNAME:$PORT
with the hostname and port where Data Grid is available on the network.- Enter your Data Grid credentials when prompted.
Perform CLI operations as required, for example:
List caches configured on the cluster with the
ls
command.[//containers/default]> ls caches mycache
View cache configuration with the
describe
command.[//containers/default]> describe caches/mycache
17.4. Accessing Data Grid Console
Access the console to create caches, perform adminstrative operations, and monitor your Data Grid clusters.
Prerequisites
-
Expose Data Grid on the network so you can access the console through a browser.
For example, configure a load balancer service or create a route.
Procedure
Access the console from any browser at
$SERVICE_HOSTNAME:$PORT
.Replace
$SERVICE_HOSTNAME:$PORT
with the hostname and port where Data Grid is available on the network.
17.5. Hot Rod clients
Hot Rod is a binary TCP protocol that Data Grid provides for high-performance data transfer capabilities with remote clients.
Client intelligence
Client intelligence refers to mechanisms the Hot Rod protocol provides so that clients can locate and send requests to Data Grid pods.
Hot Rod clients running on OpenShift can access internal IP addresses for Data Grid pods so you can use any client intelligence. The default intelligence, HASH_DISTRIBUTION_AWARE
, is recommended because it allows clients to route requests to primary owners, which improves performance.
Hot Rod clients running outside OpenShift must use BASIC
intelligence.
17.5.1. Hot Rod client configuration API
You can programmatically configure Hot Rod client connections with the ConfigurationBuilder
interface.
$SERVICE_HOSTNAME:$PORT
denotes the hostname and port that allows access to your Data Grid cluster. You should replace these variables with the actual hostname and port for your environment.
On OpenShift
Hot Rod clients running on OpenShift can use the following configuration:
import org.infinispan.client.hotrod.configuration.ConfigurationBuilder; import org.infinispan.client.hotrod.configuration.SaslQop; import org.infinispan.client.hotrod.impl.ConfigurationProperties; ... ConfigurationBuilder builder = new ConfigurationBuilder(); builder.addServer() .host("$SERVICE_HOSTNAME") .port(ConfigurationProperties.DEFAULT_HOTROD_PORT) .security().authentication() .username("username") .password("changeme") .realm("default") .saslQop(SaslQop.AUTH) .saslMechanism("SCRAM-SHA-512") .ssl() .sniHostName("$SERVICE_HOSTNAME") .trustStoreFileName("/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt") .trustStoreType("pem");
Outside OpenShift
Hot Rod clients running outside OpenShift can use the following configuration:
import org.infinispan.client.hotrod.configuration.ClientIntelligence; import org.infinispan.client.hotrod.configuration.ConfigurationBuilder; import org.infinispan.client.hotrod.configuration.SaslQop; ... ConfigurationBuilder builder = new ConfigurationBuilder(); builder.addServer() .host("$SERVICE_HOSTNAME") .port("$PORT") .security().authentication() .username("username") .password("changeme") .realm("default") .saslQop(SaslQop.AUTH) .saslMechanism("SCRAM-SHA-512") .ssl() .sniHostName("$SERVICE_HOSTNAME") //Create a client trust store with tls.crt from your project. .trustStoreFileName("/path/to/truststore.pkcs12") .trustStorePassword("trust_store_password") .trustStoreType("PCKS12"); builder.clientIntelligence(ClientIntelligence.BASIC);
17.5.2. Hot Rod client properties
You can configure Hot Rod client connections with the hotrod-client.properties
file on the application classpath.
$SERVICE_HOSTNAME:$PORT
denotes the hostname and port that allows access to your Data Grid cluster. You should replace these variables with the actual hostname and port for your environment.
On OpenShift
Hot Rod clients running on OpenShift can use the following properties:
# Connection infinispan.client.hotrod.server_list=$SERVICE_HOSTNAME:$PORT # Authentication infinispan.client.hotrod.use_auth=true infinispan.client.hotrod.auth_username=developer infinispan.client.hotrod.auth_password=$PASSWORD infinispan.client.hotrod.auth_server_name=$CLUSTER_NAME infinispan.client.hotrod.sasl_properties.javax.security.sasl.qop=auth infinispan.client.hotrod.sasl_mechanism=SCRAM-SHA-512 # Encryption infinispan.client.hotrod.sni_host_name=$SERVICE_HOSTNAME infinispan.client.hotrod.trust_store_file_name=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt infinispan.client.hotrod.trust_store_type=pem
Outside OpenShift
Hot Rod clients running outside OpenShift can use the following properties:
# Connection infinispan.client.hotrod.server_list=$SERVICE_HOSTNAME:$PORT # Client intelligence infinispan.client.hotrod.client_intelligence=BASIC # Authentication infinispan.client.hotrod.use_auth=true infinispan.client.hotrod.auth_username=developer infinispan.client.hotrod.auth_password=$PASSWORD infinispan.client.hotrod.auth_server_name=$CLUSTER_NAME infinispan.client.hotrod.sasl_properties.javax.security.sasl.qop=auth infinispan.client.hotrod.sasl_mechanism=SCRAM-SHA-512 # Encryption infinispan.client.hotrod.sni_host_name=$SERVICE_HOSTNAME # Create a client trust store with tls.crt from your project. infinispan.client.hotrod.trust_store_file_name=/path/to/truststore.pkcs12 infinispan.client.hotrod.trust_store_password=trust_store_password infinispan.client.hotrod.trust_store_type=PCKS12
17.5.3. Configuring Hot Rod clients for certificate authentication
If you enable client certificate authentication, clients must present valid certificates when negotiating connections with Data Grid.
Validate strategy
If you use the Validate
strategy, you must configure clients with a keystore so they can present signed certificates. You must also configure clients with Data Grid credentials and any suitable authentication mechanism.
Authenticate strategy
If you use the Authenticate
strategy, you must configure clients with a keystore that contains signed certificates and valid Data Grid credentials as part of the distinguished name (DN). Hot Rod clients must also use the EXTERNAL
authentication mechanism.
If you enable security authorization, you should assign the Common Name (CN) from the client certificate a role with the appropriate permissions.
The following example shows a Hot Rod client configuration for client certificate authentication with the Authenticate
strategy:
import org.infinispan.client.hotrod.configuration.ConfigurationBuilder; ... ConfigurationBuilder builder = new ConfigurationBuilder(); builder.security() .authentication() .saslMechanism("EXTERNAL") .ssl() .keyStoreFileName("/path/to/keystore") .keyStorePassword("keystorepassword".toCharArray()) .keyStoreType("PCKS12");
17.5.4. Creating caches from Hot Rod clients
You can remotely create caches on Data Grid clusters running on OpenShift with Hot Rod clients. However, Data Grid recommends that you create caches using Data Grid Console, the CLI, or with Cache
CRs instead of with Hot Rod clients.
Programmatically creating caches
The following example shows how to add cache configurations to the ConfigurationBuilder
and then create them with the RemoteCacheManager
:
import org.infinispan.client.hotrod.DefaultTemplate; import org.infinispan.client.hotrod.RemoteCache; import org.infinispan.client.hotrod.RemoteCacheManager; ... builder.remoteCache("my-cache") .templateName(DefaultTemplate.DIST_SYNC); builder.remoteCache("another-cache") .configuration("<infinispan><cache-container><distributed-cache name=\"another-cache\"><encoding media-type=\"application/x-protostream\"/></distributed-cache></cache-container></infinispan>"); try (RemoteCacheManager cacheManager = new RemoteCacheManager(builder.build())) { // Get a remote cache that does not exist. // Rather than return null, create the cache from a template. RemoteCache<String, String> cache = cacheManager.getCache("my-cache"); // Store a value. cache.put("hello", "world"); // Retrieve the value and print it. System.out.printf("key = %s\n", cache.get("hello"));
This example shows how to create a cache named CacheWithXMLConfiguration using the XMLStringConfiguration()
method to pass the cache configuration as XML:
import org.infinispan.client.hotrod.RemoteCacheManager; import org.infinispan.commons.configuration.XMLStringConfiguration; ... private void createCacheWithXMLConfiguration() { String cacheName = "CacheWithXMLConfiguration"; String xml = String.format("<distributed-cache name=\"%s\">" + "<encoding media-type=\"application/x-protostream\"/>" + "<locking isolation=\"READ_COMMITTED\"/>" + "<transaction mode=\"NON_XA\"/>" + "<expiration lifespan=\"60000\" interval=\"20000\"/>" + "</distributed-cache>" , cacheName); manager.administration().getOrCreateCache(cacheName, new XMLStringConfiguration(xml)); System.out.println("Cache with configuration exists or is created."); }
Using Hot Rod client properties
When you invoke cacheManager.getCache()
calls for named caches that do not exist, Data Grid creates them from the Hot Rod client properties instead of returning null.
Add cache configuration to hotrod-client.properties
as in the following example:
# Add cache configuration infinispan.client.hotrod.cache.my-cache.template_name=org.infinispan.DIST_SYNC infinispan.client.hotrod.cache.another-cache.configuration=<infinispan><cache-container><distributed-cache name=\"another-cache\"/></cache-container></infinispan> infinispan.client.hotrod.cache.my-other-cache.configuration_uri=file:/path/to/configuration.xml
17.6. Accessing the REST API
Data Grid provides a RESTful interface that you can interact with using HTTP clients.
Prerequisites
-
Expose Data Grid on the network so you can access the REST API.
For example, configure a load balancer service or create a route.
Procedure
Access the REST API with any HTTP client at
$SERVICE_HOSTNAME:$PORT/rest/v2
.Replace
$SERVICE_HOSTNAME:$PORT
with the hostname and port where Data Grid is available on the network.
Additional resources
17.7. Adding caches to Cache service pods
Cache service pods include a default cache configuration with recommended settings. This default cache lets you start using Data Grid without the need to create caches.
Because the default cache provides recommended settings, you should create caches only as copies of the default. If you want multiple custom caches you should create Data Grid service pods instead of Cache service pods.
Procedure
- Access the Data Grid Console and provide a copy of the default configuration in XML or JSON format.
Use the Data Grid CLI to create a copy from the default cache as follows:
[//containers/default]> create cache --template=default mycache
17.7.1. Default cache configuration
This topic describes default cache configuration for Cache service pods.
<distributed-cache name="default" mode="SYNC" owners="2"> <memory storage="OFF_HEAP" max-size="<maximum_size_in_bytes>" when-full="REMOVE" /> <partition-handling when-split="ALLOW_READ_WRITES" merge-policy="REMOVE_ALL"/> </distributed-cache>
Default caches:
- Use synchronous distribution to store data across the cluster.
- Create two replicas of each entry on the cluster.
- Store cache entries as bytes in native memory (off-heap).
- Define the maximum size for the data container in bytes. Data Grid Operator calculates the maximum size when it creates pods.
- Evict cache entries to control the size of the data container. You can enable automatic scaling so that Data Grid Operator adds pods when memory usage increases instead of removing entries.
- Use a conflict resolution strategy that allows read and write operations for cache entries, even if segment owners are in different partitions.
- Specify a merge policy that removes entries from the cache when Data Grid detects conflicts.