Running Data Grid on OpenShift

Procedure

1. Log in to the OpenShift Web Console.
2. Navigate to OperatorHub.
3. Find and select Data Grid Operator.
4. Select Install and continue to Create Operator Subscription.

5. Specify options for your subscription.

   Installation Mode

   You can install Data Grid Operator into a Specific namespace or All namespaces.

   Update Channel

   Get updates for Data Grid Operator 8.1.x.

   Approval Strategies

   Automatically install updates from the 8.1.x channel or require approval before installation.

6. Select Subscribe to install Data Grid Operator.
7. Navigate to Installed Operators to verify the Data Grid Operator installation.

Procedure

1. Set up projects.

   1. Create a project for Data Grid Operator.

   2. If you want Data Grid Operator to control a specific Data Grid cluster only, create a project for that cluster.

      $ oc new-project ${INSTALL_NAMESPACE} 1
      $ oc new-project ${WATCH_NAMESPACE} 2

      1

      Creates a project into which you install Data Grid Operator.

      2

      Optionally creates a project for a specific Data Grid cluster if you do not want Data Grid Operator to watch all projects.

2. Create an OperatorGroup resource.

   Control all Data Grid clusters

   $ oc apply -f - << EOF
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
    name: datagrid
    namespace: ${INSTALL_NAMESPACE}
   EOF

   Control a specific Data Grid cluster

   $ oc apply -f - << EOF
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
    name: datagrid
    namespace: ${INSTALL_NAMESPACE}
   spec:
    targetNamespaces:
    - ${WATCH_NAMESPACE}
   EOF

3. Create a subscription for Data Grid Operator.

   $ oc apply -f - << EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
    name: datagrid-operator
    namespace: ${INSTALL_NAMESPACE}
   spec:
    channel: 8.1.x
    installPlanApproval: Automatic 1
    name: datagrid
    source: redhat-operators
    sourceNamespace: openshift-marketplace
   EOF

   1

   Specify Manual if you want to manually approve updates from the 8.1.x channel.

4. Verify the installation.

   $ oc get pods -n ${INSTALL_NAMESPACE}
   NAME                                   READY   STATUS
   infinispan-operator-<id>               1/1     Running

Procedure

1. Specify the number of Data Grid nodes in the cluster with spec.replicas in your Infinispan CR.

   For example, create a cr_minimal.yaml file as follows:

   $ cat > cr_minimal.yaml<<EOF
   apiVersion: infinispan.org/v1
   kind: Infinispan
   metadata:
     name: example-infinispan
   spec:
     replicas: 2
   EOF

2. Apply your Infinispan CR.

   $ oc apply -f cr_minimal.yaml

3. Watch Data Grid Operator create the Data Grid nodes.

   $ oc get pods -w
   
   NAME                        READY  STATUS              RESTARTS   AGE
   example-infinispan-1        0/1    ContainerCreating   0          4s
   example-infinispan-2        0/1    ContainerCreating   0          4s
   example-infinispan-3        0/1    ContainerCreating   0          5s
   infinispan-operator-0       1/1    Running             0          3m
   example-infinispan-3        1/1    Running             0          8s
   example-infinispan-2        1/1    Running             0          8s
   example-infinispan-1        1/1    Running             0          8s

Procedure

1. Create an Infinispan CR.

   apiVersion: infinispan.org/v1
   kind: Infinispan
   metadata:
     name: example-infinispan
   spec:
     replicas: 2
     service:
       type: Cache 1

   1

   Creates nodes Cache service nodes. This is the default for the Infinispan CR.

2. Apply your Infinispan CR to create the cluster.

Procedure

1. Specify DataGrid as the value for spec.service.type in your Infinispan CR.

   apiVersion: infinispan.org/v1
   kind: Infinispan
   metadata:
     name: example-infinispan
   spec:
     replicas: 2
     service:
       type: DataGrid

   Note

   You cannot change the spec.service.type field after you create nodes. To change the service type, you must delete the existing nodes and create new ones.

2. Configure nodes with any other Data Grid service resources.
3. Apply your Infinispan CR to create the cluster.

Procedure

1. Open your Infinispan CR for editing.
2. Add any labels that you want Data Grid Operator to attach to resources with metadata.annotations.

3. Add values for your labels with metadata.labels.

   apiVersion: infinispan.org/v1
   kind: Infinispan
   metadata:
     annotations:
       # Add labels that you want to attach to services.
       infinispan.org/targetLabels: svc-label1, svc-label2
       # Add labels that you want to attach to pods.
       infinispan.org/podTargetLabels: pod-label1, pod-label2
     labels:
       # Add values for your labels.
       svc-label1: svc-value1
       svc-label2: svc-value2
       pod-label1: pod-value1
       pod-label2: pod-value2
       # The operator does not attach these labels to resources.
       my-label: my-value
       environment: development

4. Apply your Infinispan CR.

Procedure

1. Include spec.expose in your Infinispan CR.

2. Specify LoadBalancer as the service type with spec.expose.type.

   spec:
     ...
     expose:
       type: LoadBalancer 1
       nodePort: 30000 2

   1

   Exposes Data Grid on the network through a load balancer service on port 11222.

   2

   Optionally defines a node port to which the load balancer service forwards traffic.

3. Apply the changes.

4. Verify that the -external service is available.

   $ oc get services | grep external
   
   NAME                         TYPE            CLUSTER-IP    EXTERNAL-IP   PORT(S)
   example-infinispan-external  LoadBalancer    192.0.2.24    hostname.com  11222/TCP

Procedure

1. Include spec.expose in your Infinispan CR.

2. Specify NodePort as the service type with spec.expose.type.

   spec:
     ...
     expose:
       type: NodePort 1
       nodePort: 30000 2

   1

   Exposes Data Grid on the network through a node port service.

   2

   Defines the port where Data Grid is exposed. If you do not define a port, the platform selects one.

3. Apply the changes.

4. Verify that the -external service is available.

   $ oc get services | grep external
   
   NAME                         TYPE            CLUSTER-IP       EXTERNAL-IP   PORT(S)
   example-infinispan-external  NodePort        192.0.2.24       <none>        11222:30000/TCP

Procedure

1. Include spec.expose in your Infinispan CR.
2. Specify Route as the service type with spec.expose.type.

3. Optionally add a hostname with spec.expose.host.

   spec:
     ...
     expose:
       type: Route 1
       host: www.example.org 2

   1

   Exposes Data Grid on the network through an OpenShift Route.

   2

   Optionally specifies the hostname where Data Grid is exposed.

4. Apply the changes.

5. Verify that the route is available.

   $ oc get routes
   
   NAME                 CLASS    HOSTS   ADDRESS   PORTS   AGE
   example-infinispan   <none>   *                 443     73s

Procedure

1. Log in to an OpenShift cluster.

2. Create a service account.

   For example, create a service account at LON:

   $ oc create sa lon
   serviceaccount/lon created

3. Add the view role to the service account with the following command:

   $ oc policy add-role-to-user view system:serviceaccount:<namespace>:lon

4. Repeat the preceding steps on your other OpenShift clusters.

Procedure

1. Log in to an OpenShift cluster.

2. Add the service account token for a backup location with the following command:

   $ oc create secret generic <token-name> --from-literal=token=<token>

   For example, log in to the OpenShift cluster at NYC and create a lon-token secret as follows:

   $ oc create secret generic lon-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...

3. Repeat the preceding steps on your other OpenShift clusters.

Procedure

1. Create an Infinispan CR for each Data Grid cluster.
2. Specify a matching name for each Data Grid cluster with metadata.name.
3. Specify the name of the local site with spec.service.sites.local.name.
4. Set the expose service type for the local site with spec.service.sites.local.expose.type.

5. Provide the name, URL, and secret for each Data Grid cluster that acts as a backup location with spec.service.sites.locations.

   The following are example Infinispan CR definitions for LON and NYC:

   • LON

     apiVersion: infinispan.org/v1
     kind: Infinispan
     metadata:
       name: example-infinispan
     spec:
       replicas: 3
       service:
         type: DataGrid
         sites:
           local:
             name: LON
             expose:
               type: LoadBalancer
           locations:
             - name: LON
               url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443
               secretName: lon-token
             - name: NYC
               url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443
               secretName: nyc-token

   • NYC

     apiVersion: infinispan.org/v1
     kind: Infinispan
     metadata:
       name: example-infinispan
     spec:
       replicas: 2
       service:
         type: DataGrid
         sites:
           local:
             name: NYC
             expose:
               type: LoadBalancer
           locations:
             - name: NYC
               url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443
               secretName: nyc-token
             - name: LON
               url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443
               secretName: lon-token

6. Adjust logging levels for cross-site replication as follows:

   ...
     logging:
       categories:
         org.jgroups.protocols.TCP: error
         org.jgroups.protocols.relay.RELAY2: fatal

   The preceding configuration decreases logging for JGroups TCP and RELAY2 protocols to reduce excessive messages about cluster backup operations, which can result in a large number of log files that use container storage.

7. Configure nodes with any other Data Grid service resources.
8. Apply the Infinispan CRs.

9. Check node logs to verify that Data Grid clusters form a cross-site view, for example:

   $ oc logs example-infinispan-0 | grep x-site
   
   INFO  [org.infinispan.XSITE] (jgroups-5,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC]
   INFO  [org.infinispan.XSITE] (jgroups-7,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC, LON]

Procedure

1. Define a Secret object type that provides valid user credentials for accessing Data Grid service clusters in a StringData map.

   For example, create an basic-auth.yaml file that provides credentials for the developer user as follows:

   apiVersion: v1
   stringData:
     username: developer 1
     password: G8ZdJvSaY3lOOwfM 2
   kind: Secret
   metadata:
     name: basic-auth 3
   type: Opaque

   1

   Names a user that can create caches.

   2

   Specifies the password that corresponds to the user.

   3

   Specifies a name for the secret.

2. Create a secret from the file, as in the following example:

   $ oc apply -f basic-auth.yaml

Procedure

1. Create a Cache CR that contains the XML cache definition you want to create.

   apiVersion: infinispan.org/v2alpha1
   kind: Cache
   metadata:
     name: mycachedefinition 1
   spec:
     adminAuth: 2
       secretName: basic-auth
     clusterName: example-infinispan 3
     name: mycache 4
     template: <infinispan><cache-container><distributed-cache name="mycache" mode="SYNC"><persistence><file-store/></persistence></distributed-cache></cache-container></infinispan> 5

   1

   Names the Cache CR.

   2

   Specifies a secret that provides credentials with username and password keys or an override for custom credentials secrets.

   3

   Specifies the name of the target Data Grid cluster where you want Data Grid Operator to create the cache.

   4

   Names the cache on the Data Grid cluster.

   5

   Specifies the XML cache definition to create the cache. Note that the name attribute is ignored. Only spec.name applies to the resulting cache.

2. Apply the Cache CR, for example:

   $ oc apply -f mycache.yaml
   cache.infinispan.org/mycachedefinition created

Procedure

1. Create a Cache CR that specifies the name of the template you want to use.

   For example, the following CR creates a cache named "mycache" that uses the org.infinispan.DIST_SYNC cache configuration template:

   apiVersion: infinispan.org/v2alpha1
   kind: Cache
   metadata:
     name: mycachedefinition 1
   spec:
     adminAuth: 2
       secretName: basic-auth
     clusterName: example-infinispan 3
     name: mycache 4
     templateName: org.infinispan.DIST_SYNC 5

   1

   Names the Cache CR.

   2

   Specifies a secret that provides credentials with username and password keys or an override for custom credentials secrets.

   3

   Specifies the name of the target Data Grid cluster where you want Data Grid Operator to create the cache.

   4

   Names the Data Grid cache instance.

   5

   Specifies the infinispan.org cache configuration template to create the cache.

2. Apply the Cache CR, for example:

   $ oc apply -f mycache.yaml
   cache.infinispan.org/mycachedefinition created

Procedure

1. Create cache configurations with identical names for each site.

   Cache configurations at each site can use different cache modes and backup strategies. Data Grid replicates data based on cache names.

2. Configure backup locations to go offline automatically with the take-offline element.

   1. Set the amount of time, in milliseconds, before backup locations go offline with the min-wait attribute.
3. Define any other valid cache configuration.

4. Add backup locations to the named cache on all sites in the global cluster.

   For example, if you add LON as a backup for NYC you should add NYC as a backup for LON.

Procedure

1. Access the console from any browser at $SERVICE_HOSTNAME:$PORT.

   Replace $SERVICE_HOSTNAME:$PORT with the hostname and port where Data Grid is available on the network.

2. Enter your Data Grid credentials when prompted.

Procedure

1. Create an authentication secret that contains Data Grid credentials so that Prometheus can authenticate with your Data Grid cluster.

   apiVersion: v1
   stringData:
     username: developer 1
     password: dIRs5cAAsHIeeRIL 2
   kind: Secret
   metadata:
     name: basic-auth
   type: Opaque

   1

   Specifies an application user. developer is the default.

   2

   Specifies the corresponding password.

2. Add the authentication secret to your Prometheus namespace.

   $ oc apply -f basic-auth.yaml

3. Create a service monitor that configures Prometheus to monitor your Data Grid cluster.

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus
     name: datagrid-monitoring 1
     namespace: infinispan-monitoring 2
   spec:
     endpoints:
       - targetPort: 11222 3
         path: /metrics 4
         honorLabels: true
         basicAuth:
           username:
             key: username
             name: basic-auth 5
           password:
             key: password
             name: basic-auth
         interval: 30s
         scrapeTimeout: 10s
         scheme: https 6
         tlsConfig:
           insecureSkipVerify: true
           serverName: example-infinispan 7
     namespaceSelector:
       matchNames:
         - infinispan 8
     selector:
       matchLabels:
         app: infinispan-service
         clusterName: example-infinispan 9

   1

   Names the service monitor instances.

   2

   Specifies the namespace of your Prometheus stack.

   3

   Sets the port of 11222 for the Data Grid metrics endpoint.

   4

   Sets the path where Data Grid exposes metrics.

   5

   Specifies the authentication secret with Data Grid credentials.

   6

   Specifies that Data Grid clusters use endpoint encryption.

   7

   Specifies the Common Name (CN) of the TLS certificate for Data Grid encryption. If you use an OpenShift service certificate, the CN matches the metadata.name resource for your Data Grid cluster.

   8

   Specifies the namespace of your Data Grid cluster.

   9

   Specifies the name of your Data Grid cluster.

4. Add the service monitor instance to your Prometheus namespace.

   $ oc apply -f service-monitor.yaml

Procedure

1. Add the spec.affinity block to your Infinispan CR.
2. Configure anti-affinity strategies as necessary.
3. Apply your Infinispan CR.

Procedure

1. Specify logging configuration with spec.logging in your Infinispan CR and then apply the changes.

   spec:
     ...
     logging: 1
       categories: 2
         org.infinispan: debug 3
         org.jgroups: debug

   1

   Configures Data Grid logging.

   2

   Adds logging categories.

   3

   Names logging categories and levels.

   Note

   The root logging category is org.infinispan and is INFO by default.

2. Retrieve logs from Data Grid nodes as required.

   $ oc logs -f $POD_NAME