Data Grid Server Guide

Data Grid Server requires a Java Virtual Machine. See the Data Grid Supported Configurations for details on supported versions.

The Data Grid server distribution is an archive of Java libraries (JAR files), configuration files, and a data directory.

Install the Data Grid Server distribution on a host system.

$ unzip redhat-datagrid-8.2.3-server.zip

The resulting directory is your $RHDG_HOME.

Run Data Grid Server instances in a Java Virtual Machine (JVM) on any supported host.

Data Grid Server is running successfully when it logs the following messages:

ISPN080004: Protocol SINGLE_PORT listening on 127.0.0.1:11222
ISPN080034: Server '...' listening on http://127.0.0.1:11222
ISPN080001: Data Grid Server <version> started in <mm>ms

Add Data Grid user credentials and assign permissions to control access to data.

Data Grid server installations use a property realm to authenticate users for the Hot Rod and REST endpoints. This means you need to create at least one user before you can access Data Grid.

By default, users also need roles with permissions to access caches and interact with Data Grid resources. You can assign roles to users individually or add users to groups that have role permissions.

You create users and assign roles with the user command in the Data Grid command line interface (CLI).

Data Grid nodes on the same network automatically discover each other and form clusters.

Complete this procedure to observe cluster discovery with the MPING protocol in the default TCP stack with locally running Data Grid Server instances. If you want to adjust cluster transport for custom network requirements, see the documentation for setting up Data Grid clusters.

Data Grid also logs the following messages when nodes join clusters:

INFO  [org.infinispan.CLUSTER] (jgroups-11,<server_hostname>)
ISPN000094: Received new cluster view for channel cluster:
[<server_hostname>|3] (2) [<server_hostname>, <server2_hostname>]

INFO  [org.infinispan.CLUSTER] (jgroups-11,<server_hostname>)
ISPN100000: Node <server2_hostname> joined the cluster

Stop individually running servers or bring down clusters gracefully.

ISPN080002: Data Grid Server stopping
ISPN000080: Disconnecting JGroups channel cluster
ISPN000390: Persisted state, version=<$version> timestamp=YYYY-MM-DDTHH:MM:SS
ISPN080003: Data Grid Server stopped

Data Grid Server uses the following folders on the host filesystem under $RHDG_HOME:

├── bin
├── boot
├── docs
├── lib
├── server
└── static

├── server
├── server1
├── server2
├── server3
└── server4

├── server
│   ├── conf
│   ├── data
│   ├── lib
│   └── log

Data Grid Server multiplexes endpoints to a single TCP/IP port and automatically detects protocols of inbound client requests. You can configure how Data Grid Server binds to network interfaces to listen for client requests.

<!-- Selects a specific IPv4 address, which can be public, private, or loopback.
     This is the default network interface for Data Grid Server. -->
<interfaces>
  <interface name="public">
    <inet-address value="${infinispan.bind.address:127.0.0.1}"/>
  </interface>
</interfaces>

<!-- Selects an IP address in an IPv4 or IPv6 loopback address block. -->
<interfaces>
  <interface name="public">
    <loopback/>
  </interface>
</interfaces>

<!-- Selects an IP address in an IPv4 or IPv6 non-loopback address block. -->
<interfaces>
  <interface name="public">
    <non-loopback/>
  </interface>
</interfaces>

<!-- Uses the `INADDR_ANY` wildcard address which means Data Grid Server
     listens for inbound client requests on all interfaces. -->
<interfaces>
  <interface name="public">
    <any-address/>
  </interface>
</interfaces>

<!-- Selects a link-local IP address in an IPv4 or IPv6 address block. -->
<interfaces>
  <interface name="public">
    <link-local/>
  </interface>
</interfaces>

<!-- Selects a site-local (private) IP address in an IPv4 or IPv6 address block. -->
<interfaces>
  <interface name="public">
    <site-local/>
  </interface>
</interfaces>

Match and fallback strategies

Data Grid Server can enumerate all network interfaces on the host system and bind to an interface, host, or IP address that matches a value, which can include regular expressions for additional flexibility.

<!-- Selects an IP address that is assigned to a matching host name. -->
<interfaces>
  <interface name="public">
    <match-host value="my_host_name"/>
  </interface>
</interfaces>

<!--Selects an IP address assigned to a matching network interface. -->
<interfaces>
  <interface name="public">
    <match-interface value="eth0"/>
  </interface>
</interfaces>

<!-- Selects an IP address that matches a regular expression. -->
<interfaces>
  <interface name="public">
    <match-address value="132\..*"/>
  </interface>
</interfaces>

<!-- Includes multiple strategies that Data Grid Server tries in the
     declared order until it finds a match. -->
<interfaces>
  <interface name="public">
    <match-host value="my_host_name"/>
    <match-address value="132\..*"/>
    <any-address/>
  </interface>
</interfaces>

Socket bindings map endpoint connectors to server interfaces and ports.

By default, Data Grid servers provide the following socket bindings:

<socket-bindings default-interface="public" port-offset="${infinispan.socket.binding.port-offset:0}">
    <socket-binding name="default" port="${infinispan.bind.port:11222}"/>
    <socket-binding name="memcached" port="11221"/>
</socket-bindings>

To override the default interface for socket-binding declarations, specify the interface attribute.

For example, you add an interface declaration named "private":

<interfaces>
  ...
  <interface name="private">
    <inet-address value="10.1.2.3"/>
  </interface>
</interfaces>

You can then specify interface="private" in a socket-binding declaration to bind to the private IP address, as follows:

<socket-bindings default-interface="public" port-offset="${infinispan.socket.binding.port-offset:0}">
  ...
  <socket-binding name="private_binding" interface="private" port="1234"/>
</socket-bindings>

You can use the server -b switch or the infinispan.bind.address system property to bind to a different address.

For example, bind the public interface to 127.0.0.2 as follows:

$ bin/server.sh -b 127.0.0.2

bin\server.bat -b 127.0.0.2

Configure port offsets with Data Grid servers when running multiple instances on the same host. The default port offset is 0.

Use the -o switch with the Data Grid CLI or the infinispan.socket.binding.port-offset system property to set port offsets.

For example, start a server instance with an offset of 100 as follows. With the default configuration, this results in the Data Grid server listening on port 11322.

$ bin/server.sh -o 100

bin\server.bat -o 100

Data Grid endpoints expose the CacheManager interface over different connector protocols so you can remotely access data and perform operations to manage and maintain Data Grid clusters.

You can define multiple endpoint connectors on different socket bindings.

You configure Data Grid server endpoints with connector declarations that specify socket bindings, authentication mechanisms, and encryption configuration.

The default endpoint connector configuration is as follows:

<endpoints socket-binding="default" security-realm="default"/>

Declaring an empty <endpoints/> element implicitly enables the Hot Rod and REST connectors.

It is possible to have multiple endpoints bound to different sockets. These can use different security realms and offer different authentication and encryption configurations. The following configuration enables two endpoints on distinct socket bindings, each one with a dedicated security realm. Additionally, the public endpoint disables administrative features, such as the console and CLI.

<endpoints socket-binding="public" security-realm="application-realm" admin="false">
   <hotrod-connector/>
   <rest-connector/>
</endpoints>
<endpoints socket-binding="private" security-realm="management-realm">
   <hotrod-connector/>
   <rest-connector/>
</endpoints>

<hotrod-connector name="hotrod">
  <topology-state-transfer />
  <authentication>
    <!-- Hot Rod endpoint authentication configuration. -->
  </authentication>
  <encryption>
    <!-- Hot Rod endpoint SSL/TLS encryption configuration. -->
  </encryption>
</hotrod-connector>

<rest-connector name="rest">
  <authentication>
    <!-- REST endpoint authentication configuration. -->
  </authentication>
  <cors-rules>
    <!-- Cross-Origin Resource Sharing (CORS) rules. -->
  </cors-rules>
  <encryption>
    <!-- REST endpoint SSL/TLS encryption configuration. -->
  </encryption>
</rest-connector>

Data Grid Server exposes endpoints on your network for remote client access.

Data Grid Server exposes multiple protocols through a single TCP port, which is 11222 by default. Handling multiple protocols with a single port simplifies configuration and reduces management complexity when deploying Data Grid clusters. Using a single port also enhances security by minimizing the attack surface on the network.

Data Grid Server handles HTTP/1.1, HTTP/2, and Hot Rod protocol requests from clients via the single port in different ways.

# firewall-cmd --add-port=11222/tcp --permanent
success
# firewall-cmd --list-ports | grep 11222
11222/tcp

Property realms use property files to define users and groups.

users.properties maps usernames to passwords in plain-text format. Passwords can also be pre-digested if you use the DIGEST-MD5 SASL mechanism or Digest HTTP mechanism.

myuser=a_password
user2=another_password

groups.properties maps users to roles.

myuser=supervisor,reader,writer
user2=supervisor

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
         <!-- Defines groups as roles for server authorization. -->
         <properties-realm groups-attribute="Roles">
            <!-- Specifies the properties file that holds usernames and passwords. -->
            <!-- The plain-text="true" attribute stores passwords in plain text. -->
            <user-properties path="users.properties"
                             relative-to="infinispan.server.config.path"
                             plain-text="true"/>
            <!-- Specifies the properties file that defines roles for users. -->
            <group-properties path="groups.properties"
                              relative-to="infinispan.server.config.path"/>
         </properties-realm>
      </security-realm>
   </security-realms>
</security>

LDAP realms connect to LDAP servers, such as OpenLDAP, Red Hat Directory Server, Apache Directory Server, or Microsoft Active Directory, to authenticate users and obtain membership information.

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
        <!-- Names an LDAP realm and specifies connection properties. -->
        <ldap-realm name="ldap"
                    url="ldap://my-ldap-server:10389"
                    principal="uid=admin,ou=People,dc=infinispan,dc=org"
                    credential="strongPassword"
                    connection-timeout="3000"
                    read-timeout="30000"
                    connection-pooling="true"
                    referral-mode="ignore"
                    page-size="30"
                    direct-verification="true">
            <!-- Defines how principals are mapped to LDAP entries. -->
            <identity-mapping rdn-identifier="uid"
                              search-dn="ou=People,dc=infinispan,dc=org"
                              search-recursive="false">
               <!-- Retrieves all the groups of which the user is a member. -->
               <attribute-mapping>
                  <attribute from="cn"
                             to="Roles"
                             filter="(&amp;(objectClass=groupOfNames)(member={1}))"
                             filter-dn="ou=Roles,dc=infinispan,dc=org"/>
               </attribute-mapping>
            </identity-mapping>
         </ldap-realm>
      </security-realm>
   </security-realms>
</security>

As an alternative to verifying user credentials with the direct-verification attribute, you can specify an LDAP password with the user-password-mapper element.

The rdn-identifier attribute specifies an LDAP attribute that finds the user entry based on a provided identifier, which is typically a username; for example, the uid or sAMAccountName attribute. Add search-recursive="true" to the configuration to search the directory recursively. By default, the search for the user entry uses the (rdn_identifier={0}) filter. Specify a different filter with the filter-name attribute.

The attribute-mapping element retrieves all the groups of which the user is a member. There are typically two ways in which membership information is stored:

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
         <ldap-realm name="ldap"
                     url="ldap://${org.infinispan.test.host.address}:10389"
                     principal="uid=admin,ou=People,dc=infinispan,dc=org"
                     credential="strongPassword">
            <name-rewriter>
               <!-- Defines a rewriter that extracts the username from the principal using a regular expression. -->
               <regex-principal-transformer name="domain-remover"
                                            pattern="(.*)@INFINISPAN\.ORG"
                                            replacement="$1"/>
            </name-rewriter>
            <identity-mapping rdn-identifier="uid"
                              search-dn="ou=People,dc=infinispan,dc=org">
               <attribute-mapping>
                  <attribute from="cn" to="Roles"
                             filter="(&amp;(objectClass=groupOfNames)(member={1}))"
                             filter-dn="ou=Roles,dc=infinispan,dc=org" />
               </attribute-mapping>
               <user-password-mapper from="userPassword" />
            </identity-mapping>
         </ldap-realm>
      </security-realm>
   </security-realms>
</security>

Token realms use external services to validate tokens and require providers that are compatible with RFC-7662 (OAuth2 Token Introspection), such as Red Hat SSO.

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
         <!-- Specifies the URL of the authentication server. -->
         <token-realm name="token"
                      auth-server-url="https://oauth-server/auth/">
            <!-- Specifies the URL of the token introspection endpoint. -->
            <oauth2-introspection
                    introspection-url="https://oauth-server/auth/realms/infinispan/protocol/openid-connect/token/introspect"
                    client-id="infinispan-server"
                    client-secret="1fdca4ec-c416-47e0-867a-3d471af7050f"/>
         </token-realm>
      </security-realm>
   </security-realms>
</security>

Trust store realms use certificates, or certificates chains, that verify Data Grid Server and client identities when they negotiate connections.

When you configure Data Grid Server to use a client trust store, you must configure endpoints to use the following authentication mechanisms:

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
         <server-identities>
            <ssl>
               <!-- Provides an SSL/TLS identity with a keystore that
                    contains server certificates. -->
               <keystore path="server.p12"
                         relative-to="infinispan.server.config.path"
                         keystore-password="secret"
                         alias="server"/>
               <!-- Configures a trust store that contains client certificates
                    or part of a certificate chain. -->
               <truststore path="trust.p12"
                           relative-to="infinispan.server.config.path"
                           password="secret"/>
            </ssl>
         </server-identities>
         <!-- Authenticates client certificates against the trust store.
              If you configure this, the trust store must contain
              the public certificates for all clients. -->
         <truststore-realm/>
      </security-realm>
   </security-realms>
</security>
<!-- Configures Data Grid Server to require client certificates
     with the "require-ssl-client-auth" attribute. -->
<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
           xmlns="urn:infinispan:server:12.1"
           socket-binding="default"
           security-realm="default"
           require-ssl-client-auth="true">
   <hotrod-connector>
     <!-- Configures the Hot Rod endpoint for client certificate authentication. -->
     <authentication>
       <sasl mechanisms="EXTERNAL"
             server-name="infinispan"
             qop="auth"/>
     </authentication>
   </hotrod-connector>
   <rest-connector>
      <!-- Configures the REST endpoint for client certificate authentication. -->
      <authentication mechanisms="CLIENT_CERT"/>
   </rest-connector>
</endpoints>

Data Grid servers automatically configure authentication mechanisms based on the security realm that you assign to endpoints.

<security-realm name="default">
  <properties-realm groups-attribute="Roles">
    <user-properties path="users.properties"
                     relative-to="infinispan.server.config.path"
                     plain-text="true"/>
    <group-properties path="groups.properties"
                      relative-to="infinispan.server.config.path" />
  </properties-realm>
</security-realm>

The endpoints configuration assigns the "default" security realm to the Hot Rod and REST connectors, as follows:

<endpoints socket-binding="default" security-realm="default">
  <hotrod-connector name="hotrod"/>
  <rest-connector name="rest"/>
</endpoints>

As a result of the preceding configuration, Data Grid servers require authentication with a mechanism that the property realm supports.

Explicitly configure Hot Rod connector authentication to override the default SASL authentication mechanisms that Data Grid servers use for security realms.

<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:12.1
           https://infinispan.org/schemas/infinispan-server-12.1.xsd"
           xmlns="urn:infinispan:server:12.1"
           socket-binding="default"
           security-realm="default">
   <hotrod-connector>
      <authentication>
         <!-- Specifies SASL mechanisms to use for authentication. -->
         <!-- Defines the name that the server declares to clients. -->
         <sasl mechanisms="SCRAM-SHA-512 SCRAM-SHA-384 SCRAM-SHA-256
                           SCRAM-SHA-1 DIGEST-SHA-512 DIGEST-SHA-384
                           DIGEST-SHA-256 DIGEST-SHA DIGEST-MD5 PLAIN"
               server-name="infinispan"
               qop="auth"/>
      </authentication>
   </hotrod-connector>
</endpoints>

<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
           xmlns="urn:infinispan:server:12.1"
           socket-binding="default"
           security-realm="default">
   <hotrod-connector>
      <authentication>
         <!-- Enables the GSSAPI and GS2-KRB5 mechanisms for Kerberos authentication. -->
         <!-- Defines the server name, which is equivalent to the Kerberos service name, and specifies the Kerberos identity for the server. -->
         <sasl mechanisms="GSSAPI GS2-KRB5"
               server-name="datagrid"
               server-principal="hotrod/datagrid@INFINISPAN.ORG"/>
      </authentication>
   </hotrod-connector>
</endpoints>

<hotrod-connector socket-binding="hotrod" cache-container="default">
   <authentication security-realm="ApplicationRealm">
      <!-- Specifies multiple SASL authentication mechanisms for the Hot Rod connector. -->
      <sasl server-name="myhotrodserver"
            mechanisms="PLAIN DIGEST-MD5 GSSAPI EXTERNAL"
            qop="auth">
         <!-- Defines policies for SASL mechanisms. -->
         <policy>
            <no-active value="true" />
            <no-anonymous value="true" />
            <no-plain-text value="true" />
         </policy>
      </sasl>
   </authentication>
</hotrod-connector>

Explicitly configure REST connector authentication to override the default HTTP authentication mechanisms that Data Grid servers use for security realms.

<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
           xmlns="urn:infinispan:server:12.1"
           socket-binding="default"
           security-realm="default">
   <rest-connector>
      <!-- Specifies SASL mechanisms to use for authentication. -->
      <authentication mechanisms="DIGEST BASIC"/>
   </rest-connector>
</endpoints>

<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
           xmlns="urn:infinispan:server:12.1"
           socket-binding="default"
           security-realm="default">
   <rest-connector>
      <!-- Enables the `SPENGO` mechanism for Kerberos authentication and specifies an identity for the server. -->
      <authentication mechanisms="SPNEGO"
                      server-principal="HTTP/localhost@INFINISPAN.ORG"/>
   </rest-connector>
</endpoints>

In local development environments or on isolated networks you can configure Data Grid to allow unauthenticated client requests.

When you disable user authentication you should also disable authorization in your Data Grid security configuration.

Add keystores to Data Grid Server and configure it to present SSL/TLS certificates that verify its identity to clients. If a security realm contains TLS/SSL identities, it encrypts any connections to Data Grid Server endpoints that use that security realm.

Data Grid Server supports the following keystore formats: JKS, JCEKS, PKCS12, BKS, BCFKS, and UBER.

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
         <server-identities>
            <ssl>
               <!-- Adds a keystore that contains server certificates
                    that provide SSL/TLS identities to clients. -->
               <keystore path="server.pfx"
                         relative-to="infinispan.server.config.path"
                         keystore-password="secret"
                         alias="rhdg-server"/>
            </ssl>
         </server-identities>
      </security-realm>
   </security-realms>
</security>

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
         <server-identities>
            <ssl>
               <!-- Generates a keystore that includes a self-signed certificate with the specified hostname. -->
               <keystore path="server.p12"
                         relative-to="infinispan.server.config.path"
                         keystore-password="secret"
                         alias="server"
                         generate-self-signed-certificate-host="localhost"/>
            </ssl>
         </server-identities>
      </security-realm>
   </security-realms>
</security>

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1
          https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
         <server-identities>
            <ssl>
               <keystore path="server.p12"
                         relative-to="infinispan.server.config.path"
                         keystore-password="secret" alias="server"/>
               <!-- Configures Data Grid Server to use specific TLS versions and cipher suites. -->
               <engine enabled-protocols="TLSv1.3"
                       enabled-ciphersuites="TLS_AES_256_GCM_SHA384 TLS_AES_128_GCM_SHA256 TLS_AES_128_CCM_8_SHA256"/>
            </ssl>
         </server-identities>
      </security-realm>
   </security-realms>
</security>

Configure Data Grid Server to use mutual TLS to secure client connections.

You can configure Data Grid to verify client identities from certificates in a trust store in two ways:

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
         <server-identities>
            <ssl>
               <!-- Provides an SSL/TLS identity with a keystore that
                    contains server certificates. -->
               <keystore path="server.p12"
                         relative-to="infinispan.server.config.path"
                         keystore-password="secret"
                         alias="server"/>
               <!-- Configures a trust store that contains client certificates
                    or part of a certificate chain. -->
               <truststore path="trust.p12"
                           relative-to="infinispan.server.config.path"
                           password="secret"/>
            </ssl>
         </server-identities>
         <!-- Authenticates client certificates against the trust store.
              If you configure this, the trust store must contain
              the public certificates for all clients. -->
         <truststore-realm/>
      </security-realm>
   </security-realms>
</security>
<!-- Configures Data Grid Server to require client certificates
     with the "require-ssl-client-auth" attribute. -->
<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
           xmlns="urn:infinispan:server:12.1"
           socket-binding="default"
           security-realm="default"
           require-ssl-client-auth="true">
   <hotrod-connector>
     <!-- Configures the Hot Rod endpoint for client certificate authentication. -->
     <authentication>
       <sasl mechanisms="EXTERNAL"
             server-name="infinispan"
             qop="auth"/>
     </authentication>
   </hotrod-connector>
   <rest-connector>
      <!-- Configures the REST endpoint for client certificate authentication. -->
      <authentication mechanisms="CLIENT_CERT"/>
   </rest-connector>
</endpoints>

Enabling client certificate authentication means you do not need to specify Data Grid user credentials in client configuration, which means you must associate roles with the Common Name (CN) field in the client certificate(s).

Kerberos identities use keytab files that contain service principal names and encrypted keys, derived from Kerberos passwords.

In most cases, you create unique principals for the Hot Rod and REST connectors. For example, you have a "datagrid" server in the "INFINISPAN.ORG" domain. In this case you should create the following service principals:

The following example configures Kerberos identities for Data Grid Server:

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <security-realms>
      <security-realm name="default">
         <server-identities>
            <!-- Specifies a keytab file that provides a Kerberos identity for the Hot Rod connector. -->
            <!-- Names the Kerberos service principal for the Hot Rod connector. -->
            <!-- The required="true" attribute specifies that the keytab file must be present when the server starts. -->
            <kerberos keytab-path="hotrod.keytab"
                      principal="hotrod/datagrid@INFINISPAN.ORG"
                      required="true"/>
            <!-- Specifies a keytab file that provides a Kerberos identity for the REST connector. -->
            <!-- Names the Kerberos service principal for the REST connector. -->
            <kerberos keytab-path="http.keytab"
                      principal="HTTP/localhost@INFINISPAN.ORG"
                      required="true"/>
         </server-identities>
      </security-realm>
   </security-realms>
</security>

Create keystores that encrypt credential for Data Grid Server access.

A credential keystore contains at least one alias that is associated with an encrypted password. After you create a keystore, you specify the alias in a connection configuration such as a database connection pool. Data Grid Server then decrypts the password for that alias from the keystore when the service attempts authentication.

You can create as many credential keystores with as many aliases as required.

Review example configurations for credential keystores in Data Grid Server configuration.

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <!-- Uses a keystore to manage server credentials. -->
   <credential-stores>
      <!-- Specifies the name and filesystem location of a keystore. -->
      <credential-store name="credentials" path="credentials.pfx">
         <!-- Specifies the password for the credential keystore. -->
         <clear-text-credential clear-text="secret1234!"/>
      </credential-store>
   </credential-stores>
</security>

<data-sources xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
              xmlns="urn:infinispan:server:12.1">
   <data-source name="postgres" jndi-name="jdbc/postgres">
      <!-- Specifies the database username in the connection factory. -->
      <connection-factory driver="org.postgresql.Driver"
                          username="dbuser"
                          url="${org.infinispan.server.test.postgres.jdbcUrl}">
         <!-- Specifies the credential keystore that contains an encrypted password and the alias for it. -->
         <credential-reference store="credentials" alias="dbpassword"/>
      </connection-factory>
      <connection-pool max-size="10" min-size="1" background-validation="1000" idle-removal="1" initial-size="1" leak-detection="10000"/>
   </data-source>
</data-sources>

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:12.1 https://infinispan.org/schemas/infinispan-server-12.1.xsd"
          xmlns="urn:infinispan:server:12.1">
   <credential-stores>
      <credential-store name="credentials" path="credentials.pfx">
         <clear-text-credential clear-text="secret1234!"/>
      </credential-store>
   </credential-stores>
   <security-realms>
      <security-realm name="default">
         <!-- Specifies the LDAP principal in the connection factory. -->
         <ldap-realm name="ldap" url="ldap://my-ldap-server:10389"
                     principal="uid=admin,ou=People,dc=infinispan,dc=org"
                     connection-timeout="3000"
                     read-timeout="30000"
                     connection-pooling="true"
                     referral-mode="ignore"
                     page-size="30">
            <!-- Specifies the credential keystore that contains an encrypted password and the alias for it. -->
            <credential-reference store="credentials" alias="ldappassword"/>
         </ldap-realm>
      </security-realm>
   </security-realms>
</security>

Data Grid endpoints and connectors can specify one or more IP filtering rules. These rules specify the type of action to take when a client which matches a supplied CIDR block connects. IP filtering rules are applied in order up until the first one that matches.

A CIDR block is a compact representation of an IP address and its associated network mask. CIDR notation specifies an IP address, a slash ('/') character, and a decimal number. The decimal number is the count of leading 1 bits in the network mask. The number can also be thought of as the width, in bits, of the network prefix. The IP address in CIDR notation is always represented according to the standards for IPv4 or IPv6.

The address can denote a specific interface address, including a host identifier, such as 10.0.0.1/8, or it can be the beginning address of an entire network interface range using a host identifier of 0, as in 10.0.0.0/8 or 10/8.

For example:

<endpoints socket-binding="default" security-realm="default">
  <ip-filter>
    <accept from="192.168.0.0/16"/>
    <accept from="10.0.0.0/8"/>
    <reject from="/0"/>
  </ip-filter>
  <hotrod-connector name="hotrod"/>
  <rest-connector name="rest"/>
</endpoints>

As a result of the preceding configuration, Data Grid servers accept connections only from addresses in the 192.168.0.0/16 and 10.0.0.0/8 CIDR blocks. Data Grid servers reject all other connections.

Server IP filter rules can be manipulated via the CLI.

Use authorization in your cache configuration to restrict user access. Before they can read or write cache entries, or create and delete caches, users must have a role with a sufficient level of permission.

The following configuration example shows how to use implicit authorization configuration with default roles and permissions:

<infinispan>
  <cache-container default-cache="rbac-cache" name="restricted">
    <security>
      <!-- Enable authorization with the default roles and permissions. -->
      <authorization />
    </security>
    <local-cache name="rbac-cache">
      <security>
        <!-- Inherit authorization settings from the cache-container. -->
        <authorization/>
      </security>
    </local-cache>
  </cache-container>
</infinispan>

Data Grid includes a default set of roles that grant users with permissions to access data and interact with Data Grid resources.

ClusterRoleMapper is the default mechanism that Data Grid uses to associate security principals to authorization roles.

Data Grid authorization secures your installation by restricting user access.

User applications or clients must belong to a role that is assigned with sufficient permissions before they can perform operations on Cache Managers or caches.

For example, you configure authorization on a specific cache instance so that invoking Cache.get() requires an identity to be assigned a role with read permission while Cache.put() requires a role with write permission.

In this scenario, if a user application or client with the io role attempts to write an entry, Data Grid denies the request and throws a security exception. If a user application or client with the writer role sends a write request, Data Grid validates authorization and issues a token for subsequent operations.

The following image illustrates how security principals correspond to roles:

Data Grid caches roles that you grant to users internally for optimal performance. Whenever you grant or deny roles to users, Data Grid flushes the ACL cache to ensure user permissions are applied correctly.

If necessary, you can disable the ACL cache or configure it with the cache-size and cache-timeout attributes.

<security cache-size="1000" cache-timeout="300000">
  <authorization />
</security>

You can customize authorization settings in your Data Grid configuration to use role mappers with different combinations of roles and permissions.

The following configuration example shows how to configure security authorization with roles and permissions:

<infinispan>
  <cache-container default-cache="restricted" name="custom-authorization">
    <security>
      <authorization>
        <!-- Declare a role mapper that associates a security principal
             to each role. -->
        <identity-role-mapper />
        <!-- Specify user roles and corresponding permissions. -->
        <role name="admin" permissions="ALL" />
        <role name="reader" permissions="READ" />
        <role name="writer" permissions="WRITE" />
        <role name="supervisor" permissions="READ WRITE EXEC"/>
      </authorization>
    </security>
    <local-cache name="implicit-authorization">
      <security>
        <!-- Inherit roles and permissions from the cache-container. -->
        <authorization/>
      </security>
    </local-cache>
    <local-cache name="restricted">
      <security>
        <!-- Explicitly define which roles can access the cache. -->
        <authorization roles="admin supervisor"/>
      </security>
    </local-cache>
  </cache-container>
</infinispan>

In local development environments you can disable authorization so that users do not need roles and permissions. Disabling security authorization means that any user can access data and interact with Data Grid resources.

Enabling client certificate authentication means you do not need to specify Data Grid user credentials in client configuration, which means you must associate roles with the Common Name (CN) field in the client certificate(s).

Data Grid provides default JGroups stack files, default-jgroups-*.xml, in the default-configs directory inside the infinispan-core-12.1.11.Final-redhat-00001.jar file.

You can find this JAR file in the $RHDG_HOME/lib directory.

Data Grid supports different protocols that allow nodes to automatically find each other on the network and form clusters.

There are two types of discovery mechanisms that Data Grid can use:

<PING num_discovery_runs="3"/>

<TCP bind_port="7800" />
<TCPPING timeout="3000"
         initial_hosts="${jgroups.tcpping.initial_hosts:hostname1[port1],hostname2[port2]}"
          port_range="0"
          num_initial_members="3"/>

<MPING mcast_addr="${jgroups.mcast_addr:228.6.7.8}"
       mcast_port="${jgroups.mcast_port:46655}"
       num_discovery_runs="3"
       ip_ttl="${jgroups.udp.ip_ttl:2}"/>

<TCP bind_port="7800" />
<TCPGOSSIP timeout="3000"
           initial_hosts="${GossipRouterAddress}"
           num_initial_members="3" />

<JDBC_PING connection_url="jdbc:mysql://localhost:3306/database_name"
           connection_username="user"
           connection_password="password"
           connection_driver="com.mysql.jdbc.Driver"/>

<dns.DNS_PING dns_query="myservice.myproject.svc.cluster.local" />

Data Grid uses JGroups protocol stacks so nodes can send each other messages on dedicated cluster channels.

Data Grid provides preconfigured JGroups stacks for UDP and TCP protocols. You can use these default stacks as a starting point for building custom cluster transport configuration that is optimized for your network requirements.

[org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel cluster with stack udp

Adjust and tune properties to create a cluster transport configuration that works for your network requirements.

Data Grid provides attributes that let you extend the default JGroups stacks for easier configuration. You can inherit properties from the default stacks while combining, removing, and replacing other properties.

Pass system properties to Data Grid at startup to tune cluster transport.

For example, set a custom bind port and IP address as follows:

$ bin/server.sh -Djgroups.bind.port=1234 -Djgroups.bind.address=192.0.2.0

You can insert complete JGroups stack definitions into infinispan.xml files.

Reference external files that define custom JGroups stacks in infinispan.xml files.

Secure cluster transport so that nodes communicate with encrypted messages. You can also configure Data Grid clusters to perform certificate authentication so that only nodes with valid identities can join.

[org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel cluster with stack <encrypted_stack_name>

[org.jgroups.protocols.ASYM_ENCRYPT] <hostname>: received message without encrypt header from <hostname>; dropping it

<infinispan>
  <jgroups>
    <!-- Creates a secure JGroups stack named "encrypt-tcp" that extends the default TCP stack. -->
    <stack name="encrypt-tcp" extends="tcp">
      <!-- Adds a keystore from which nodes obtain secret keys. -->
      <!-- Uses the stack.combine and stack.position attributes to insert SYM_ENCRYPT into the default TCP stack after VERIFY_SUSPECT. -->
      <SYM_ENCRYPT keystore_name="myKeystore.p12"
                   keystore_type="PKCS12"
                   store_password="changeit"
                   key_password="changeit"
                   alias="myKey"
                   stack.combine="INSERT_AFTER"
                   stack.position="VERIFY_SUSPECT"/>
    </stack>
  </jgroups>
  <cache-container name="default" statistics="true">
    <!-- Configures the cluster to use the JGroups stack. -->
    <transport cluster="${infinispan.cluster.name}"
               stack="encrypt-tcp"
               node-name="${infinispan.node.name:}"/>
  </cache-container>
</infinispan>

[org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel cluster with stack <encrypted_stack_name>

[org.jgroups.protocols.SYM_ENCRYPT] <hostname>: received message without encrypt header from <hostname>; dropping it

Data Grid uses the following ports for cluster transport messages:

Cross-Site Replication

Data Grid uses the following ports for the JGroups RELAY2 protocol:

Caches configure the data container on Data Grid Server.

You create caches at run-time by adding definitions based on org.infinispan templates or Data Grid configuration through the console, the Command Line Interface (CLI), the Hot Rod endpoint, or the REST endpoint.

Data Grid Server provides a default Cache Manager configuration. When you start Data Grid Server, it instantiates the Cache Manager so you can remotely create caches at run-time.

<!-- Creates a Cache Manager named "default" and exports metrics. -->
<cache-container name="default"
                 statistics="true">
  <!-- Adds cluster transport that uses the default JGroups TCP stack. -->
  <transport cluster="${infinispan.cluster.name:cluster}"
             stack="${infinispan.cluster.stack:tcp}"
             node-name="${infinispan.node.name:}"/>
</cache-container>

Dynamically add caches from templates or configuration files through the Data Grid console.

Use the Data Grid CLI to add caches from templates or configuration files in XML or JSON format.

When Hot Rod Java clients attempt to access caches that do not exist, they return null for remoteCacheManager.getCache("myCache") invocations. To avoid this scenario, you can configure Hot Rod clients to create caches on first access using cache configuration.

File file = new File("path/to/infinispan.xml")
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.remoteCache("another-cache")
       .configuration("<distributed-cache name=\"another-cache\"/>");
builder.remoteCache("my.other.cache")
       .configurationURI(file.toURI());

infinispan.client.hotrod.cache.another-cache.configuration=<distributed-cache name=\"another-cache\"/>
infinispan.client.hotrod.cache.[my.other.cache].configuration_uri=file:///path/to/infinispan.xml

You can also create remote caches through the RemoteCacheManager API in other ways, such as the following example that adds a cache configuration with the XMLStringConfiguration() method and then calls the getOrCreateCache() method.

However, Data Grid does not recommend this approach because it can more difficult to ensure XML validity and is generally a more cumbersome way to create caches. If you are creating complex cache configurations, you should save them to separate files in your project and reference them in your Hot Rod client configuration.

String cacheName = "CacheWithXMLConfiguration";
String xml = String.format("<distributed-cache name=\"%s\" mode=\"SYNC\">" +
                              "<encoding media-type=\"application/x-protostream\"/>" +
                              "<locking isolation=\"READ_COMMITTED\"/>" +
                              "<transaction mode=\"NON_XA\"/>" +
                              "<expiration lifespan=\"60000\" interval=\"20000\"/>" +
                           "</distributed-cache>" , cacheName);
remoteCacheManager.administration().getOrCreateCache(cacheName, new XMLStringConfiguration(xml));

Visit Data Grid code examples.

Add cache definitions to Data Grid servers through the REST endpoint with any suitable HTTP client.

Use XML or JSON configuration by including it in the request payload.

POST /rest/v2/caches/mycache

Use the ?template= parameter to create caches from org.infinispan templates.

POST /rest/v2/caches/mycache?template=org.infinispan.DIST_SYNC

You can provide cache configuration in XML or JSON format.

<distributed-cache name="myCache" mode="SYNC">
  <encoding media-type="application/x-protostream"/>
  <memory max-count="1000000" when-full="REMOVE"/>
</distributed-cache>

{
  "distributed-cache": {
    "name": "myCache",
    "mode": "SYNC",
    "encoding": {
      "media-type": "application/x-protostream"
      },
    "memory": {
      "max-count": 1000000,
      "when-full": "REMOVE"
    }
  }
}

JSON format

Cache configuration in JSON format must follow the structure of an XML configuration. * XML elements become JSON objects. * XML attributes become JSON fields.

Data Grid server configuration for datasources is composed of two sections:

<data-sources>
   <!-- Defines a unique name for the datasource, JNDI name, and enables statistics. -->
   <data-source name="ds" jndi-name="jdbc/datasource" statistics="true">
      <!-- Specifies the JDBC driver that creates connections. -->
      <connection-factory driver="org.database.Driver"
                          username="db_user"
                          password="secret"
                          url="jdbc:db://database-host:10000/dbname"
                          new-connection-sql="SELECT 1"
                          transaction-isolation="READ_COMMITTED">
         <!-- Sets optional JDBC driver-specific connection properties. -->
         <connection-property name="name">value</connection-property>
      </connection-factory>
      <!-- Defines connection pool properties. -->
      <connection-pool initial-size="1"
                       max-size="10"
                       min-size="3"
                       background-validation="1000"
                       idle-removal="1"
                       blocking-timeout="1000"
                       leak-detection="10000"/>
   </data-source>
</data-sources>

Connection pools can be tuned using the following parameters:

Use a shared, managed datasource in your JDBC cache store configuration instead of specifying individual connection properties for each cache definition.

<distributed-cache-configuration name="persistent-cache" xmlns:jdbc="urn:infinispan:config:store:jdbc:12.1">
    <persistence>
        <jdbc:string-keyed-jdbc-store>
            <!-- Specifies the JNDI name that you provided for the datasource connection in the server configuration. -->
            <jdbc:data-source jndi-url="jdbc/postgres"/>
            <jdbc:string-keyed-table drop-on-exit="true"
                          create-on-start="true"
                          prefix="TBL">
                <jdbc:id-column name="ID" type="VARCHAR(255)"/>
                <jdbc:data-column name="DATA" type="BYTEA"/>
                <jdbc:timestamp-column name="TS" type="BIGINT"/>
                <jdbc:segment-column name="S" type="INT"/>
            </jdbc:string-keyed-table>
        </jdbc:string-keyed-jdbc-store>
    </persistence>
</distributed-cache-configuration>

Verify that connections to data sources are functioning correctly with the CLI.

Create custom task implementations and add them to Data Grid servers.

package example;

import org.infinispan.tasks.ServerTask;
import org.infinispan.tasks.TaskContext;

public class HelloTask implements ServerTask<String> {

   private TaskContext ctx;

   @Override
   public void setTaskContext(TaskContext ctx) {
      this.ctx = ctx;
   }

   @Override
   public String call() throws Exception {
      String name = (String) ctx.getParameters().get().get("name");
      return "Hello " + name;
   }

   @Override
   public String getName() {
      return "hello-task";
   }

}

Create custom scripts and add them to Data Grid servers.

// mode=local,language=javascript,parameters=[greetee]
"Hello " + greetee

// name=test, language=javascript
// mode=local, parameters=[a,b,c]

// mode=local,language=javascript
multiplicand * multiplier

RemoteCache<String, String> scriptCache = cacheManager.getCache("___script_cache");
scriptCache.put("multiplication.js",
  "// mode=local,language=javascript\n" +
  "multiplicand * multiplier\n");

Execute tasks and custom scripts on Data Grid servers.

RemoteCache<String, Integer> cache = cacheManager.getCache();
// Create parameters for script execution.
Map<String, Object> params = new HashMap<>();
params.put("multiplicand", 10);
params.put("multiplier", 20);
// Run the script with the parameters.
Object result = cache.execute("multiplication.js", params);

// Add configuration for a locally running server.
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.addServer().host("127.0.0.1").port(11222);

// Connect to the server.
RemoteCacheManager cacheManager = new RemoteCacheManager(builder.build());

// Retrieve the remote cache.
RemoteCache<String, String> cache = cacheManager.getCache();

// Create task parameters.
Map<String, String> parameters = new HashMap<>();
parameters.put("name", "developer");

// Run the server task.
String greet = cache.execute("hello-task", parameters);
System.out.println(greet);

Data Grid writes server logs to the following files in the $RHDG_HOME/server/log directory:

Access logs record all inbound client requests for Hot Rod and REST endpoints to files in the $RHDG_HOME/server/log directory.

<Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="TRACE">
   <AppenderRef ref="HR-ACCESS-FILE"/>
</Logger>

%X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] "%X{method} %m
%X{protocol}" %X{status} %X{requestSize} %X{responseSize} %X{duration}%n

Audit logs let you track changes to your Data Grid environment so you know when changes occur and which users make them. Enable and configure audit logging to record server configuration events and administrative operations.

<!-- Set to INFO to enable audit logging -->
<Logger name="org.infinispan.AUDIT" additivity="false" level="INFO">
   <AppenderRef ref="AUDIT-FILE"/>
</Logger>

Configure Data Grid to export statistics for Cache Managers and caches.

Data Grid Server enables Cache Manager statistics by default. You must explicitly enable statistics for your caches.

<!-- Enables statistics for the Cache Manager. -->
<cache-container statistics="true">
  <!-- Enables statistics for the named cache. -->
  <local-cache name="mycache" statistics="true"/>
</cache-container>

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder()
  //Enables statistics for the Cache Manager.
  .cacheContainer().statistics(true)
  .build();

Configuration config = new ConfigurationBuilder()
  //Enables statistics for the named cache.
  .statistics().enable()
  .build();

Configure Data Grid to export gauges and histograms via the metrics endpoint.

<!-- Computes and collects statistics for the Cache Manager. -->
<cache-container statistics="true">
  <!-- Exports collected statistics as gauge and histogram metrics. -->
  <metrics gauges="true" histograms="true" />
</cache-container>

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder()
  //Computes and collects statistics for the Cache Manager.
  .statistics().enable()
  //Exports collected statistics as gauge and histogram metrics.
  .metrics().gauges(true).histograms(true)
  .build();

Collect Data Grid metrics with monitoring tools such as Prometheus.

static_configs:
    - targets: ['localhost:11222']

Data Grid can register JMX MBeans that you can use to collect statistics and perform administrative operations. You must enable statistics separately to JMX otherwise Data Grid provides 0 values for all statistic attributes.

<cache-container>
  <jmx enabled="true" />
</cache-container>

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder()
  .jmx().enable()
  .build();

Retrieve Data Grid cluster health statistics via JMX.

Get Data Grid cluster health via the REST API.

Data Grid responds with a JSON document such as the following:

{
    "cluster_health":{
        "cluster_name":"ISPN",
        "health_status":"HEALTHY",
        "number_of_nodes":2,
        "node_names":[
            "NodeA-36229",
            "NodeB-28703"
        ]
    },
    "cache_health":[
        {
            "status":"HEALTHY",
            "cache_name":"___protobuf_metadata"
        },
        {
            "status":"HEALTHY",
            "cache_name":"cache2"
        },
        {
            "status":"HEALTHY",
            "cache_name":"mycache"
        },
        {
            "status":"HEALTHY",
            "cache_name":"cache1"
        }
    ]

}

GET /rest/v2/cache-managers/{cacheManagerName}/health/status

Create a cluster that runs the target Data Grid version and uses a remote cache store to load data from the source cluster.

<!-- Remote cache stores for rolling upgrades must disable passivation. -->
<persistence passivation="false">
   <!-- The value of the cache attribute matches the name of a cache in the source cluster. Target clusters load data from this cache using the remote cache store. -->
   <!-- The "protocol-version" attribute matches the Hot Rod protocol version of the source cluster. 2.5 is the minimum version and is suitable for any upgrade path. -->
   <!-- You should enable segmentation for remote cache stores only if the number of segments in the target cluster matches the number of segments for the cache in the source cluster. -->
   <remote-store xmlns="urn:infinispan:config:store:remote:12.1"
                 cache="myDistCache"
                 protocol-version="2.5"
                 hotrod-wrapping="true"
                 raw-values="true"
                 segmented="false">
      <!-- Configures authentication and encryption according to the security realm of the source cluster. -->
      <security>
        <authentication server-name="infinispan">
          <digest username="admin"
                  password="changeme"
                  realm="default"/>
        </authentication>
      </security>
      <!-- Points to the location of the source cluster. -->
      <remote-server host="127.0.0.1" port="11222"/>
   </remote-store>
</persistence>

When your target cluster is running and handling client requests using a remote cache store to load data on demand, you can synchronize data from the source cluster to the target cluster.

This operation reads data from the source cluster and writes it to the target cluster. Data migrates to all nodes in the target cluster in parallel, with each node receiving a subset of the data. You must perform the synchronization for each cache in your Data Grid configuration.

Data Grid servers provide aggregated reports in tar.gz archives that contain diagnostic information about both the Data Grid server and the host. The report provides details about CPU, memory, open files, network sockets and routing, threads, in addition to configuration and log files.

Modify the logging configuration for Data Grid servers at runtime to temporarily adjust logging to troubleshoot issues and perform root cause analysis.

Modifying the logging configuration through the CLI is a runtime-only operation, which means that changes:

[//containers/default]> logging list-appenders

The preceding command returns:

{
  "STDOUT" : {
    "name" : "STDOUT"
  },
  "JSON-FILE" : {
    "name" : "JSON-FILE"
  },
  "HR-ACCESS-FILE" : {
    "name" : "HR-ACCESS-FILE"
  },
  "FILE" : {
    "name" : "FILE"
  },
  "REST-ACCESS-FILE" : {
    "name" : "REST-ACCESS-FILE"
  }
}

[//containers/default]> logging list-loggers

The preceding command returns:

[ {
  "name" : "",
  "level" : "INFO",
  "appenders" : [ "STDOUT", "FILE" ]
}, {
  "name" : "org.infinispan.HOTROD_ACCESS_LOG",
  "level" : "INFO",
  "appenders" : [ "HR-ACCESS-FILE" ]
}, {
  "name" : "com.arjuna",
  "level" : "WARN",
  "appenders" : [ ]
}, {
  "name" : "org.infinispan.REST_ACCESS_LOG",
  "level" : "INFO",
  "appenders" : [ "REST-ACCESS-FILE" ]
} ]

For example, the following command sets the logging level for the org.infinispan package to DEBUG:

[//containers/default]> logging set --level=DEBUG org.infinispan

For example, the following command removes the org.infinispan logger configuration, which means the root configuration is used instead:

[//containers/default]> logging remove org.infinispan

You can inspect server-collected statistics for some of the resources within a Data Grid server using the stats command.

Use the stats command either from the context of a resource which collects statistics (containers, caches) or with a path to such a resource:

[//containers/default]> stats
{
  "statistics_enabled" : true,
  "number_of_entries" : 0,
  "hit_ratio" : 0.0,
  "read_write_ratio" : 0.0,
  "time_since_start" : 0,
  "time_since_reset" : 49,
  "current_number_of_entries" : 0,
  "current_number_of_entries_in_memory" : 0,
  "total_number_of_entries" : 0,
  "off_heap_memory_used" : 0,
  "data_memory_used" : 0,
  "stores" : 0,
  "retrievals" : 0,
  "hits" : 0,
  "misses" : 0,
  "remove_hits" : 0,
  "remove_misses" : 0,
  "evictions" : 0,
  "average_read_time" : 0,
  "average_read_time_nanos" : 0,
  "average_write_time" : 0,
  "average_write_time_nanos" : 0,
  "average_remove_time" : 0,
  "average_remove_time_nanos" : 0,
  "required_minimum_number_of_nodes" : -1
}

[//containers/default]> stats /containers/default/caches/mycache
{
  "time_since_start" : -1,
  "time_since_reset" : -1,
  "current_number_of_entries" : -1,
  "current_number_of_entries_in_memory" : -1,
  "total_number_of_entries" : -1,
  "off_heap_memory_used" : -1,
  "data_memory_used" : -1,
  "stores" : -1,
  "retrievals" : -1,
  "hits" : -1,
  "misses" : -1,
  "remove_hits" : -1,
  "remove_misses" : -1,
  "evictions" : -1,
  "average_read_time" : -1,
  "average_read_time_nanos" : -1,
  "average_write_time" : -1,
  "average_write_time_nanos" : -1,
  "average_remove_time" : -1,
  "average_remove_time_nanos" : -1,
  "required_minimum_number_of_nodes" : -1
}

Information about the Data Grid Server 12.1.11.Final-redhat-00001 distribution.

$RHDG_HOME/bin/server.sh

$RHDG_HOME\bin\server.bat

$RHDG_HOME/bin/server.sh -c infinispan-local.xml

$RHDG_HOME\bin\server.bat -c infinispan-local.xml

$RHDG_HOME/bin/server.sh -b 0.0.0.0

$RHDG_HOME\bin\server.bat -b 0.0.0.0

$RHDG_HOME/bin/server.sh -p 30000

$RHDG_HOME\bin\server.bat -p 30000

$RHDG_HOME/bin/server.sh -k 192.168.1.100

$RHDG_HOME\bin\server.bat -k 192.168.1.100

$RHDG_HOME/bin/server.sh -j udp

$RHDG_HOME\bin\server.bat -j udp

$RHDG_HOME/bin/cli.sh user create username -p "qwer1234!"

$RHDG_HOME\bin\cli.bat user create username -p "qwer1234!"

├── bin
├── boot
├── docs
├── lib
├── server
└── static

$RHDG_HOME/bin/server.sh -s server2

$RHDG_HOME\bin\server.bat -s server2

├── server
│   ├── conf
│   ├── data
│   ├── lib
│   └── log

$RHDG_HOME/bin/server.sh --logging-config=/path/to/log4j2.xml

$RHDG_HOME\bin\server.bat --logging-config=path\to\log4j2.xml