Cache Encoding and Marshalling

Configure Data Grid to use the ProtoStream library to store cache entries as Protocol Buffers (Protobuf).

<distributed-cache>
   <encoding>
      <key media-type="application/x-protostream"/>
      <value media-type="application/x-protostream"/>
   </encoding>
</distributed-cache>

<distributed-cache>
   <encoding>
      <key media-type="application/x-protostream"/>
      <value media-type="application/x-protostream"/>
   </encoding>
</distributed-cache>

//Create cache configuration that encodes keys and values as ProtoStream
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.clustering().cacheMode(CacheMode.DIST_SYNC)
       .encoding().key().mediaType("application/x-protostream")
       .encoding().value().mediaType("application/x-protostream");

//Create cache configuration that encodes keys and values as ProtoStream
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.clustering().cacheMode(CacheMode.DIST_SYNC)
       .encoding().key().mediaType("application/x-protostream")
       .encoding().value().mediaType("application/x-protostream");

Alternatively you can use the same encoding for keys and values:

<encoding media-type="application/x-protostream"/>

<encoding media-type="application/x-protostream"/>

.encoding().mediaType("application/x-protostream");

.encoding().mediaType("application/x-protostream");

If you configure encoding to store keys and values with a text-based media type, then you also need to configure clients to operate on those caches.

Hot Rod clients

Data Grid uses the ProtoStream library to handle String and byte[] types natively. If you configure cache encoding with the text/plain media type, Hot Rod clients might not necessarily require any marshaller configuration to perform cache operations.

For other text-based media types, such as JSON or XML, Hot Rod clients can use the org.infinispan.commons.marshall.UTF8StringMarshaller marshaller that converts to and from the text/plain media type.

REST clients

REST clients must include the media type for caches in the request headers.

For example if you configure cache encoding as text/plain; charset=UTF-8 then REST clients should send the following headers:

When you configure Hot Rod Java clients to use a marshaller, you must configure your cache with the encoding for that marshaller.

Each marshaller uses a different media type to produce byte[] content that the client can transmit to Data Grid Server. When reading from the server, the client marshaller performs the opposite operation, using the media type to produce data from byte[] content.

Your cache encoding must be compatible with the Hot Rod client marshaller. For example, if you configure a cache encoding as application/x-protostream, you can use the ProtoStream marshaller with your clients to operate on that cache. However if the client marshaller uses an encoding that Data Grid cannot convert to and from application/x-protostream, Data Grid throws an error message.

If you use JavaSerializationMarshaller or GenericJBossMarshaller you should encode caches with the application/x-java-serialized-object or application/x-jboss-marshalling media type, respectively.

This allows REST clients to include the JSON media type in request headers and perform operations on caches that use ProtoStream encoding:

Even though Data Grid does not recommend doing so, clients can operate on caches that store unmarshalled POJOs with the application/x-java-object media type.

Hot Rod clients

Hot Rod client marshallers must be available to Data Grid Server so it can deserialize your Java objects. By default, the ProtoStream and Java Serialization marshallers are available on the server.

REST clients

REST clients must use either JSON or XML for keys and values so Data Grid can convert to and from POJOs.

Data Grid uses a ProtoStream library that can handle the following types for keys and values, as well as the unboxed equivalents in the case of primitive types:

Additional type collections

The ProtoStream library includes several adapter classes for common Java types, for example:

Data Grid provides all adapter classes for some common JDK classes in the protostream-types artifact, which is included in the infinispan-core and infinispan-client-hotrod dependencies. You do not need any configuration to store adapter classes as keys or values.

However, if you want to use adapter classes as marshallable fields in ProtoStream-annotated POJOs, you can do so in the following ways:

@ProtoSchema(dependsOn = {org.infinispan.protostream.types.java.CommonTypes, org.infinispan.protostream.types.java.CommonContainerTypes}, schemaFileName = "library.proto", schemaFilePath = "proto", schemaPackageName = "example")
public interface LibraryInitalizer extends SerializationContextInitializer {
}

@ProtoSchema(dependsOn = {org.infinispan.protostream.types.java.CommonTypes, org.infinispan.protostream.types.java.CommonContainerTypes}, schemaFileName = "library.proto", schemaFilePath = "proto", schemaPackageName = "example")
public interface LibraryInitalizer extends SerializationContextInitializer {
}

@ProtoSchema(includeClasses = { Author.class, Book.class, UUIDAdapter.class, java.math.BigInteger }, schemaFileName = "library.proto", schemaFilePath = "proto", schemaPackageName = "library")
public interface LibraryInitalizer extends SerializationContextInitializer {

}

@ProtoSchema(includeClasses = { Author.class, Book.class, UUIDAdapter.class, java.math.BigInteger }, schemaFileName = "library.proto", schemaFilePath = "proto", schemaPackageName = "library")
public interface LibraryInitalizer extends SerializationContextInitializer {

}

The ProtoStream API includes annotations that you can add to Java applications to define Protobuf schemas, which provide a structured format for your objects.

This topic provides additional details about ProtoStream annotations. You should refer to the documentation in the org.infinispan.protostream.annotations package for complete information.

Proto

@Proto defines a Protocol Buffers message without the requirement of having to annotate all fields with the @ProtoField annotation.

ProtoField

@ProtoField defines a Protobuf message field.

This annotation applies to fields as well as getter and setter methods. Unless you are using the @Proto annotation, a class must have at least one field annotated with @ProtoField before Data Grid can marshall it as Protobuf.

ProtoFactory

@ProtoFactory marks a single constructor or static factory method for creating instances of the message class.

You can use this annotation to support immutable message classes. All fields annotated with @ProtoField must be included in the parameters.

ProtoSchema

@ProtoSchema generates an implementation of a class or interface that extends SerializationContextInitializer.

If active, the ProtoStream processor generates the implementation at compile time in the same package with the Impl suffix or a name that you specify with the className parameter.

The includeClasses or basePackages parameters reference classes that the ProtoStream processor should scan and include in the Protobuf schema and marshaller. If you do not set either of these parameters, the ProtoStream processor scans the entire source path, which can lead to unexpected results and is not recommended. You can also use the excludeClasses parameter with the basePackages parameter to exclude classes.

The schemaFileName and schemaPackageName parameters register the generated Protobuf schema under this name. If you do not set these parameters, the annotated simple class name is used with the unnamed, or default, package. Schema names must end with the .proto file extension. You can also use the marshallersOnly to generate marshallers only and suppress the Protobuf schema generation.

The ProtoStream process automatically generates META-INF/services service metadata files, which you can use so that Data Grid Server automatically picks up the JAR to register the Protobuf schema.

The dependsOn parameter lists annotated classes that implement SerializedContextInitializer to execute first. If the class does not implement SerializedContextInitializer or is not annotated with ProtoSchema, a compile time error occurs.

ProtoAdapter

@ProtoAdapter is a marshalling adapter for a class or enum that you cannot annotate directly.

If you use this annotation for:

ProtoName

@ProtoName is an optional annotation that specifies the Protobuf message or enum type name. It can be used on classes, records and enums.

ProtoEnumValue

@ProtoEnumValue defines a Protobuf enum value. You can apply this annotation to members of a Java enum only.

ProtoReserved and ProtoReservedStatements

@ProtoReserved and @ProtoReservedStatements add reserved statements to generated messages or enum definitions to prevent future usage of numbers, ranges, and names.

ProtoTypeId

@ProtoTypeId optionally specifies a globally unique numeric type identifier for a Protobuf message or enum type.

ProtoUnknownFieldSet

@ProtoUnknownFieldSet optionally indicates the field, or JavaBean property of type {@link org.infinispan.protostream.UnknownFieldSet}, which stores any unknown fields.

Other annotations

Data Grid copies any other annotations on classes, fields, and methods as comments in the generated Protobuf schema. This includes indexing annotations such as @Indexed and @Basic.

Data Grid provides a ProtoStream processor artifact that processes Java annotations in your classes at compile time to generate Protobuf schemas, accompanying marshallers, and a concrete implementation of the SerializationContextInitializer interface.

Declare ProtoStream metadata by adding annotations to a Java class and its members. Data Grid then uses the ProtoStream processor to generate Protobuf schema and related marshallers from those annotations.

The following Author.java and Book.java examples show Java classes annotated with @ProtoField and @ProtoFactory:

import org.infinispan.protostream.annotations.ProtoFactory;
import org.infinispan.protostream.annotations.ProtoField;

public class Author {
   @ProtoField(1)
   final String name;

   @ProtoField(2)
   final String surname;

   @ProtoFactory
   Author(String name, String surname) {
      this.name = name;
      this.surname = surname;
   }
   // public Getter methods omitted for brevity
}

import org.infinispan.protostream.annotations.ProtoFactory;
import org.infinispan.protostream.annotations.ProtoField;

public class Author {
   @ProtoField(1)
   final String name;

   @ProtoField(2)
   final String surname;

   @ProtoFactory
   Author(String name, String surname) {
      this.name = name;
      this.surname = surname;
   }
   // public Getter methods omitted for brevity
}

import org.infinispan.protostream.annotations.ProtoFactory;
import org.infinispan.protostream.annotations.ProtoField;

public class Book {
   @ProtoField(number = 1)
   public final UUID id;

   @ProtoField(number = 2)
   final String title;

   @ProtoField(number = 3)
   final String description;

   @ProtoField(number = 4, defaultValue = "0")
   final int publicationYear;

   @ProtoField(number = 5, collectionImplementation = ArrayList.class)
   final List<Author> authors;

   @ProtoField(number = 6)
   public Language language;

   @ProtoFactory
   Book(UUID id, String title, String description, int publicationYear, List<Author> authors, Language language) {
      this.id = id;
      this.title = title;
      this.description = description;
      this.publicationYear = publicationYear;
      this.authors = authors;
      this.language = language;
   }
   // public Getter methods not included for brevity
}

import org.infinispan.protostream.annotations.ProtoFactory;
import org.infinispan.protostream.annotations.ProtoField;

public class Book {
   @ProtoField(number = 1)
   public final UUID id;

   @ProtoField(number = 2)
   final String title;

   @ProtoField(number = 3)
   final String description;

   @ProtoField(number = 4, defaultValue = "0")
   final int publicationYear;

   @ProtoField(number = 5, collectionImplementation = ArrayList.class)
   final List<Author> authors;

   @ProtoField(number = 6)
   public Language language;

   @ProtoFactory
   Book(UUID id, String title, String description, int publicationYear, List<Author> authors, Language language) {
      this.id = id;
      this.title = title;
      this.description = description;
      this.publicationYear = publicationYear;
      this.authors = authors;
      this.language = language;
   }
   // public Getter methods not included for brevity
}

The following Language.java example shows a Java enum annotated with @ProtoEnumValue along with the corresponding Protobuf schema:

import org.infinispan.protostream.annotations.ProtoEnumValue;

public enum Language {
  @ProtoEnumValue(number = 0, name = "EN")
  ENGLISH,
  @ProtoEnumValue(number = 1, name = "DE")
  GERMAN,
  @ProtoEnumValue(number = 2, name = "IT")
  ITALIAN,
  @ProtoEnumValue(number = 3, name = "ES")
  SPANISH,
  @ProtoEnumValue(number = 4, name = "FR")
  FRENCH;

}

import org.infinispan.protostream.annotations.ProtoEnumValue;

public enum Language {
  @ProtoEnumValue(number = 0, name = "EN")
  ENGLISH,
  @ProtoEnumValue(number = 1, name = "DE")
  GERMAN,
  @ProtoEnumValue(number = 2, name = "IT")
  ITALIAN,
  @ProtoEnumValue(number = 3, name = "ES")
  SPANISH,
  @ProtoEnumValue(number = 4, name = "FR")
  FRENCH;

}

enum Language {

   EN = 0;

   DE = 1;

   IT = 2;

   ES = 3;

   FR = 4;
}

enum Language {

   EN = 0;

   DE = 1;

   IT = 2;

   ES = 3;

   FR = 4;
}

ProtoStream provides a @ProtoAdapter annotation that you can use to marshall external, third-party Java object classes that you cannot annotate directly.

After you add the ProtoStream processor and annotate your Java classes, you can add the @ProtoSchema annotation to an interface so that Data Grid generates the Protobuf schema, accompanying marshallers, and a concrete implementation of the SerializationContextInitializer.

The following example shows a GeneratedSchema interface annotated with @ProtoSchema:

@ProtoSchema(
      includeClasses = {
            Book.class,
            Author.class,
            UUIDAdapter.class,
            Language.class
      },
      schemaFileName = "library.proto",
      schemaFilePath = "proto/",
      schemaPackageName = "book_sample")
interface LibraryInitializer extends GeneratedSchema {
}

@ProtoSchema(
      includeClasses = {
            Book.class,
            Author.class,
            UUIDAdapter.class,
            Language.class
      },
      schemaFileName = "library.proto",
      schemaFilePath = "proto/",
      schemaPackageName = "book_sample")
interface LibraryInitializer extends GeneratedSchema {
}

If you use remote caches, you must register your SerializationContextInitializer implementation with Data Grid Server.

The Protocol Buffers documentation provides a list of best practices on how to design messages and how to evolve the schema in order to maintain backwards compatibility.

Data Grid can automatically perform compatibility checks when schemas are updated and reject updates when incompatibilities are detected. The types of checks can be configured via the schema-compatibility attribute of the global serialization configuration. The available levels are:

The following table shows the compatibility check rules enabled for each level

For embedded caches, Data Grid automatically registers serialization contexts and marshallers in your annotated SerializationContextInitializer implementation using the java.util.ServiceLoader.

If you prefer, you can disable automatic registration of SerializationContextInitializer implementations and then register them manually.

<serialization>
    <context-initializer class="org.infinispan.example.LibraryInitializerImpl"/>
    <context-initializer class="org.infinispan.example.another.SCIImpl"/>
</serialization>

<serialization>
    <context-initializer class="org.infinispan.example.LibraryInitializerImpl"/>
    <context-initializer class="org.infinispan.example.another.SCIImpl"/>
</serialization>

GlobalConfigurationBuilder builder = new GlobalConfigurationBuilder();
builder.serialization()
       .addContextInitializers(new LibraryInitializerImpl(), new SCIImpl());

GlobalConfigurationBuilder builder = new GlobalConfigurationBuilder();
builder.serialization()
       .addContextInitializers(new LibraryInitializerImpl(), new SCIImpl());

Register Protobuf schemas with Data Grid Server to perform Ickle queries or convert from application/x-protostream to other media types such as application/json.

ConfigurationBuilder remoteBuilder = new ConfigurationBuilder();
remoteBuilder.addServer().host(host).port(Integer.parseInt(port));

// Add your generated SerializationContextInitializer implementation.
LibraryInitalizer initializer = new LibraryInitalizerImpl();
remoteBuilder.addContextInitializer(initializer);

ConfigurationBuilder remoteBuilder = new ConfigurationBuilder();
remoteBuilder.addServer().host(host).port(Integer.parseInt(port));

// Add your generated SerializationContextInitializer implementation.
LibraryInitalizer initializer = new LibraryInitalizerImpl();
remoteBuilder.addContextInitializer(initializer);

It is possible to manually implement SerializationContextInitializer or GeneratedSchema interfaces using ProtobufTagMarshaller and RawProtobufMarshaller annotations.

However, manual implementations require a lot of tedious overhead and are prone to error. Implementations that you generate with the protostream-processor artifact are a much more efficient and reliable way to configure ProtoStream marshalling.