Chapter 1. Introduction to Service Registry

Service Registry is a datastore for sharing standard event schemas and API designs across event-driven and API architectures. You can use Service Registry to decouple the structure of your data from your client applications, and to share and manage your data types and API descriptions at runtime using a REST interface.

Client applications can dynamically push or pull the latest schema updates to or from Service Registry at runtime without needing to redeploy. Developer teams can query Service Registry for existing schemas required for services already deployed in production, and can register new schemas required for new services in development.

You can enable client applications to use schemas and API designs stored in Service Registry by specifying the Service Registry URL in your client application code. Service Registry can store schemas used to serialize and deserialize messages, which are referenced from your client applications to ensure that the messages that they send and receive are compatible with those schemas.

Using Service Registry to decouple your data structure from your applications reduces costs by decreasing overall message size, and creates efficiencies by increasing consistent reuse of schemas and API designs across your organization. Service Registry provides a web console to make it easy for developers and administrators to manage registry content.

You can configure optional rules to govern the evolution of your Service Registry content. These include rules to ensure that uploaded content is valid, or is compatible with other versions. Any configured rules must pass before new versions can be uploaded to Service Registry, which ensures that time is not wasted on invalid or incompatible schemas or API designs.

Service Registry is based on the Apicurio Registry open source community project. For details, see https://github.com/apicurio/apicurio-registry.

Service Registry capabilities

The items stored in Service Registry, such as event schemas and API designs, are known as registry artifacts. The following shows an example of an Apache Avro schema artifact in JSON format for a simple share price application:

{
   "type": "record",
   "name": "price",
   "namespace": "com.example",
   "fields": [
       {
           "name": "symbol",
           "type": "string"
       },
       {
           "name": "price",
           "type": "string"
       }
   ]
}

When a schema or API design is added as an artifact in Service Registry, client applications can then use that schema or API design to validate that the client messages conform to the correct data structure at runtime.

Groups of schemas and APIs

An artifact group is an optional named collection of schema or API artifacts. Each group contains a logically related set of schemas or API designs, typically managed by a single entity, belonging to a particular application or organization.

You can create optional artifact groups when adding your schemas and API designs to organize them in Service Registry. For example, you could create groups to match your development and production application environments, or your sales and engineering organizations.

Schema and API groups can contain multiple artifact types. For example, you could have Protobuf, Avro, JSON Schema, OpenAPI, or AsyncAPI artifacts all in the same group.

You can create schema and API artifacts and groups using the Service Registry web console, REST API, command line, Maven plug-in, or Java client application. The following simple example shows using the Core Registry REST API:

$ curl -X POST -H "Content-type: application/json; artifactType=AVRO" \
  -H "X-Registry-ArtifactId: share-price" \
  --data '{"type":"record","name":"price","namespace":"com.example", \
   "fields":[{"name":"symbol","type":"string"},{"name":"price","type":"string"}]}' \
  https://my-registry.example.com/apis/registry/v2/groups/my-group/artifacts

This example creates an artifact group named my-group and adds an Avro schema with an artifact ID of share-price.

References to other schemas and APIs

Some Service Registry artifact types can include artifact references from one artifact file to another. You can create efficiencies by defining reusable schema or API components, and then referencing them from multiple locations. For example, you can specify a reference in JSON Schema or OpenAPI using a $ref statement, or in Google Protobuf using an import statement, or in Apache Avro using a nested namespace.

The following example shows a simple Avro schema named TradeKey that includes a reference to another schema named Exchange using a nested namespace:

{
 "namespace": "com.kubetrade.schema.trade",
 "type": "record",
 "name": "TradeKey",
 "fields": [
   {
     "name": "exchange",
     "type": "com.kubetrade.schema.common.Exchange"
   },
   {
     "name": "key",
     "type": "string"
   }
 ]
}

{
 "namespace": "com.kubetrade.schema.common",
 "type": "enum",
 "name": "Exchange",
 "symbols" : ["GEMINI"]
}

An artifact reference is stored in Service Registry as a collection of artifact metadata that maps from an artifact type-specific reference to an internal Service Registry reference. Each artifact reference in Service Registry is composed of the following:

You can manage artifact references using the Service Registry core REST API, Maven plug-in, and Java serializers/deserializers (SerDes). Service Registry stores the artifact references along with the artifact content. Service Registry also maintains a collection of all artifact references so you can search them or list all references for a specific artifact.

Supported artifact types

Service Registry currently supports artifact references for the following artifact types only:

You can use the Service Registry web console to browse and search the schema and API artifacts and optional groups stored in the registry, and to add new schema and API artifacts, groups, and versions. You can search for artifacts by label, name, group, and description. You can view an artifact’s content or its available versions, or download an artifact file locally.

You can also configure optional rules for registry content, both globally and for each schema and API artifact. These optional rules for content validation and compatibility are applied when new schema and API artifacts or versions are uploaded to the registry.

For more details, see Chapter 9, Service Registry artifact and rule reference.

Figure 1.1. Service Registry web console

The Service Registry web console is available from http://MY_REGISTRY_URL/ui.

Service Registry provides the following options for the underlying storage of registry data:

Kafka producer applications can use serializers to encode messages that conform to a specific event schema. Kafka consumer applications can then use deserializers to validate that messages have been serialized using the correct schema, based on a specific schema ID.

Figure 1.2. Service Registry and Kafka client SerDes architecture

Service Registry provides Kafka client serializers/deserializers (SerDes) to validate the following message types at runtime:

The Service Registry Maven repository and source code distributions include the Kafka SerDes implementations for these message types, which Kafka client application developers can use to integrate with Service Registry.

These implementations include custom Java classes for each supported message type, for example, io.apicurio.registry.serde.avro, which client applications can use to pull schemas from Service Registry at runtime for validation.

You can use Service Registry with Apache Kafka Connect to stream data between Kafka and external systems. Using Kafka Connect, you can define connectors for different systems to move large volumes of data into and out of Kafka-based systems.

Figure 1.3. Service Registry and Kafka Connect architecture

Service Registry provides the following features for Kafka Connect:

You can use the Avro and JSON Schema converters to map Kafka Connect schemas into Avro or JSON schemas. Those schemas can then serialize message keys and values into the compact Avro binary format or human-readable JSON format. The converted JSON is also less verbose because the messages do not contain the schema information, only the schema ID.

Service Registry can manage and track the Avro and JSON schemas used in the Kafka topics. Because the schemas are stored in Service Registry and decoupled from the message content, each message must only include a tiny schema identifier. For an I/O bound system like Kafka, this means more total throughput for producers and consumers.

The Avro and JSON Schema serializers and deserializers (SerDes) provided by Service Registry are also used by Kafka producers and consumers in this use case. Kafka consumer applications that you write to consume change events can use the Avro or JSON SerDes to deserialize these change events. You can install these SerDes into any Kafka-based system and use them along with Kafka Connect, or with Kafka Connect-based systems such as Debezium and Camel Kafka Connector.

Service Registry provides open source example applications that demonstrate how to use Service Registry in different use case scenarios. For example, these include storing schemas used by Kafka serializer and deserializer (SerDes) Java classes. These classes fetch the schema from Service Registry for use when producing or consuming operations to serialize, deserialize, or validate the Kafka message payload.

These applications demonstrate use cases such as the following examples:

Service Registry provides the following distribution options.