Chapter 8. Configure observability

Procedure

1. Create a main.go file for a Go application that starts a server on port 8080 and writes events to standard output:

   package main
   
   import (
   	"io/ioutil"
   	"net/http"
   
   	"go.uber.org/zap"
   )
   
   var logger *zap.SugaredLogger
   
   func event(w http.ResponseWriter, req *http.Request) {
   	switch req.Method {
   	case "GET":
   		logger.Info("GET /event")
   	case "POST":
   		logger.Info("POST /event")
   	}
   	body, err := req.GetBody()
   	if err != nil {
   		logger.With("err", err).Info("error getting body")
   		return
   	}
   	responseBody, err := ioutil.ReadAll(body)
   	if err != nil {
   		logger.With("error", err).Info("error reading response body")
   		return
   	}
   	logger.With("body", string(responseBody)).Info("got event")
   }
   
   func activity(w http.ResponseWriter, req *http.Request) {
   	switch req.Method {
   	case "GET":
   		logger.Info("GET /activity, doing nothing")
   	case "POST":
   		logger.Info("POST /activity")
   		body, err := req.GetBody()
   		if err != nil {
   			logger.With("error", err).Info("error getting body")
   			return
   		}
   		responseBody, err := ioutil.ReadAll(body)
   		if err != nil {
   			logger.With("error", err).Info("error reading response body")
   			return
   		}
   		logger.With("body", string(responseBody)).Info("got activity")
   	}
   }
   
   func main() {
   
   	log, _ := zap.NewProduction()
   	logger = log.Sugar()
   
   	http.HandleFunc("/event", event)
   	http.HandleFunc("/activity", activity)
   	logger.Info("Added Handlers")
   
   	logger.Info("Starting to serve")
   	http.ListenAndServe(":8080", nil)
   }

   The code for the example telemetry server is available in the telemetry-server-example repository.

2. Create a container image based on this code and expose it as a deployment in OpenShift in the openshift-devspaces project. Clone the repository and build the container:

   $ git clone https://github.com/che-incubator/telemetry-server-example
   $ cd telemetry-server-example
   $ podman build -t registry/organization/telemetry-server-example:latest .
   $ podman push registry/organization/telemetry-server-example:latest

3. Deploy the telemetry server to OpenShift.

   Both manifest_with_ingress.yaml and manifest_with_route contain definitions for a Deployment and Service. The former also defines a Kubernetes Ingress, while the latter defines an OpenShift Route.

   In the manifest file, replace the image and host fields to match the image you pushed, and the public hostname of your OpenShift cluster. Then run:

   $ oc apply -f manifest_with_[ingress|route].yaml -n openshift-devspaces

Procedure

1. Create a Maven Quarkus project:

   mvn io.quarkus:quarkus-maven-plugin:2.7.1.Final:create \
       -DprojectGroupId=mygroup -DprojectArtifactId=devworkspace-telemetry-example-plugin \
   -DprojectVersion=1.0.0-SNAPSHOT

2. Remove the files under src/main/java/mygroup and src/test/java/mygroup.

3. Consult the GitHub packages for the latest version of backend-base and add the following dependencies to your pom.xml:

   <!-- Required -->
   <dependency>
       <groupId>org.eclipse.che.incubator.workspace-telemetry</groupId>
       <artifactId>backend-base</artifactId>
       <version><latest_version></version>
   </dependency>
   
   
   <!-- Used to make http requests to the telemetry server -->
   <dependency>
       <groupId>io.quarkus</groupId>
       <artifactId>quarkus-rest-client</artifactId>
   </dependency>
   <dependency>
       <groupId>io.quarkus</groupId>
       <artifactId>quarkus-rest-client-jackson</artifactId>
   </dependency>

4. Create a personal access token with read:packages permissions from GitHub packages and add your GitHub username, the token, and che-incubator repository details in your ~/.m2/settings.xml file:

   <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
   http://maven.apache.org/xsd/settings-1.0.0.xsd">
      <servers>
         <server>
            <id>che-incubator</id>
            <username><github_username></username>
            <password><github_token></password>
         </server>
      </servers>
   
      <profiles>
         <profile>
            <id>github</id>
            <activation>
               <activeByDefault>true</activeByDefault>
            </activation>
            <repositories>
               <repository>
                  <id>central</id>
                  <url>https://repo1.maven.org/maven2</url>
                  <releases><enabled>true</enabled></releases>
                  <snapshots><enabled>false</enabled></snapshots>
                  </repository>
                  <repository>
                  <id>che-incubator</id>
                  <url>https://maven.pkg.github.com/che-incubator/che-workspace-telemetry-client</url>
               </repository>
            </repositories>
         </profile>
      </profiles>
   </settings>

5. Create MainConfiguration.java under src/main/java/mygroup. This file contains configuration provided to AnalyticsManager:

   package org.my.group;
   
   import java.util.Optional;
   
   import javax.enterprise.context.Dependent;
   import javax.enterprise.inject.Alternative;
   
   import org.eclipse.che.incubator.workspace.telemetry.base.BaseConfiguration;
   import org.eclipse.microprofile.config.inject.ConfigProperty;
   
   @Dependent
   @Alternative
   public class MainConfiguration extends BaseConfiguration {
       @ConfigProperty(name = "welcome.message")
       Optional<String> welcomeMessage;
   }

   where:

   @ConfigProperty(name = "welcome.message")

   A MicroProfile configuration annotation that injects the welcome.message configuration. For more details on how to set configuration properties specific to your backend, see the Quarkus Configuration Reference Guide.

6. Create AnalyticsManager.java under src/main/java/mygroup. This file contains logic specific to the telemetry system:

   package org.my.group;
   
   import java.util.HashMap;
   import java.util.Map;
   
   import javax.enterprise.context.Dependent;
   import javax.enterprise.inject.Alternative;
   import javax.inject.Inject;
   
   import org.eclipse.che.incubator.workspace.telemetry.base.AbstractAnalyticsManager;
   import org.eclipse.che.incubator.workspace.telemetry.base.AnalyticsEvent;
   import org.eclipse.che.incubator.workspace.telemetry.finder.DevWorkspaceFinder;
   import org.eclipse.che.incubator.workspace.telemetry.finder.UsernameFinder;
   import org.eclipse.microprofile.rest.client.inject.RestClient;
   import org.slf4j.Logger;
   
   import static org.slf4j.LoggerFactory.getLogger;
   
   @Dependent
   @Alternative
   public class AnalyticsManager extends AbstractAnalyticsManager {
   
       private static final Logger LOG = getLogger(AbstractAnalyticsManager.class);
   
       public AnalyticsManager(MainConfiguration mainConfiguration, DevWorkspaceFinder devworkspaceFinder, UsernameFinder usernameFinder) {
           super(mainConfiguration, devworkspaceFinder, usernameFinder);
   
           mainConfiguration.welcomeMessage.ifPresentOrElse(
               (str) -> LOG.info("The welcome message is: {}", str),
               () -> LOG.info("No welcome message provided")
           );
       }
   
       @Override
       public boolean isEnabled() {
           return true;
       }
   
       @Override
       public void destroy() {}
   
       @Override
       public void onEvent(AnalyticsEvent event, String ownerId, String ip, String userAgent, String resolution, Map<String, Object> properties) {
           LOG.info("The received event is: {}", event);
       }
   
       @Override
       public void increaseDuration(AnalyticsEvent event, Map<String, Object> properties) { }
   
       @Override
       public void onActivity() {}
   }

   where:

   ifPresentOrElse()

   Log the welcome message if it was provided.

   LOG.info("The received event is: {}", event)

   Log the event received from the front-end plugin.

7. Add the quarkus.arc.selected-alternatives property to src/main/resources/application.properties to specify the alternative beans org.my.group.AnalyticsManager and org.my.group.MainConfiguration:

   quarkus.arc.selected-alternatives=MainConfiguration,AnalyticsManager

Procedure

1. Set the DEVWORKSPACE_TELEMETRY_BACKEND_PORT environment variable in the Dev Workspace. Here, the value is set to 4167.

   spec:
     template:
       attributes:
         workspaceEnv:
           - name: DEVWORKSPACE_TELEMETRY_BACKEND_PORT
             value: '4167'

2. Restart the Dev Workspace from the Red Hat OpenShift Dev Spaces dashboard.

3. Run the following command within a Dev Workspace’s terminal window to start the application. Use the --settings flag to specify the path to the settings.xml file that contains the GitHub access token.

   $ mvn --settings=settings.xml quarkus:dev -Dquarkus.http.port=${DEVWORKSPACE_TELEMETRY_BACKEND_PORT}

   The application now receives telemetry events through port 4167 from the front-end plugin. Verify that the following output is logged:

   INFO  [org.ecl.che.inc.AnalyticsManager] (Quarkus Main Thread) No welcome message provided
   INFO  [io.quarkus] (Quarkus Main Thread) devworkspace-telemetry-example-plugin 1.0.0-SNAPSHOT on JVM (powered by Quarkus 2.7.2.Final) started in 0.323s. Listening on: http://localhost:4167
   INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
   INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, kubernetes-client, rest-client, rest-client-jackson, resteasy, resteasy-jsonb, smallrye-context-propagation, smallrye-openapi, swagger-ui, vertx]

4. Customize isEnabled() in AnalyticsManager.java. For this example, the method always returns true:

   @Override
   public boolean isEnabled() {
       return true;
   }

   The hosted OpenShift Dev Spaces Woopra backend demonstrates a more advanced isEnabled() implementation that checks for a configuration property before enabling the backend.

5. Implement onEvent() to send events to the telemetry server. For the example application, it sends an HTTP POST payload to the /event endpoint.

   1. Configure the RESTEasy REST Client by creating a TelemetryService.java interface:

      package org.my.group;
      
      import java.util.Map;
      
      import javax.ws.rs.Consumes;
      import javax.ws.rs.POST;
      import javax.ws.rs.Path;
      import javax.ws.rs.core.MediaType;
      import javax.ws.rs.core.Response;
      
      import org.eclipse.microprofile.rest.client.inject.RegisterRestClient;
      
      @RegisterRestClient
      public interface TelemetryService {
          @POST
          @Path("/event")
          @Consumes(MediaType.APPLICATION_JSON)
          Response sendEvent(Map<String, Object> payload);
      }

      where:

      @Path("/event")

      The endpoint to make the POST request to.

   2. Specify the base URL for TelemetryService in src/main/resources/application.properties:

      org.my.group.TelemetryService/mp-rest/url=http://little-telemetry-server-che.apps-crc.testing

   3. Inject TelemetryService into AnalyticsManager.java and send a POST request in onEvent():

      @Dependent
      @Alternative
      public class AnalyticsManager extends AbstractAnalyticsManager {
          @Inject
          @RestClient
          TelemetryService telemetryService;
      
      ...
      
      @Override
      public void onEvent(AnalyticsEvent event, String ownerId, String ip, String userAgent, String resolution, Map<String, Object> properties) {
          Map<String, Object> payload = new HashMap<String, Object>(properties);
          payload.put("event", event);
          telemetryService.sendEvent(payload);
      }

      This sends an HTTP request to the telemetry server and automatically delays identical events for a small period of time. The default duration is 1500 milliseconds.

6. Implement increaseDuration() in AnalyticsManager.java. Many telemetry systems recognize event duration. The AbstractAnalyticsManager merges similar events that happen in the same frame of time into one event. This implementation is a no-op:

   @Override
   public void increaseDuration(AnalyticsEvent event, Map<String, Object> properties) {}

7. Implement onActivity() in AnalyticsManager.java. Set an inactive timeout limit and send a WORKSPACE_INACTIVE event if the last event time exceeds the timeout:

   public class AnalyticsManager extends AbstractAnalyticsManager {
   
       ...
   
       private long inactiveTimeLimit = 60000 * 3;
   
       ...
   
       @Override
       public void onActivity() {
           if (System.currentTimeMillis() - lastEventTime >= inactiveTimeLimit) {
               onEvent(WORKSPACE_INACTIVE, lastOwnerId, lastIp, lastUserAgent, lastResolution, commonProperties);
           }
       }

8. Implement destroy() in AnalyticsManager.java. When called, send a WORKSPACE_STOPPED event and shut down any resources such as connection pools:

   @Override
   public void destroy() {
       onEvent(WORKSPACE_STOPPED, lastOwnerId, lastIp, lastUserAgent, lastResolution, commonProperties);
   }

Verification

1. To verify that the onEvent() method receives events from the front-end plugin, press the l key to disable Quarkus live coding and edit any file within the IDE. The following output should be logged:

   INFO  [io.qua.dep.dev.RuntimeUpdatesProcessor] (Aesh InputStream Reader) Live reload disabled
   INFO  [org.ecl.che.inc.AnalyticsManager] (executor-thread-2) The received event is: Edit Workspace File in Che

2. Stop the application with Ctrl+C and verify that a WORKSPACE_STOPPED event is sent to the server.

Procedure

1. Package the Quarkus application as a container image and push it to a container registry by using one of the following options. See the Quarkus documentation for details.

   Option A: JVM image

   1. Create a Dockerfile.jvm:

      FROM registry.access.redhat.com/ubi8/openjdk-11:1.11
      
      ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en'
      
      COPY --chown=185 target/quarkus-app/lib/ /deployments/lib/
      COPY --chown=185 target/quarkus-app/*.jar /deployments/
      COPY --chown=185 target/quarkus-app/app/ /deployments/app/
      COPY --chown=185 target/quarkus-app/quarkus/ /deployments/quarkus/
      
      EXPOSE 8080
      USER 185
      
      ENTRYPOINT ["java", "-Dquarkus.http.host=0.0.0.0", "-Djava.util.logging.manager=org.jboss.logmanager.LogManager", "-Dquarkus.http.port=${DEVWORKSPACE_TELEMETRY_BACKEND_PORT}", "-jar", "/deployments/quarkus-run.jar"]

   2. Build and push the image:

      mvn package && \
      podman build -f src/main/docker/Dockerfile.jvm -t image:tag .

   Option B: Native image

   1. Create a Dockerfile.native:

      FROM registry.access.redhat.com/ubi8/ubi-minimal:8.5
      WORKDIR /work/
      RUN chown 1001 /work \
          && chmod "g+rwX" /work \
          && chown 1001:root /work
      COPY --chown=1001:root target/*-runner /work/application
      
      EXPOSE 8080
      USER 1001
      
      CMD ["./application", "-Dquarkus.http.host=0.0.0.0", "-Dquarkus.http.port=${DEVWORKSPACE_TELEMETRY_BACKEND_PORT}"]

   2. Build and push the image:

      mvn package -Pnative -Dquarkus.native.container-build=true && \
      podman build -f src/main/docker/Dockerfile.native -t image:tag .

2. Create a plugin.yaml devfile v2 file representing a Dev Workspace plugin that runs your custom backend in a Dev Workspace Pod. For more information about devfile v2, see Devfile v2 documentation.

   schemaVersion: 2.1.0
   metadata:
     name: devworkspace-telemetry-backend-plugin
     version: 0.0.1
     description: A Demo telemetry backend
     displayName: Devworkspace Telemetry Backend
   components:
     - name: devworkspace-telemetry-backend-plugin
       attributes:
         workspaceEnv:
           - name: DEVWORKSPACE_TELEMETRY_BACKEND_PORT
             value: '4167'
       container:
         image: <your_image>
         env:
           - name: WELCOME_MESSAGE
             value: 'hello world!'

   where:

   <your_image>

   The container image built in the previous step.

   WELCOME_MESSAGE

   Set the value for the welcome.message optional configuration property.

3. Create a ConfigMap object that references the plugin.yaml file:

   $ oc create configmap --from-file=plugin.yaml -n openshift-devspaces telemetry-plugin-yaml

4. Create a manifest.yaml file with a Deployment, a Service, and a Route to expose the Apache web server. The Deployment references this ConfigMap object and places the plugin.yaml in the /var/www/html directory.

   kind: Deployment
   apiVersion: apps/v1
   metadata:
     name: apache
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: apache
     template:
       metadata:
         labels:
           app: apache
       spec:
         volumes:
           - name: plugin-yaml
             configMap:
               name: telemetry-plugin-yaml
               defaultMode: 420
         containers:
           - name: apache
             image: 'registry.redhat.io/rhscl/httpd-24-rhel7:latest'
             ports:
               - containerPort: 8080
                 protocol: TCP
             resources: {}
             volumeMounts:
               - name: plugin-yaml
                 mountPath: /var/www/html
     strategy:
       type: RollingUpdate
       rollingUpdate:
         maxUnavailable: 25%
         maxSurge: 25%
     revisionHistoryLimit: 10
     progressDeadlineSeconds: 600
   ---
   kind: Service
   apiVersion: v1
   metadata:
     name: apache
   spec:
     ports:
       - protocol: TCP
         port: 8080
         targetPort: 8080
     selector:
       app: apache
     type: ClusterIP
   ---
   kind: Route
   apiVersion: route.openshift.io/v1
   metadata:
     name: apache
   spec:
     host: apache-che.apps-crc.testing
     to:
       kind: Service
       name: apache
       weight: 100
     port:
       targetPort: 8080
     wildcardPolicy: None

5. Apply the manifest:

   $ {orch-cli} apply -f manifest.yaml

Procedure

1. Add the telemetry plugin to the components field of an existing Dev Workspace:

   components:
     ...
     - name: telemetry-plugin
       plugin:
         uri: <telemetry_plugin_url>

2. Start the Dev Workspace from the OpenShift Dev Spaces dashboard.

3. Optional: Configure the CheCluster Custom Resource to apply the telemetry plugin as a default for all Dev Workspaces. Default plugins are applied on Dev Workspace startup for new and existing Dev Workspaces.

   spec:
     devEnvironments:
       defaultPlugins:
       - editor: eclipse/che-theia/next
         plugins:
         - '<telemetry_plugin_url>'

   where:

   editor

   The editor identification to set the default plugins for.

   plugins

   List of URLs to devfile v2 plugins.

Verification

1. Verify that the telemetry plugin container is running in the Dev Workspace pod by checking the Workspace view within the editor.

   
2. Edit files within the editor and observe their events in the example telemetry server’s logs.

Procedure

1. Configure the CheCluster Custom Resource. See Section 2.2, “Use the CLI to configure the CheCluster Custom Resource”.

   spec:
     components:
       cheServer:
         extraProperties:
           CHE_LOGGER_CONFIG: "<key1=value1,key2=value2>"

   where:

   <key1=value1,key2=value2>

   Comma-separated list of key-value pairs, where keys are the names of the loggers as seen in the OpenShift Dev Spaces server log output and values are the required log levels.

   For example, to configure debug mode for the WorkspaceManager:

   spec:
     components:
       cheServer:
         extraProperties:
           CHE_LOGGER_CONFIG: "org.eclipse.che.api.workspace.server.WorkspaceManager=DEBUG"

Procedure

1. Configure the CheCluster Custom Resource:

   spec:
     components:
       cheServer:
         extraProperties:
           CHE_LOGGER_CONFIG: "che.infra.request-logging=TRACE"