Chapter 3. Configuring Java applications
To enable Cryostat to gather, store, and analyze Java Flight Recorder (JFR) data about target applications that run on Java Virtual Machine (JVM)s, you must configure the applications so that Cryostat can detect and connect to them.
You can configure the applications in any of the following ways:
- By using the Cryostat agent component for detection and connectivity, which is implemented as a Java Instrumentation Agent and acts as a plug-in for applications that run on the JVM
- By configuring the applications to allow Java Management Extensions (JMX) connections and using an OpenShift Service for detection and JMX for connectivity
- By using the Cryostat agent for detection and JMX for connectivity
Cryostat agent
From Red Hat build of Cryostat 2.4 onward, the Cryostat agent provides an HTTP API that the Cryostat server can use as an alternative to an application’s JMX port. By attaching a properly configured Cryostat agent to the workload applications that you deploy, you can use the full Cryostat feature set without any need for the target applications to expose a JMX port.
Before Red Hat build of Cryostat 2.4, the Cryostat agent provided a read-only HTTP API that supported a limited set of JFR operations only.
The Cryostat agent’s HTTP API can offer the following benefits compared to a JMX port:
- Greater security due to the reduced API surface area
- Deployment flexibility due to the Cryostat agent’s dual role as a Cryostat discovery plug-in
If the Cryostat agent detects that JMX is also configured on your application, the agent publishes itself to the Cryostat server with both agent HTTP API definitions and JMX URL definitions. In this situation, you can use whichever configuration option you prefer.
To use the Cryostat agent with your workload application, you must configure the application to pass the -javaagent
JVM flag with the path to the Cryostat agent‘s JAR file (for example, -javaagent:/deployment/app/lib/cryostat-agent.jar
). This setting enables your workload application’s JVM to load and initialize the Cryostat agent at startup. Once the Cryostat agent’s basic initialization is complete, your workload application’s normal startup process begins as usual.
Options for including the agent’s JAR file into your workload application
You can include the Cryostat agent’s JAR file into the workload application in different ways:
-
The simplest option is to add the JAR file to your application’s dependencies in the
pom.xml
orbuild.gradle
file. Your build tool (Maven or Gradle) downloads the JAR file for inclusion in your application build output. -
You can use a Maven plug-in, such as the
maven-dependency-plugin
, to provide more fine-grained control over the download and inclusion of the JAR file in the application build output. -
You can create a PersistentVolume storage volume that contains the JAR file. Then reconfigure your application’s
Deployment/DeploymentConfig
to mount the PersistentVolume and use-javaagent:/path/to/persistentvolume/cryostat-agent.jar
. The exact way to accomplish this task depends on the type of PersistentVolume provider you enabled in your OpenShift cluster.
Once the Cryostat agent is successfully added to your application container and loaded, your application’s stdout
and console
logs start displaying log messages from the Cryostat agent.
Agent configuration properties
You can specify configuration properties for the Cryostat agent in either of two ways:
-
Use JVM system property flags on the application (for example,
-Dcryostat.agent.api.writes-enabled=true
). -
Use environment variables by making all letters upper case and replacing any punctuation with underscores (for example,
CRYOSTAT_AGENT_API_WRITES_ENABLED=true
).
You must configure the following properties to enable the Cryostat agent to operate successfully:
|
This specifies the URL location of the Cryostat server back end that the Cryostat agent advertises itself to (that is, the internal OpenShift Service object) such as |
|
This specifies the URL location of the Cryostat agent instance or application itself. Cryostat uses this URL to perform health checks and request data from the agent. You can use the OpenShift/Kubernetes Downward API to determine this dynamically. For more information, see the Kubernetes Downward API documentation on |
Depending on your setup requirements, you can also configure the following agent properties:
|
This indicates whether the Cryostat agent allows write operations. This is set to Note
Even if this property is set to |
| This specifies the HTTP port number that the agent uses to bind its HTTP API (by default, 9977). If this conflicts with an existing port that your application or another tooling agent uses, you must specify a different port number. |
|
This specifies a label for identifying which application this Cryostat agent instance is attached to (by default, |
Remote Java Management Extensions (JMX) connections
JMX is a standard feature on a JVM with which you can monitor and manage target applications that run on the JVM. For Cryostat to use JMX, you must enable and configure JMX when you start the JVM because Cryostat requires the target applications to expose a JMX port.
Cryostat communicates with the target applications over this JMX port to start and stop JFR recordings and to pull JFR data over the network, enabling Cryostat to store and analyze this JFR data. Remote monitoring requires security to ensure that unauthorized persons cannot access application. Cryostat prompts you to enter your credentials before Cryostat can access any of the application’s JFR recordings.
Cryostat agent and JMX hybrid
You can configure your target applications to use a hybrid approach where you use both the Cryostat agent and JMX. With this approach, you use the Cryostat agent to detect the target applications and JMX to expose the JFR data to Cryostat, which allows for more flexibility.
For example, you can use the agent to detect the applications without needing to depend on specific port numbers and also use the JMX connections to start and stop JFR flight recordings on demand.
3.1. Configuring applications by using the Cryostat agent
You can use the Cryostat agent, implemented as a Java Instrumentation Agent, to configure target applications, so that Cryostat can detect the applications, collect data, and send the data to Cryostat for analysis. You can also optionally enable the Cryostat agent to accept requests from the Cryostat server to start, stop, and delete JFR recordings.
Red Hat build of Cryostat 2.4 distributes two different variations of the Cryostat agent’s JAR file. Depending on your setup requirements, you can use either of the following types of agent JAR file:
An all-in-one "shaded" JAR file that is self-contained and includes the agent code and all of its dependencies
This "shaded" JAR file provides the most convenient form of Cryostat agent to include in your existing applications, because you need to include only one additional agent JAR file. This is a common distribution pattern for similar agents and tools.
A standard JAR file that contains the agent code without any dependencies
This type of JAR file is useful if you know that dependency conflicts exist between the agent and your workload applications. If you intend to apply your own strategies to provide the correct versions of each dependency to satisfy both the agent and your applications’ requirements, you can use the standalone JAR file.
Previous releases provided one distribution of the Cryostat agent, which was an all-in-one "shaded" `JAR file. The following procedure describes how to install the "shaded" JAR file distribution of the Cryostat 2.4 agent.
As described in Configuring Java applications: Cryostat agent, the Cryostat 2.4 agent supports different options for including the agent’s JAR file into your workload applications. The following procedure describes how to add the "shaded" JAR file to your application’s dependencies in the pom.xml
or build.gradle
file.
Prerequisites
- Logged in to your Cryostat web console.
- Installed JDK version 11 or later.
Procedure
Install the Cryostat agent. Choose one of the following options, depending on your application build:
Using Maven:
Update the application
pom.xml
file with the Cryostat agent JAR file information.Example pom.xml
<project> ... <repositories> <repository> <id>redhat-maven-repository</id> <url>https://maven.repository.redhat.com/earlyaccess/all/</url> </repository> </repositories> ... <build> <plugins> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>3.3.0</version> <executions> <execution> <phase>prepare-package</phase> <goals> <goal>copy</goal> </goals> <configuration> <artifactItems> <artifactItem> <groupId>io.cryostat</groupId> <artifactId>cryostat-agent</artifactId> <version>0.3.0.redhat-00001</version> <classifier>shaded</classifier> </artifactItem> </artifactItems> <stripVersion>true</stripVersion> </configuration> </execution> </executions> </plugin> </plugins> ... </build> ... </project>
The next time you build your application, the Cryostat agent JAR file is available at
target/dependency/cryostat-agent-shaded.jar
.Using Gradle:
Update the
build.gradle
file.Example
build.gradle
filerepositories { … maven { url "https://maven.repository.redhat.com/earlyaccess/all/" credentials { username "myusername" password "mytoken" } } }
How you package the agent JAR file into the application depends on the Gradle plug-ins that you use for the build. For example, if you are using the Jib plug-in, update the
build.gradle
file as follows:Example
build.gradle
fileplugins { id 'java' id 'application' id 'com.google.cloud.tools.jib' version '3.3.1' id 'com.ryandens.javaagent-jib' version '0.5.0' } … dependencies { … javaagent 'io.cryostat:cryostat-agent:0.3.0.redhat-00001:shaded'
Update the Docker file. The following example uses the
JAVA_OPTS
environment variable to pass the relevant JVM information.Example
... COPY target/dependency/cryostat-agent.jar /deployments/app/ ... ENV JAVA_OPTS="-javaagent:/deployments/app/cryostat-agent-shaded.jar"
Rebuild the container image that is specific to your application.
docker build -t docker.io/myorg/myapp:latest -f src/main/docker/Dockerfile
To supply the JVM system properties or environment variables that you need to configure the Cryostat agent, push the updated image and then modify your application deployment.
Example
apiVersion: apps/v1 kind: Deployment ... spec: ... template: ... spec: containers: - name: sample-app image: docker.io/myorg/myapp:latest env: - name: CRYOSTAT_AGENT_APP_NAME value: "myapp" # Replace this with the Kubernetes DNS record # for the Cryostat Service - name: CRYOSTAT_AGENT_BASEURI value: "http://cryostat.mynamespace.mycluster.svc:8181" - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: CRYOSTAT_AGENT_CALLBACK value: "http://$(POD_IP):9977" 1 # Replace "abcd1234" with a base64-encoded authentication token - name: CRYOSTAT_AGENT_AUTHORIZATION 2 value: "Bearer abcd1234" - name: CRYOSTAT_AGENT_API_WRITES_ENABLED 3 value: true ports: - containerPort: 9977 protocol: TCP resources: {} restartPolicy: Always status: {}
-
<1>: Port number
9977
is the default HTTP port that the agent exposes for the internal web server that services Cryostat requests. You can change this port number if it conflicts with your target application into which the agent is installed. -
<2>: The
CRYOSTAT_AGENT_AUTHORIZATION
value shows the credentials that the agent includes in the API requests to Cryostat to advertise its own presence or to push JFR data. You can also create a KubernetesService Account
for this purpose and replaceabcd1234
with the base64-encoded authentication token associated with the service account. -
<3>: The
CRYOSTAT_AGENT_API_WRITES_ENABLED
variable is set tofalse
by default. If you want the Cryostat agent to accept requests from the Cryostat server to start, stop, or delete JFR flight recordings, you must set this variable totrue
.
-
<1>: Port number
3.2. Configuring applications by using JMX connections
For Cryostat to detect and communicate with your target Java applications, you can configure the applications to allow remote Java Management Extensions (JMX) connections.
Prerequisites
- Logged in to your Cryostat web console.
- Created a Cryostat instance in your project.
Procedure
To enable remote JMX connections, complete the following steps:
On your application, define the following Java system property:
-Dcom.sun.management.jmxremote.port=<port_num>
NoteTo add the
-Dcom.sun.management.jmxremote.port=<port_num>
property without having to rebuild the target application, you can set theJAVA_OPTS_APPEND
environment variable on the application.JAVA_OPTS_APPEND
is an environment variable that is used by Red Hat Universal Base Images (UBI) only.If you use Red Hat UBI to build application images, set the
JAVA_OPTS_APPEND
variable at build time in the application Docker file, or at runtime by running the following command:oc set env deployment <name> JAVA_OPTS_APPEND="..."
If you do not use Red Hat UBI to build application images, refer to the documentation for your base image for information about how to add the Java system properties at build time or runtime.
Specify that the application listens for remote JMX connections by allowing traffic to the application. Use an Red Hat OpenShift Service and specify the following values for its remote JMX port:
Example
service.yaml
apiVersion: v1 kind: Service ... spec: ports: - name: "jfr-jmx" port: 9091 targetPort: 9091 ...
Secure the remote JMX connections:
Enable and configure authentication and SSL/TLS for the remote JMX connections in your application:
-Dcom.sun.management.jmxremote.port=<port_num> # enable JMX authentication -Dcom.sun.management.jmxremote.authenticate=true # define users for JMX auth -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password> # set permissions for JMX users -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access> # enable JMX SSL -Dcom.sun.management.jmxremote.ssl=true # enable JMX registry SSL -Dcom.sun.management.jmxremote.registry.ssl=true # set your SSL keystore -Djavax.net.ssl.keyStore=</path/to/keystore> # set your SSL keystore password -Djavax.net.ssl.keyStorePassword=<password>
Configure Cryostat to trust the application TLS certificate. Create a secret for the application in the same namespace as your Cryostat application and configure Cryostat to refer to the secret. To create a secret for the certificate, run the following command:
oc create secret generic myapp-cert --from-file=tls.crt=/path/to/cert.pem
NoteThe certificate must be in a
.pem
file format.- When you create your Cryostat instance, add the secret to the list of trusted TLS certificates. For more information, see Configuring TLS certificates.
To allow your applications to verify that Cryostat is connecting to them by a means other than password authentication, enable TLS client authentication:
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true -Djavax.net.ssl.trustStore=</path/to/truststore> -Djavax.net.ssl.trustStorePassword=<password>
NoteTLS client authentication requires the cert-manager operator for Red Hat OpenShift.
If you use TLS client authentication for remote JMX connections, the application truststore must contain a Cryostat certificate. The Cryostat operator cert-manager integration creates a self-signed certificate for the Cryostat deployment. This certificate is located in the
<cryostat>-tls
secret, where <cryostat> is the name of the Cryostat instance you created.NoteThe cert-manager Operator also places a Java keystore truststore in the secret.
To mount this truststore in your application deployment, run the following command, replacing "<myapp>" with the name of your application deployment and "<cryostat>" with the name of your Cryostat instance:
oc set volumes deploy <myapp> --add --name=truststore \ --secret-name=<cryostat>-tls --sub-path=truststore.p12 \ --mount-path=/var/run/secrets/<myapp>/truststore.p12
The Cryostat operator generates the truststore password, which you can find in the
<cryostat>-keystore
secret. To mount this as an environment variable in your application deployment, run the following command:oc set env deploy <myapp> --from='secret/<cryostat>-keystore'
Configure the Java arguments for the container. Run the following commands:
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true -Djavax.net.ssl.trustStore=/var/run/secrets/<myapp>/truststore.p12 -Djavax.net.ssl.trustStorePassword="$(KEYSTORE_PASS)"
WarningIf you deployed Cryostat and your applications in a testing environment, you might want to configure the target applications without any JMX or TLS authentication. You can do so by using the following set of Java system properties, however, this configuration is not secure and not recommended.
-Dcom.sun.management.jmxremote.port=<port_num> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
Additional resources
3.3. Configuring applications by using the Cryostat agent and JMX connections
You can configure target applications that run on Java Virtual Machines (JVM) to use a combination of the Cryostat agent and Java Management Extensions (JMX) connections to detect and communicate with the target applications.
You use the Cryostat agent to detect and communicate with the target application and use JMX to expose the Java Flight Recorder (JFR) data.
You must configure the Cryostat agent to communicate with Cryostat about itself and that the agent is reachable through JMX rather than through HTTP.
Prerequisites
- Logged in to your Cryostat web console.
- Created a Cryostat instance in your project.
Procedure
Install the Cryostat agent. For application builds that use Maven, update the application
pom.xml
file with the Cryostat agent JAR file information.Example
pom.xml
file<project> ... <repositories> <repository> <id>redhat-maven-repository</id> <url>https://maven.repository.redhat.com/earlyaccess/all/</url> </repository> </repositories> ... <build> <plugins> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>3.3.0</version> <executions> <execution> <phase>prepare-package</phase> <goals> <goal>copy</goal> </goals> <configuration> <artifactItems> <artifactItem> <groupId>io.cryostat</groupId> <artifactId>cryostat-agent</artifactId> <version>0.3.0.redhat-00001</version> <classifier>shaded</classifier> </artifactItem> </artifactItems> <stripVersion>true</stripVersion> </configuration> </execution> </executions> </plugin> </plugins> ... </build> ... </project>
Modify your application deployment:
Example
apiVersion: apps/v1 kind: Deployment ... spec: ... template: ... spec: containers: - name: sample-app image: docker.io/myorg/myapp:latest env: - name: CRYOSTAT_AGENT_APP_NAME value: "myapp" # Replace this with the Kubernetes DNS record # for the Cryostat Service - name: CRYOSTAT_AGENT_BASEURI value: "http://cryostat.mynamespace.mycluster.svc:8181" - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: CRYOSTAT_AGENT_CALLBACK value: "http://$(POD_IP):9977" - name: CRYOSTAT_AGENT_AUTHORIZATION # Replace "abcd1234" with a base64-encoded authentication token value: "Bearer abcd1234" # This environment variable is key to the Cryostat agent # and JMX "hybrid" setup. # Set the Cryostat agent to register itself with Cryostat # as reachable through JMX, rather than reachable through HTTP. - name: CRYOSTAT_AGENT_REGISTRATION_PREFER_JMX value: "true" # Configure the application to load the agent JAR file and # to enable JMX, so that the Cryostat agent can register # itself as reachable through JMX. # To configure authentication and SSL/TLS for the JMX # connections, see <1>. - name: JAVA_OPTS value: >- -javaagent:/deployments/app/cryostat-agent-shaded.jar -Dcom.sun.management.jmxremote.port=9091 1 ports: - containerPort: 9977 protocol: TCP resources: {} restartPolicy: Always status: {}
<1>: To configure authentication and SSL/TLS for the JMX connections and view further configuration options, see Configuring applications by using JMX connections.
To enable Cryostat to detect the target application and connect to the Cryostat agent, configure an application
Service
:Example
apiVersion: v1 kind: Service ... spec: ports: - name: "jfr-jmx" port: 9091 targetPort: 9091 - name: "cryostat-agent" port: 9977 targetPort: 9977 ...