Getting started with Cryostat
Abstract
Preface Copy linkLink copied to clipboard!
The Red Hat build of Cryostat is a container-native implementation of JDK Flight Recorder (JFR) that you can use to securely monitor the Java Virtual Machine (JVM) performance in workloads that run on an OpenShift Container Platform cluster. You can use Cryostat to start, stop, retrieve, archive, import, and export JFR data for JVMs inside your containerized applications by using a web console or an HTTP API.
Depending on your use case, you can store and analyze your recordings directly on your Red Hat OpenShift cluster by using the built-in tools that Cryostat provides or you can export recordings to an external monitoring application to perform a more in-depth analysis of your recorded data.
Making open source more inclusive Copy linkLink copied to clipboard!
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Chapter 1. Overview of Cryostat Copy linkLink copied to clipboard!
Cryostat is a container-native Java application based on JDK Flight Recorder (JFR) that you can use to monitor Java Virtual Machine (JVM) performance for containerized workloads that run on a Red Hat OpenShift cluster.
You can deploy Cryostat in a container in a Red Hat OpenShift project that hosts your containerized Java applications. You can create JVM targets that correspond to the JVM instances that you use to run your containerized workload. You can connect Cryostat to the JVM targets to record and analyze data about heap and non-heap memory usage, thread count, garbage collection, and other performance metrics for each JVM target.
You can use the tools that are included with Cryostat to monitor the performance of your JVMs in real time, capture JDK Flight Recorder (JFR) recordings and snapshots, generate Automated Analysis reports, and visualize your recorded performance data by using a Grafana dashboard.
The Cryostat web console and HTTP API provides a way to analyze your JVM performance data inside the container without having to rely on an external monitoring application. However, you can also export your recordings from Cryostat into an external instance of JDK Mission Control (JMC) when you need to perform a deeper analysis of your data outside of a cluster environment.
Cryostat supports role-based access control (RBAC) as a standard feature of OpenShift Container Platform.
You can install Cryostat inside a Red Hat OpenShift project by using the Operator Lifecycle Manager (OLM).
You can also download the latest Cryostat component images from the Red Hat Ecosystem Catalog. The following container images exist for Cryostat 4.0 on the Red Hat Ecosystem Catalog:
- Cryostat
- Red Hat build of Cryostat Operator
- Red Hat build of Cryostat Operator Bundle
- Cryostat Agent Init
- Cryostat Console Plugin
- Cryostat Database
- Cryostat Grafana Dashboard
- Cryostat OpenShift OAuth Proxy
- Cryostat Reports
- Cryostat Storage
- JFR data source
Chapter 2. Installing Cryostat on Red Hat OpenShift by using a Red Hat build of Cryostat Operator Copy linkLink copied to clipboard!
You can install the Red Hat build of Cryostat Operator in a project on your Red Hat OpenShift cluster. You can then use the Red Hat build of Cryostat Operator to create single namespace or multi-namespace Cryostat instances. You can control these instances by using a GUI that is accessible from the Red Hat OpenShift web console.
2.1. Installing the Cryostat Operator Copy linkLink copied to clipboard!
You can use the Operator Lifecycle Manager (OLM) to install the Red Hat build of Cryostat Operator in a project on your Red Hat OpenShift cluster.
If you need to upgrade your Red Hat build of Cryostat Operator subscription from Cryostat 3.0 to Cryostat 4.0, you must change the update channel from stable-3.0
to stable
.
Prerequisites
Created an OpenShift Container Platform 4.12 or later cluster.
NoteThe range of Cryostat Operator functionality that will be available depends on the version of OpenShift Container Platform that you are using. If you want the Cryostat Operator to support automatic installation of the Cryostat web console plug-in, you must use OpenShift Container Platform 4.15 or later.
- Created a Red Hat OpenShift user account with permissions to install Red Hat build of Cryostat Operator in a project.
- Installed Operator Lifecycle Manager (OLM) on your cluster.
Installed cert-manager with the cert-manager Operator for Red Hat OpenShift.
- If you are using OpenShift Container Platform 4.12 or later, you can install the cert-manager Operator for Red Hat OpenShift. For more information, see cert-manager Operator for Red Hat OpenShift (OpenShift Container Platform).
- Logged in to Red Hat OpenShift by using the Red Hat OpenShift web console.
Procedure
- In the Red Hat OpenShift Container Platform, navigate to Home > Projects.
- Select the name of the project in which you want to install the Red Hat build of Cryostat Operator.
Install the Red Hat build of Cryostat Operator:
- In the navigation menu of your web console, navigate to Operators > OperatorHub.
- Select Red Hat build of Cryostat from the list. You can use the search box in the upper part of the screen to find the Red Hat build of Cryostat Operator.
To install the Red Hat build of Cryostat Operator in your project, click Install.
NoteIn the Installation mode area, the All namespaces on the cluster (default) radio button is the only available option.
When you click Install, the Red Hat build of Cryostat Operator is automatically added to your list of installed Operators in the Red Hat OpenShift Container Platform web console.
2.2. Automatic installation of the Cryostat web console plug-in Copy linkLink copied to clipboard!
When you install the Cryostat Operator on Red Hat OpenShift Container Platform 4.15 or later, the Cryostat Operator automatically installs the Cryostat web console plug-in. In this situation, a prompt is displayed in the upper-right corner of the panel to indicate that a web console update is available.
Figure 2.1. Prompt about web console update
When you click Refresh web console in the prompt area, a Cryostat option is automatically added to the left-hand navigation pane in the Red Hat OpenShift Container Platform web console. By clicking the Cryostat option, you can access Cryostat web console functionality directly from the OpenShift Container Platform console without needing to access separate user interfaces for individual Cryostat instances.
The Cryostat Operator supports installation of the Cryostat web console plug-in on Red Hat OpenShift Container Platform 4.15 or later only. If the Operator detects that a cluster is on a version earlier than 4.15, the Operator cannot install the web console plug-in but all other Cryostat functionality works as expected.
You must create at least one Cryostat instance that the web console plug-in can interact with.
2.3. Creating a Cryostat instance Copy linkLink copied to clipboard!
When you install the Cryostat Operator, the Red Hat OpenShift Container Platform web console prompts you to create the custom resource (CR) for a Cryostat instance. You can create a CR either manually or automatically. If you want to create the CR manually, see step 1 of this procedure. If you want to create the CR automatically, see step 2 of this procedure.
Prerequisites
- Installed the Cryostat Operator.
Procedure
If you want to create the CR manually, complete the following steps:
Navigate to Operators > Installed Operators by using the web console and select Red Hat build of Cryostat Operator from the list of installed operators:
Figure 2.2. Viewing the Red Hat build of Cryostat operator in the list of installed operators
- Click the Details tab.
To create a Cryostat instance, go to the Provided APIs section. Then, under Cryostat, click Create instance.
NoteThe Cryostat API enables you to create both single-namespace and multi-namespace Cryostat instances.
Figure 2.3. Selecting the Cryostat API that is provided by the Red Hat build of Cryostat Operator
- Click either the Form view radio button or the YAML view radio button. If you want to enter your information in the YAML configuration file, click YAML view .
- Specify a unique name for the instance of Cryostat that you want to create.
- Optional: In the Labels field, specify a label or annotation for the Operand workload you want to deploy.
In the Target Namespaces field, select namespaces whose workloads you want to permit this instance of Cryostat to access and work with. Optionally, you can select the same namespace where you installed Cryostat or you can choose a different namespace. To add additional namespaces, click +Add Target Namespace.
NoteIf you leave this field blank, the Operator uses the installation namespace of your Cryostat instance as the target namespace.
ImportantUsers who can access the Cryostat instance have access to all target applications in any namespace that is visible to that Cryostat instance. Therefore, when you deploy a multi-namespace Cryostat instance, you must consider which namespaces to select for monitoring, which namespace to install Cryostat into, and which users you want to grant access to.
You can also specify additional configuration options for your deployment:
Figure 2.4. Creating an instance of Cryostat by using a form in the web console
Alternatively, you can use a YAML template to create your instance and specify additional configuration options instead of using the form:
Figure 2.5. Creating an instance of Cryostat by using a YAML template in the web console
If you want to create the CR by using the automatic prompt option, follow the prompt’s instructions and then complete the following steps:
- Click either the Form view radio button or the YAML view radio button. If you want to enter your information in the YAML configuration file, click YAML view.
- Specify a unique name for the instance of Cryostat that you want to create.
- Optional: In the Labels field, specify a label or annotation for the Operand workload you want to deploy.
In the Target Namespaces field, select namespaces whose workloads you want to permit this instance of Cryostat to access and work with. Optionally, you can select the same namespace where you installed Cryostat or you can choose a different namespace. To add additional namespaces, click +Add Target Namespace.
NoteIf you leave this field blank, the Operator uses the installation namespace of your Cryostat instance as the target namespace.
ImportantUsers who can access the Cryostat instance have access to all target applications in any namespace that is visible to that Cryostat instance. Therefore, when you deploy a multi-namespace Cryostat instance, you must consider which namespaces to select for monitoring, which namespace to install Cryostat into, and which users you want to grant access to.
You can also specify additional configuration options for your deployment:
Figure 2.6. Creating an instance of Cryostat by using a form in the web console
Alternatively, you can use a YAML template to create your instance and specify additional configuration options instead of using the form:
Figure 2.7. Creating an instance of Cryostat by using a YAML template in the web console
To start the creation process for your Cryostat instance, click Create.
You must wait for all resources of your Cryostat instance to be ready before you can access it.
Verification
- In the navigation menu of the web console, click Operators, then click Installed Operators.
- From the table of installed operators, select Red Hat build of Cryostat.
Select the Cryostat tab.
Your Cryostat instance opens in the table of instances and lists the following conditions:
-
TLSSetupComplete
is set totrue
. -
MainDeploymentAvailable
is set totrue
. Optional: If you enabled the reports generator service then
ReportsDeploymentAvailable
is shown and set totrue
.Figure 2.8. Example of conditions set to True under the Status column for a Cryostat instance on OpenShift
-
Optional: Select your Cryostat instance from the Cryostat table. Go to the Cryostat Conditions table, where you can see more information for each condition.
Figure 2.9. Example of a Cryostat Conditions table that lists each condition and its criteria
Additional resources
Chapter 3. Configuring Java applications Copy linkLink copied to clipboard!
To enable Cryostat to gather, store, and analyze Java Flight Recorder (JFR) data about target applications that run on Java Virtual Machines (JVMs), you must configure the applications so that Cryostat can detect and connect to them.
You can configure the applications in any of the following ways:
- Cryostat agent
You can use the Cryostat agent component for detection and connectivity, which is implemented as a Java Instrumentation Agent and acts as a plug-in for applications that run on the JVM.
The Cryostat agent provides an HTTP API that the Cryostat server can use as an alternative to an application’s JMX port. By attaching a properly configured Cryostat agent to the workload applications that you deploy, you can use the full Cryostat feature set without any need for the target applications to expose a JMX port.
The Cryostat agent’s HTTP API can offer the following benefits compared to a JMX port:
- Greater security due to the reduced API surface area
- Deployment flexibility due to the Cryostat agent’s dual role as a Cryostat discovery plug-in
For more information, see Working with the Cryostat agent.
- Remote Java Management Extensions (JMX) connections
You can configure your target applications to allow Java Management Extensions (JMX) connections. This type of configuration uses an OpenShift Service for detection and JMX for connectivity.
JMX is a standard feature on a JVM with which you can monitor and manage target applications that run on the JVM. For Cryostat to use JMX, you must enable and configure JMX when you start the JVM, because Cryostat requires the target applications to expose a JMX port.
Cryostat communicates with the target applications over this JMX port to start and stop JFR recordings and to pull JFR data over the network, enabling Cryostat to store and analyze this JFR data. Remote monitoring requires security to ensure that unauthorized persons cannot access application. Cryostat prompts you to enter your credentials before Cryostat can access any of the application’s JFR recordings.
For more information, see Configuring applications to use JMX connections.
- Cryostat agent and JMX hybrid
You can configure your target applications to use a hybrid approach where you use both the Cryostat agent and JMX. With this approach, you use the Cryostat agent to detect the target applications and JMX to expose the JFR data to Cryostat, which allows for more flexibility.
For example, you can use the agent to detect the applications without needing to depend on specific port numbers and also use the JMX connections to start and stop JFR flight recordings on demand.
If the Cryostat agent detects that JMX is also configured on your application, the agent publishes itself to the Cryostat server with both agent HTTP API definitions and JMX URL definitions. In this situation, you can use whichever configuration option you prefer.
For more information, see Configuring applications to use the Cryostat agent and JMX connections.
3.1. Working with the Cryostat agent Copy linkLink copied to clipboard!
The Cryostat agent is implemented as a Java Instrumentation Agent that acts as a plug-in for applications that run on the JVM. The Cryostat agent provides an HTTP API that offers greater deployment flexibility than a JMX port due to the agent’s dual role as a discovery plug-in. You can configure your target applications to use the agent’s HTTP API for both detection and connectivity with Cryostat.
3.1.1. Cryostat agent features Copy linkLink copied to clipboard!
Red Hat build of Cryostat offers different distributions of the Cryostat agent and supports different ways of deploying and using the agent. Consider the following information to help you determine the type of Cryostat agent deployment and use cases that best suit your needs.
3.1.1.1. Automatic or manual configuration of the Cryostat agent Copy linkLink copied to clipboard!
You can use the Cryostat Operator to configure the Cryostat agent automatically. Alternatively, you can choose to configure the Cryostat agent manually, if you prefer.
Due to a known issue that Red Hat is currently investigating, manual configuration of the Cryostat agent does not work correctly in Cryostat 4.0. Red Hat plans to fix this issue in a future release. If you want your applications to use the Cryostat agent in the 4.0 release, you must allow the Cryostat Operator to configure the Cryostat agent automatically.
Automatic configuration of the agent
From Cryostat 4.0 onward, you can use the Cryostat Operator to configure the Cryostat agent automatically. To enable this feature, you can add cryostat.io/name
and cryostat.io/namespace
labels to your application deployment that identify which Cryostat instance the pod should work with. When the Cryostat Operator detects that an application deployment has the cryostat.io/name
and cryostat.io/namespace
labels, the Operator mounts a volume containing the agent JAR file to this application.
As part of this automatic configuration feature, Cryostat uses a mutating admission webhook that modifies pods at creation time to inject the Cryostat agent. The webhook also configures the agent to connect to a selected Cryostat instance automatically. In this situation, only pods that contain the required cryostat.io/name
and cryostat.io/namespace
labels invoke the webhook, which helps to simplify setup of the Cryostat agent with arbitrary pods. In addition, the Operator acts only if these labels are specified in an application deployment within the list of target namespaces for the Cryostat instance. Otherwise, for security reasons, the Operator does not configure the application to communicate with this Cryostat instance.
The webhook uses a container-based distribution of the Cryostat agent (cryostat-agent-init
) for automatic configuration and injection of the agent into selected pods. The webhook’s use of the cryostat-agent-init
container image removes any need for users to download the agent JAR file from the Red Hat Maven repository.
For more information about allowing the Cryostat Operator to configure the agent automatically, see Enabling automatic configuration of the agent by the Cryostat Operator.
Manual configuration of the agent
As an alternative to the automatic configuration feature, you can choose to configure the Cryostat agent manually, if you prefer. However, manual configuration incurs more time overheads and the potential for misconfiguration compared to automatic configuration by the Cryostat Operator. Before Cryostat 4.0, a manual configuration approach was the only available way to configure the Cryostat agent.
This section is included for informational purposes only. Due to a known issue that Red Hat is currently investigating, manual configuration of the Cryostat agent does not work correctly in Cryostat 4.0. If you want your applications to use the Cryostat agent in the 4.0 release, you must allow the Cryostat Operator to configure the Cryostat agent automatically.
For manual configuration of the agent, Red Hat build of Cryostat 4.0 distributes two different variations of the Cryostat agent’s JAR file:
An all-in-one "shaded" JAR file that is self-contained and includes the agent code and all of its dependencies
This "shaded" JAR file provides the most convenient form of Cryostat agent to include in your existing applications, because you need to include only one additional agent JAR file. This is a common distribution pattern for similar agents and tools. Before Cryostat 2.4, the “shaded” JAR file was the only available distribution of the Cryostat agent. The “shaded” JAR file name is in the form
cryostat-agent-0.5.X.redhat-xxxxx-shaded.jar
, where0.5.X
andxxxxx
represent the agent version and build number, respectively.A standard JAR file that contains the agent code without any dependencies
This type of JAR file is useful if you know that dependency conflicts exist between the agent and your workload applications. If you intend to apply your own strategies to provide the correct versions of each dependency to satisfy both the agent and your applications’ requirements, you can use the standalone JAR file. The standard JAR file name is in the form
cryostat-agent-0.5.X.redhat-xxxxx.jar
, where where0.5.X
andxxxxx
represent the agent version and build number, respectively.
If you want to configure the Cryostat agent manually, the all-in-one “shaded” JAR file is the preferred way to deploy the agent. Do not use the standard JAR file unless you intend to apply your own strategies to resolve any dependency conflicts between the manually configured agent and your workload applications.
These agent JAR files are available for download from the Red Hat Maven Repository.
3.1.1.2. Static or dynamic attachment to the JVM Copy linkLink copied to clipboard!
Depending on your requirements, the Cryostat agent can use a static or dynamic approach for attaching to the JVM.
Static attachment to the JVM
Static attachment means that your workload application’s JVM loads and initializes the Cryostat agent at JVM startup.
Static attachment to the JVM is supported regardless of whether the Cryostat Operator configures the agent automatically or you configure the agent manually. If you want the Cryostat Operator to configure the agent automatically, static attachment is the only supported way for the agent to attach to the JVM.
If you use the Cryostat Operator to configure the agent automatically, the Operator handles all configuration tasks and OpenShift Container Platform automatically redeploys the application.
For more information, see Configuring applications to use the Cryostat agent.
Dynamic attachment to the JVM
Dynamic attachment means that the Cryostat agent can attach dynamically to an application JVM that is already running without requiring an application restart.
Dynamic attachment to the JVM requires that you configure the agent manually. The dynamic attachment approach is not supported for agents that the Cryostat Operator configures automatically.
This dynamic attachment feature has the following requirements:
-
You must ensure that the agent’s JAR file is copied to the JVM’s file system (for example, by using the
oc cp
command). -
You must be able to run the agent as a separate process on the same host or within the same application (for example, by using the
oc exec
command).
The dynamic attachment feature supports ad hoc one-time profiling or troubleshooting workflows where you might not need the agent to be attached every time the JVM starts. Dynamic attachment also suits situations where you cannot or do not want to reconfigure your application for the sole purpose of attaching the agent. Because the agent can attach to a running JVM without requiring an application restart, this also means there is no application downtime.
For more information, see Launching the Cryostat agent as a standalone process for dynamic attachment to the JVM.
3.1.1.3. Options for including the agent JAR file into your workload application Copy linkLink copied to clipboard!
The options for including the agent JAR file into your workload application vary depending on whether you use automatic or manual configuration of the agent and static or dynamic attachment to the JVM.
For automatic configuration of the agent
If you want the Cryostat Operator to configure the agent automatically, the Operator uses the
cryostat-agent-init
container to mount a volume containing the agent JAR file to the relevant application. In this situation, the application deployment must include thecryostat.io/name
andcryostat.io/namespace
labels to identify the Cryostat instance that the pod should work with.For manual configuration of the agent with dynamic attachment to the JVM
For dynamic one-time attachment to a running JVM, you can copy the agent’s JAR file to the JVM’s file system by using the
oc cp
command.
3.1.1.4. Agent configuration properties Copy linkLink copied to clipboard!
This section is included for informational purposes only. Due to a known issue that Red Hat is currently investigating, manual configuration of the Cryostat agent does not work correctly in Cryostat 4.0. If you want your applications to use the Cryostat agent in the 4.0 release, you must allow the Cryostat Operator to configure the Cryostat agent automatically.
If you choose to configure the agent manually, you can specify configuration properties for the Cryostat agent in either of two ways:
-
Use JVM system property flags on the application (for example,
-Dcryostat.agent.api.writes-enabled=true
). -
Use environment variables by making all letters upper case and replacing any punctuation with underscores (for example,
CRYOSTAT_AGENT_API_WRITES_ENABLED=true
).
If you use the Cryostat Operator to configure the agent automatically, the Operator configures the relevant properties.
Required agent properties
If you are configuring the agent manually, you must configure the following properties to enable the Cryostat agent to operate successfully:
|
This specifies the URL location of the Cryostat server back end that the Cryostat agent advertises itself to (that is, the internal OpenShift Service object) such as |
|
This specifies the URL location of the Cryostat agent instance or application itself. Cryostat uses this URL to perform health checks and request data from the agent. You can use the OpenShift/Kubernetes Downward API to determine this URL dynamically by setting this property to a value such as |
|
This specifies a header value that the Cryostat agent can include in API requests to the Cryostat server (for example, Note
This property takes precedence over |
|
This specifies an authorization type that is used in conjunction with the Note
The agent ignores this property if you specify a value for the Valid authorization types are as follows:
This property is set to |
|
This specifies an authorization value that is used in conjunction with the Note
The agent ignores this property if you specify a value for the Consider the following guidelines:
By default, this property is set to |
Optional agent properties
Depending on your setup requirements, if you are configuring the agent manually, you can also configure the following agent properties:
|
This indicates whether the Cryostat agent allows write operations. This is set to Note
Even if this property is set to |
| This specifies the HTTP port number that the agent uses to bind its HTTP API (by default, 9977). If this conflicts with an existing port that your application or another tooling agent uses, you must specify a different port number. |
|
This specifies a label for identifying which application this Cryostat agent instance is attached to (by default, |
The preceding list describes only a subset of the available configuration properties for the Cryostat agent. For a full list of agent properties, see Agent Properties.
3.1.2. Configuring applications to use the Cryostat agent Copy linkLink copied to clipboard!
You can configure your target applications to use the Cryostat agent, implemented as a Java Instrumentation Agent, so that Cryostat can detect these applications, collect data, and send this data to Cryostat for analysis.
The Cryostat agent supports the following configuration use cases:
3.1.2.1. Enabling automatic configuration of the agent by the Cryostat Operator Copy linkLink copied to clipboard!
If you want to allow the Cryostat Operator to configure the agent automatically, you must add certain labels to your application deployment.
Procedure
In your appllication deployment, add
cryostat.io/name
andcryostat.io/namespace
labels.Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In the preceding example, replace
<namespace>
with the installation namespace of your Cryostat instance and replace<name>
with the name of your Cryostat CR.By default, the agent’s callback server binds to container port 9977. If you want to specify a different port number for the Cryostat agent’s callback server, add a
cryostat.io/callback-port
label in your application deployment.Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In the preceding example, replace
<port>
with the port number that you want the agent’s callback server to bind to.If you want to specify which container the agent should be injected into, add a
cryostat.io/container
label in your application deployment.Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In the preceding example, replace
<container>
with the name of the relevant container.NoteIf you have multiple containers running within the same pod, you may only select one container for agent injection. In a multi-container pod, this feature is useful if you have only one Java container where you want to inject the agent. If you do not include this label, the mutation webhook injects the agent into the first container image by default.
If you want the injected agent to have read-only access to the pod, set the
cryostat.io/read-only
label totrue
in your application deployment.Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThis feature gives you more control over the security of your injected agents.
By default, logging is disabled for the Cryostat agent. If you want to enable agent logging, add a
cryostat.io/log-level
label in your application deployment.Valid values are:
trace
,debug
,info
,warn
,error
, oroff
. The default value isoff
.Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThis feature gives you more control over the appropriate level that you want to use for agent logging, which might be useful for troubleshooting or debugging the agent configuration.
When you enable agent logging, these agent logs might become confused with the host application’s log messages. Depending on the logging level that you set, agent logs might end up triggering other alerting systems that check for warning or error-level messages.
When the Cryostat Operator detects that an application deployment has the cryostat.io/name
and cryostat.io/namespace
labels, the Operator uses the cryostat-agent-init
container to mount a volume containing the agent JAR file to this application. In this situation, the Operator acts only if these labels are specified in an application deployment within the list of target namespaces for the Cryostat instance. Otherwise, for security reasons, the Operator does not configure the application to communicate with this Cryostat instance.
3.1.2.2. Launching the Cryostat agent as a standalone process for dynamic attachment to the JVM Copy linkLink copied to clipboard!
If you want the Cryostat agent to attach dynamically to an application JVM that is already running, you can launch the agent as a standalone Java process.
This procedure is only relevant if you want to use the dynamic attachment feature, which allows the Cryostat agent to attach to a running JVM on an ad hoc one-time basis. If you want your workload application’s JVM to load and initialize the Cryostat agent at JVM startup, see Enabling automatic configuration of the agent.
Prerequisites
Copied the agent’s JAR file to the JVM’s file system by using the
oc cp
command.NoteThe latest “shaded” JAR file is available for download from the Red Hat Maven repository. The “shaded” JAR file name is in the form
cryostat-agent-0.5.X.redhat-xxxxx-shaded.jar
, where0.5.X
andxxxxx
represent the agent version and build number, respectively.
Procedure
Enter the following command:
java -jar target/<agent_jar_file> <pid>
$ java -jar target/<agent_jar_file> <pid>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In the preceding command, replace <agent_jar_file> with the agent’s JAR file name and replace <pid> with the process ID (PID) of the JVM that you want the agent to attach to.
For example:
java -jar target/cryostat-agent-0.5.0.redhat-00001-shaded.jar 1234
$ java -jar target/cryostat-agent-0.5.0.redhat-00001-shaded.jar 1234
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe preceding example is shown for illustration only and might not reflect the latest available version and build of the “shaded” JAR file.
When you run the preceding command, the agent process uses its Attach providers to look up the specified PID. If the specified PID is found, the agent process attaches to this PID and attempts to load the agent’s JAR file into this JVM, which then bootstraps into the normal agent launch process.
Agent launch behavior based on the PID value
Consider the following guidelines for agent launch behavior depending on the PID value:
- If you specify an invalid PID, the agent cannot launch successfully.
-
If you specify a wildcard asterisk (
*
) as the PID value, the agent’s JAR file attempts to bootstrap into every JVM that it finds. -
If you specify
0
as the PID value or if you do not specify any PID value, the agent checks if exactly one candidate JVM is available. If only one JVM is available, the agent attempts to bootstrap into this JVM. If multiple JVMs or no JVMs are available, the agent cannot launch successfully.
Late-binding configuration options
When launching the Cryostat agent as a standalone process, you can also specify additional late-binding configuration options to the agent launcher by using command-line options.
For example:
For information about the available options and their behavior, run the java -jar target/<filename>.jar -h
help command, where <filename> represents the name of the “shaded” agent JAR file that you copied.
As shown in the preceding example, when launching the Cryostat agent as a standalone process, ensure that you specify the -D
flags for any late-binding configuration options after the -jar <path_to_jarfile>
argument. The position of the -D
flags is important here:
-
If you specify the
-D
flag for a system property before the-jar <path_to_jarfile>
argument, this property is set on the standalone launcher JVM process. This could lead to unexpected issues when launching the agent as a standalone process. -
If you specify the
-D
flags for system properties after the-jar <path_to_jarfile>
argument, these properties are passed as arguments to the agent launcher. When the launcher bootstraps the agent into the host JVM, the agent applies these properties onto the JVM at agent startup.
As shown in the preceding example, ensure that you specify any required agent properties such as cryostat.agent.baseuri
, cryostat.agent.callback
, and cryostat.agent.authorization
as configuration options to the agent launcher. For more information about configuration properties, see Agent configuration properties.
3.2. Configuring applications to use JMX connections Copy linkLink copied to clipboard!
For Cryostat to detect and communicate with your target Java applications, you can configure the applications to allow remote Java Management Extensions (JMX) connections.
Prerequisites
- Logged in to your Cryostat web console.
- Created a Cryostat instance in your project.
Procedure
To enable remote JMX connections, complete the following steps:
On your application, define the following Java system property:
-Dcom.sun.management.jmxremote.port=<port_num>
-Dcom.sun.management.jmxremote.port=<port_num>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteTo add the
-Dcom.sun.management.jmxremote.port=<port_num>
property without having to rebuild the target application, you can set theJAVA_OPTS_APPEND
environment variable on the application.JAVA_OPTS_APPEND
is an environment variable that is used by Red Hat Universal Base Images (UBI) only.If you use Red Hat UBI to build application images, set the
JAVA_OPTS_APPEND
variable at build time in the application Docker file, or at runtime by running the following command:oc set env deployment <name> JAVA_OPTS_APPEND="..."
oc set env deployment <name> JAVA_OPTS_APPEND="..."
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If you do not use Red Hat UBI to build application images, refer to the documentation for your base image for information about how to add the Java system properties at build time or runtime.
Specify that the application listens for remote JMX connections by allowing traffic to the application. Use an Red Hat OpenShift Service and specify the following values for its remote JMX port:
Example
service.yaml
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Secure the remote JMX connections:
Enable and configure authentication and SSL/TLS for the remote JMX connections in your application:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Configure Cryostat to trust the application TLS certificate. Create a secret for the application in the same namespace as your Cryostat application and configure Cryostat to refer to the secret. To create a secret for the certificate, run the following command:
oc create secret generic myapp-cert --from-file=tls.crt=/path/to/cert.pem
oc create secret generic myapp-cert --from-file=tls.crt=/path/to/cert.pem
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe certificate must be in a
.pem
file format.- When you create your Cryostat instance, add the secret to the list of trusted TLS certificates. For more information, see Configuring TLS certificates.
To allow your applications to verify that Cryostat is connecting to them by a means other than password authentication, enable TLS client authentication:
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true -Djavax.net.ssl.trustStore=</path/to/truststore> -Djavax.net.ssl.trustStorePassword=<password>
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true -Djavax.net.ssl.trustStore=</path/to/truststore> -Djavax.net.ssl.trustStorePassword=<password>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteTLS client authentication requires the cert-manager operator for Red Hat OpenShift.
If you use TLS client authentication for remote JMX connections, the application truststore must contain a Cryostat certificate. The Cryostat operator cert-manager integration creates a self-signed certificate for the Cryostat deployment. This certificate is located in the
<cryostat>-tls
secret, where <cryostat> is the name of the Cryostat instance you created.NoteThe cert-manager Operator also places a Java keystore truststore in the secret.
To mount this truststore in your application deployment, run the following command, replacing "<myapp>" with the name of your application deployment and "<cryostat>" with the name of your Cryostat instance:
oc set volumes deploy <myapp> --add --name=truststore \ --secret-name=<cryostat>-tls --sub-path=truststore.p12 \ --mount-path=/var/run/secrets/<myapp>/truststore.p12
oc set volumes deploy <myapp> --add --name=truststore \ --secret-name=<cryostat>-tls --sub-path=truststore.p12 \ --mount-path=/var/run/secrets/<myapp>/truststore.p12
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The Cryostat operator generates the truststore password, which you can find in the
<cryostat>-keystore
secret. To mount this as an environment variable in your application deployment, run the following command:oc set env deploy <myapp> --from='secret/<cryostat>-keystore'
oc set env deploy <myapp> --from='secret/<cryostat>-keystore'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the Java arguments for the container. Run the following commands:
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true -Djavax.net.ssl.trustStore=/var/run/secrets/<myapp>/truststore.p12 -Djavax.net.ssl.trustStorePassword="$(KEYSTORE_PASS)"
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true -Djavax.net.ssl.trustStore=/var/run/secrets/<myapp>/truststore.p12 -Djavax.net.ssl.trustStorePassword="$(KEYSTORE_PASS)"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow WarningIf you deployed Cryostat and your applications in a testing environment, you might want to configure the target applications without any JMX or TLS authentication. You can do so by using the following set of Java system properties, however, this configuration is not secure and not recommended.
-Dcom.sun.management.jmxremote.port=<port_num> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=<port_num> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.3. Configuring applications to use the Cryostat agent and JMX connections Copy linkLink copied to clipboard!
You can configure target applications that run on Java Virtual Machines (JVM) to use a combination of the Cryostat agent and Java Management Extensions (JMX) connections to detect and communicate with these target applications.
In this situation, you can use the Cryostat agent to detect and communicate with the target application and use JMX to expose the Java Flight Recorder (JFR) data.
The Cryostat agent must be configured to communicate with Cryostat about itself and to indicate that the agent is reachable through JMX rather than through HTTP.
This type of hybrid configuration is supported regardless of whether the Cryostat Operator configures the agent automatically or you configure the agent manually. If the Cryostat agent detects that JMX is also configured on your application, the agent publishes itself to the Cryostat server with both agent HTTP API definitions and JMX URL definitions.
Prerequisites
- Logged in to your Cryostat web console.
- Created a Cryostat instance in your project.
Procedure
-
To allow the Cryostat Operator to configure and inject the agent automatically, ensure that you add the
cryostat.io/name
andcryostat.io/namespace
labels to your application deployment, as described in Enabling automatic configuration of the agent by the Cryostat Operator. -
Also ensure that you manually add the
CRYOSTAT_AGENT_REGISTRATION_PREFER_JMX
and‑Dcom.sun.management.jmxremote.port
settings to your application deployment by using environment variables or theargLine
parameter. To enable Cryostat to detect the target application and connect to the Cryostat agent, configure an application
Service
:Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Chapter 4. Creating a JFR recording with Cryostat Copy linkLink copied to clipboard!
With Cryostat, you can create a JDK Flight Recorder (JFR) recording that monitors the performance of your JVM in your containerized application. Additionally, you can take a snapshot of an active JFR recording to capture any collected data, up to a specific point in time, for your target JVM application.
4.1. Creating a JFR recording in the Cryostat web console Copy linkLink copied to clipboard!
You can create a JFR recording that monitors the performance of your JVM located in your containerized application. After you create a JFR recording, you can start the JFR to capture real-time data for your JVM, such as heap and non-heap memory usage.
Prerequisites
- Installed Cryostat 4.0 on Red Hat OpenShift by using the OperatorHub option.
- Created a Cryostat instance in your Red Hat OpenShift project.
Logged in to your Cryostat web console.
- You can retrieve your Cryostat application’s URL by using the Red Hat OpenShift web console.
Procedure
On the Dashboard panel for your Cryostat web console, select a target JVM from the Target list.
NoteDepending on how you configured your target applications, your target JVMs might be using a JMX connection or an agent HTTP connection. For more information about configuring your target applications, see Configuring Java applications.
ImportantIf your target JVM is using an agent HTTP connection that you have configured manually, ensure that you set the
cryostat.agent.api.writes-enabled
property totrue
when you configured your target application to load the Cryostat agent. Otherwise, the Cryostat agent cannot accept requests to start and stop JFR recordings. Alternatively, if you want the Cryostat Operator to configure the agent automatically, ensure that you add thecryostat.io/name
andcryostat.io/namespace
labels to your application deployment. For more information about Cryostat agent configuration, see Working with the Cryostat agent.Figure 4.1. Example of selecting a Target JVM for your Cryostat instance
Optional: On the Dashboard panel, you can create a target JVM. From the Target list, click Create Target. The Create Custom Target window opens.
- In the Connection URL field, enter the URL for your JVM’s Java Management Extension (JMX) endpoint.
- Optional: To test if the Connection URL that you specified is valid, click the Click to test sample node image. If there is an issue with the Connection URL, an error message is displayed that provides a description of the issue and guidance to troubleshoot.
- Optional: In the Alias field, enter an alias for your JMX Service URL.
Click Create.
Figure 4.2. Create Custom Target window
- From the navigation menu on the Cryostat web console, click Recordings.
Optional: Depending on how you configured your target JVM, an Authentication Required dialog might open on your web console. In the Authentication Required dialog box, enter your Username and Password. To provide your credentials to the target JVM, click Save.
Figure 4.3. Example of a Cryostat Authentication Required window
NoteIf the selected target JMX has Secure Socket Layer (SSL) certification enabled for JMX connections, you must add its certificate when prompted.
Cryostat encrypts and stores credentials for a target JVM application in a database that is stored on a persistent volume claim (PVC) on Red Hat OpenShift. See Storing and managing credentials (Using Cryostat to manage a JFR recording).
On the Active Recordings tab, click Create.
Figure 4.4. Example of creating an active recording
On the Custom Flight Recording tab:
- In the Name field, enter the name of the recording you want to create. If you enter a name in an invalid format, the web console displays an error message.
If you want Cryostat to automatically restart an existing recording, select the Restart if recording already exists check box.
NoteIf you enter a name that already exists but you do not select Restart if recording already exists, Cryostat refuses to create a custom recording when you click the Create button.
- In the Duration field, select whether you want this recording to stop after a specified duration or to run continuously without stopping. If you want Cryostat to automatically archive your new JFR recording after the recording stops, click Archive on Stop.
- In the Template field, select the template that you want to use for the recording.
The following example shows continuous JVM monitoring, which you can enable by selecting Continuous from above the Duration field. This setting means that the recording will continue until you manually stop the recording. The example also shows selection of the Profiling template from the Template field. This provides additional JVM information to a JFR recording for troubleshooting purposes.
Figure 4.5. Example of creating a custom flight recording
To access more options, click the following expandable hyperlinks:
- Show advanced options, where you can select additional options for customizing your JFR recording.
- Show metadata options, where you can add custom labels and metadata to your JFR recording.
To create your JFR recording, click Create. The Active Recordings tab opens and lists your JFR recording.
Your active JFR recording starts collecting data on the target JVM location inside your containerized application. If you specified a fixed duration for your JFR recordings, the target JVM stops the recording when it reaches the fixed duration setting. Otherwise, you must manually stop the recording.
Optional: On the Active Recording tab, you can also stop the recording.
- Select the checkbox next to the JFR recording’s name. On the toolbar in the Active Recordings tab, the Cryostat web console activates the Stop button.
Click Stop. The JFR adopts the
STOPPED
status, so it stops monitoring the target JVM. The JFR still shows under the Active Recording tab.Figure 4.6. Example of stopping an active recording
ImportantJFR recording data might be lost in the following situations:
- Target JVM fails
- Target JVM restarts
- Target JVM Red Hat OpenShift Deployment is scaled down
Archive your JFR recordings to ensure you do not lose your JFR recording’s data.
4.2. Creating snapshots from an active recording Copy linkLink copied to clipboard!
You can take a snapshot of an active JFR recording to capture any collected data, up to a specific point in time, for your target JVM application. A snapshot is like a checkpoint marker that has a start point and an end point for a given time segment in a running JFR recording.
A snapshot gets stored in the memory of a target JVM application. This differs from an archive in that Cryostat stores an archive on a cloud storage disk, which is a more permanent solution for storing a JFR recording’s data.
You can take snapshots of recordings if you want to experiment with different configuration changes among active JFR recordings.
When you create a snapshot for your JFR recording, Cryostat creates a new target JVM named snapshot -<snapshot_number>
, where <snapshot_number>
is the number that Cryostat automatically assigns to your snapshot.
A target JVM recognizes a snapshot as an active recording. Cryostat sets any JFR snapshots in the STOPPED
state, which means that the JFR snapshot does not record new data to the target JVM. Depending on the JFR configuration, active JFR recordings can continue to monitor the target JVM regardless of the number of snapshots taken.
For a JFR recording that you set for continuous monitoring of a target JVM application, ensure that you create archived recordings to avoid losing JFR recording data.
If you choose to take regular snapshots to store your JFR recording data, the target JVM application might free some of its data storage space by replacing older recording data with newer recording data.
Prerequisites
- Entered your authentication details for your Cryostat instance.
- Created a target JVM recording and entered your authenticated details to access the Recordings menu. See Creating a JDK Flight Recorder (JFR) recording (Creating a JFR recording with Cryostat).
Procedure
On the Active Recordings tab, click the Create button. A new window opens on the web console.
Figure 4.7. Example of creating an active recording
Click the Snapshot Recording tab.
Figure 4.8. Example of creating a snapshot recording
Click Create. The Active Recordings table opens and it lists your JFR snapshot recording. The following example shows a JFR snapshot recording named
snapshot-3
.Figure 4.9. Example of a completed snapshot recording
NoteYou can identify snapshots by the snapshot prefix from the list of active recordings.
Next steps
- To archive your JFR snapshot recording, see Archiving JDK Flight Recorder (JFR) recordings.
4.3. Labels for JFR recordings Copy linkLink copied to clipboard!
When you create a JDK Flight Recorder (JFR) recording on Cryostat 4.0, you can add metadata to the recording by specifying a series of key-value label pairs.
Additionally, you can attach custom labels to JFR recordings that are inside a target JVM, so that you can easily identify and better manage your JFR recordings.
The following list details some common recording label use cases:
- Attach metadata to your JFR recording.
- Perform batch operations on recordings that contain identical labels.
- Use labels when running queries on recordings.
You can use Cryostat to create a JFR recording that monitors the performance of your JVM in your containerized application. Additionally, you can take a snapshot of an active JFR recording to capture any collected data, up to a specific point in time, for your target JVM application.
4.3.1. Adding labels to JFR recordings Copy linkLink copied to clipboard!
When you create a JFR recording on Cryostat 4.0, you can use labels to add metadata that contain key-value label pairs to the recording.
Cryostat applies default recording labels to a created JFR recording. These default labels capture information about the event template that Cryostat used to create the JFR recording.
You can add custom labels to your JFR recording so that you can run specific queries that meet your needs, such as identifying specific JFR recordings or performing batch operations on recordings with the same applied labels.
Prerequisites
- Logged in to your Cryostat web console.
- Created or selected a target JVM for your Cryostat instance.
Procedure
- From your Cryostat web console, click Recordings.
- Under the Active Recordings tab, click Create.
On the Custom Flight Recording tab, expand Show metadata options.
NoteOn the Custom Flight Recording tab, you must complete any mandatory field that is marked with an asterisk.
Click Add label.
Figure 4.10. The Add Label button that is displayed under the Custom Flight Recording tab
- Enter values in the provided Key and Value fields. For example, if you want to file an issue with the recordings, you could enter the reason in the Key field and then enter the issue type in the Value field.
Click Create to create your JFR recording. Your recording is then shown under the Active Recordings tab along with any specified recording labels and custom labels.
TipYou can access archived JFR recordings from the Archives menu. See Uploading a JFR recording to Cryostat archives location (Using Cryostat to manage a JFR recording).
Example
The following example shows two default recording labels, template.name: Profiling
and template.type: TARGET
, and one custom label, reason:service-outage
.
Figure 4.11. Example of an active recording with defined recording labels and a custom label
4.3.2. Editing a label for your JFR recording Copy linkLink copied to clipboard!
On the Cryostat web console, you can navigate to the Recordings menu and then edit a label and its metadata for your JFR recording. You can also edit the label and metadata for a JFR recording that you uploaded to archives.
Prerequisites
- Logged in to your Cryostat web console.
- Created a JFR recording and attach labels to this recording.
Procedure
- On your Cryostat web console, click the Recording menu.
- From the Active Recordings tab, locate your JFR recording, and then select the checkbox next to it.
Click Edit Labels. An Edit Recording Label pane opens in your Cryostat web console, which you can use to add, edit, or delete labels for your JFR recording.
TipYou can select multiple JFR recordings by selecting the checkbox that is next to each recording. Click the Edit Labels button if you want to bulk edit recordings that contain the same labels or add new identical labels to multiple recordings.
Optional: You can perform any of the following actions from the Edit Recording Labels pane:
- Click Add to create a label.
-
Delete a label by clicking the
X
next to the label. - Edit a label by modifying any content in a field. After you edit content, a green tick is shown in the field to indicate an edit.
- Click Save.
Optional: You can archive your JFR recordings along with their labels by completing the following steps:
- Select the checkbox next to the recording’s name.
Click the Archive button. You can locate your recording under the Archived Recordings tab.
By archiving your recording with its labels, you can enhance your search capabilities when you want to locate the recording at a later stage. You can also add additional labels to any recording that you uploaded to the Cryostat archives.
NoteCryostat preserves any labels with the recording for the lifetime of the archived recording.
Verification
- From the Active Recordings tab, check that your changes display under the Labels section for your recording.
Revised on 2025-05-29 14:59:18 UTC