Power Autonomous Root Cause Analysis for Distributed Systems

Copy link

Route telemetry through an event-driven pipeline where AI analyzes correlated logs and traces, turning raw signals into actionable root cause analysis.

Table of contents

Copy link

1. Detailed description
2. Requirements
3. Deploy
4. References
5. Tags

Detailed description

Copy link

An intelligent observability pipeline that automatically detects microservice errors in distributed applications, correlates logs and traces, and uses GenAI to provide SREs with actionable remediation steps. Built with OpenTelemetry, Kafka, Camel, Artemis, Infinispan and LLMs to dramatically reduce MTTR via AI-assisted diagnostics.

Site Reliability Engineers spend significant time manually sifting through logs and traces to diagnose production incidents. In complex microservice architectures, a single user request can span dozens of services, making root cause analysis slow and error-prone. This pipeline automates that investigation: it continuously ingests OpenTelemetry data, detects errors, correlates related signals by trace ID, and sends the correlated context to an LLM for immediate root cause analysis delivering actionable insights in seconds.

The system consists of three Apache Camel applications deployed on Red Hat® OpenShift®. The correlator consumes logs and traces from Kafka and groups them by trace ID in Infinispan. The analyzer retrieves correlated events and sends them to a Granite LLM for root cause analysis. The UI console stores results and provides a web interface for reviewing analyses and triggering interactive re-analysis with custom prompts.

Architecture diagrams

Copy link

Requirements

Copy link

Minimum hardware requirements

Copy link

No dedicated hardware is required. The application runs entirely within the Red Hat Developer Sandbox, which provides shared compute resources.

Minimum software requirements

Copy link

Required user permissions

Copy link

Regular sandbox user permissions (no cluster-admin required). All components are deployed as containers within the user's namespace.

Deploy

Copy link

1. Get access to the Developer Sandbox

Copy link

1. Go to https://developers.redhat.com/developer-sandbox
2. Click Start your sandbox for free and log in with your Red Hat account
3. Complete the registration (phone verification may be required)
4. Once provisioned, click Start using your sandbox to open the OpenShift web console

2. Activate OpenShift AI models

Copy link

In the sandbox landing page, find the OpenShift AI card and click Try it. This provisions the Granite LLM inference service that the analyzer uses for root cause analysis.

3. Log in to the cluster

Copy link

1. In the OpenShift web console, click your username in the top-right corner

2. Select Copy login command > Display Token

3. Run the oc login command in your terminal:

   oc login --token=sha256~XXXX --server=https://api.sandbox-XXXX.openshiftapps.com:6443
   

4. Verify your namespace:

   oc project
   

   You should see <username>-dev.

4. Clone and deploy

Copy link

git clone https://github.com/rh-ai-quickstart/smart-telemetry-pipeline.git
cd smart-telemetry-pipeline
./create.sh

The create.sh script automates the full installation: infrastructure (Kafka, OTel Collector, Infinispan, AMQ Broker), OpenAI credentials configuration, application builds via Tekton pipelines, and Helm deployment. It takes approximately 10-15 minutes to complete.

5. Verify the deployment

Copy link

Check that all pods are running:

oc get pods

You should see pods for kafka, infinispan, artemis, camel-otel-collector, correlator, analyzer, and ui-console.

Open the UI Console in your browser:

6. Generate test data

Copy link

Deploy the log generator to produce simulated telemetry (orders with a 30% failure rate):

./log-generator/run.sh

Open the UI Console to see AI-generated root cause analyses appearing for the detected errors. Click on a trace to view the detailed analysis for correlated traces and logs, and use interactive prompting to ask follow-up questions about the initial analysis.

To stop generating test data:

./log-generator/delete.sh

Delete

Copy link

Remove all deployed resources:

./delete.sh

For manual step-by-step deployment and deletion, see the Manual Deployment Guide.

References

Copy link

• Red Hat build of Apache Camel
• Red Hat Application Foundations

Additional documentation

Copy link

• Manual Deployment Guide — step-by-step infrastructure and application deployment
• Monitoring and Alerting — Prometheus metrics, PromQL queries, and alert configuration
• Troubleshooting — common issues and fixes
• Technical Details — component architecture and build pipeline