Optimize workflow performance

To maintain high performance in Red Hat Developer Hub, you must identify and resolve execution delays. Use distributed tracing to visualize the execution path and workflows and determine where time is spent across service boundaries.

Collect distributed traces

To monitor workflows in Red Hat Developer Hub, you must enable distributed tracing. Distributed tracing shows the executed path of workflow steps and identifies where failures occur.

Before you begin

  • You have installed and configured the SonataFlow Operator.
  • You have cluster-admin or equivalent permissions to deploy observability infrastructure and modify ConfigMaps.
  • A Kubernetes or OpenShift cluster is available.

About this task

To enable observability features such as tracing and metrics in the SonataFlow runtime, you must add the OpenTelemetry addon and configure the workflow properties. The sonataflow-addons-quarkus-opentelemetry addon provides a standard configuration with minimal setup required.

The OpenTelemetry integration for SonataFlow includes the following capabilities:

  • Distributed tracing: Track workflow execution across services and steps.
  • Metrics collection: Monitor performance, duration, and success rates.
  • Log aggregation: Centralize logs with trace correlation.
  • Context propagation: Maintain trace context across workflow boundaries and asynchronous operations.

Procedure

  1. Add the OpenTelemetry addon to the QUARKUS_EXTENSIONS environment variable during the image build process:
    export QUARKUS_EXTENSIONS="${QUARKUS_EXTENSIONS},org.apache.kie.sonataflow:sonataflow-addons-quarkus-opentelemetry"
  2. Open the <workflow-name>-props ConfigMap for your workflow.
  3. In the application.properties section, enable the OpenTelemetry integration and configure the service attributes:
    # Application Identity
    quarkus.application.name=my-workflow
    quarkus.application.version=1.0.0
    
    # OpenTelemetry Configuration
    quarkus.otel.enabled=true
    quarkus.otel.traces.enabled=true
    quarkus.otel.metrics.enabled=true
    quarkus.otel.logs.enabled=true
    
    # Service Resource Attributes
    quarkus.otel.resource.attributes=\
      service.name=my-workflow,\
      service.namespace=workflows,\
      service.version=1.0.0,\
      deployment.environment=production
    
    # SonataFlow Specific Configuration
    # Master switch for SonataFlow OpenTelemetry integration
    sonataflow.otel.enabled=true
    # Service identification (uses Quarkus application name/version as defaults)
    sonataflow.otel.service-name=${quarkus.application.name:kogito-workflow-service}
    sonataflow.otel.service-version=${quarkus.application.version:unknown}
    # Enable span creation for workflow states
    sonataflow.otel.spans.enabled=true
    # Enable process lifecycle events (start, complete, error, state transitions)
    sonataflow.otel.events.enabled=true
  4. Save the ConfigMap and restart the workflow pod to apply the changes.

Results

  1. Verify that the OpenTelemetry addon is loaded by checking the pod logs:
    oc logs -n workflows deployment/onboarding-workflow | grep "sonataflow-addons-quarkus-opentelemetry"
  2. Verify the trace report status:
    oc logs -n workflows deployment/greeting | grep -i "export\|batch"
  3. Confirm that the observability backend, such as Jaeger, is receiving data:
    oc logs -n observability deployment/jaeger | grep -i "span\|trace"

Route workflow traces to existing observability tools

Route workflow traces to the observability tools your team already uses instead of learning a new system. Exporter configuration sends telemetry data to Jaeger, Grafana, or any OTLP-compatible platform.

Before you begin

  • You have enabled OpenTelemetry in your workflow.
  • An observability platform (Jaeger or OpenTelemetry Collector) is available in your cluster.

Procedure

  1. Configure an export strategy based on your environment requirements:
    • Configure the OTLP exporter with batch processing (Recommended)

      For production environments, use an OTLP exporter with batch processing to reduce network overhead and improve performance:

      # OTLP exporter - Direct to Jaeger
      quarkus.otel.exporter.otlp.endpoint=http://jaeger-collector.observability.svc.cluster.local:4317
      quarkus.otel.exporter.otlp.protocol=grpc
      quarkus.otel.traces.exporter=cdi
      
      # Batch processing for production
      quarkus.otel.bsp.schedule.delay=5s
      quarkus.otel.bsp.max.export.batch.size=512
      quarkus.otel.bsp.export.timeout=2s
      quarkus.otel.bsp.max.queue.size=2048
    • Configure direct export to an external platform

      For development or simple integrations, use a direct export configuration:

      # Example: Direct export to Jaeger
      quarkus.otel.exporter.otlp.endpoint=http://jaeger-collector:4317
      quarkus.otel.exporter.otlp.protocol=grpc
      quarkus.otel.traces.exporter=cdi
  2. Externalize the configuration for production deployments by using environment variables. This ensures that your deployment remains secure and flexible across environments.
    # Externalized Configuration
    quarkus.otel.exporter.otlp.endpoint=${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4317}
    quarkus.otel.exporter.otlp.headers=${OTEL_EXPORTER_OTLP_HEADERS:}
    quarkus.application.name=${OTEL_SERVICE_NAME:my-workflow}
    quarkus.otel.resource.attributes=${OTEL_RESOURCE_ATTRIBUTES:deployment.environment=dev}

Troubleshoot trace connectivity

Symptom-to-solution reference table for missing traces, authentication failures, and broken context propagation. Use this troubleshooting guide when traces fail to appear in Jaeger, authentication errors occur, or telemetry delivery stops working.

Symptom Potential cause Resolution

Traces do not appear in the dashboard.

OpenTelemetry is disabled or the endpoint is unreachable.

Verify the quarkus.otel.enabled property and test endpoint connectivity.

Authentication errors (401/403).

Missing or invalid authorization headers.

Configure the quarkus.otel.exporter.otlp.headers property with a valid token.

High memory usage in the collector.

Large telemetry batches or high traffic volume.

Implement a memory_limiter processor in the collector configuration.

Context is lost between workflow steps.

Incorrect propagator configuration.

Ensure quarkus.otel.propagators includes all required formats (for example, tracecontext and baggage).

Diagnose missing traces

  1. Verify that OpenTelemetry is enabled in the workflow ConfigMap:
    oc get cm <workflow-name>-props -n workflows -o yaml
    1. Check the pod logs for initialization errors:
      oc logs deployment/<deployment-name> -n workflows | grep -i "otel"
    2. Test the connection to the Jaeger collector from within the workflow pod:
      oc exec deployment/<deployment-name> -- curl -v http://jaeger-collector:4317
  2. Configure authentication headers. If your observability platform requires authentication, add the following property to your application.properties file:
    quarkus.otel.exporter.otlp.headers=authorization=Bearer ${API_TOKEN}
  3. Resolve context propagation issues. To ensure trace IDs are maintained across service boundaries, configure the following propagators and enable JSON logging to verify the IDs in the output:
# Include required propagators
quarkus.otel.propagators=tracecontext,baggage,jaeger

# Enable JSON logging to verify trace IDs
quarkus.log.console.json=true