Chapter 2. Monitoring and logging Red Hat Developer Hub on Amazon Web Services (AWS)
You can configure Red Hat Developer Hub to use Amazon CloudWatch for real-time monitoring and Amazon Prometheus for comprehensive logging. This is convenient when hosting Developer Hub on Amazon Web Services (AWS) infrastructure.
2.1. Monitoring with Amazon Prometheus
You can configure Red Hat Developer Hub to use Amazon Prometheus for comprehensive logging. Amazon Prometheus extracts data from pods that have specific pod annotations.
2.1.1. Prerequisites
- You configured Prometheus for your Elastic Kubernetes Service (EKS) clusters.
- You created an Amazon managed service for the Prometheus workspace.
- You configured Prometheus to import the Developer Hub metrics.
- You ingested Prometheus metrics into the created workspace.
2.1.2. Configuring annotations for monitoring with Amazon Prometheus by using the Red Hat Developer Hub Operator
To enable logging to Amazon Prometheus, you can configure the required pod annotations by using the Red Hat Developer Hub Operator.
Procedure
As an administrator of the Red Hat Developer Hub Operator, edit the default configuration to add Prometheus annotations as follows:
Update OPERATOR_NS accordingly OPERATOR_NS=rhdh-operator kubectl edit configmap backstage-default-config -n "${OPERATOR_NS}"
# Update OPERATOR_NS accordingly $ OPERATOR_NS=rhdh-operator $ kubectl edit configmap backstage-default-config -n "${OPERATOR_NS}"
Copy to Clipboard Copied! Find the
deployment.yaml
key in the config map and add the annotations to thespec.template.metadata.annotations
field as follows:deployment.yaml: |- apiVersion: apps/v1 kind: Deployment # --- truncated --- spec: template: # --- truncated --- metadata: labels: rhdh.redhat.com/app: # placeholder for 'backstage-<cr-name>' # --- truncated --- annotations: prometheus.io/scrape: 'true' prometheus.io/path: '/metrics' prometheus.io/port: '9464' prometheus.io/scheme: 'http' # --- truncated ---
deployment.yaml: |- apiVersion: apps/v1 kind: Deployment # --- truncated --- spec: template: # --- truncated --- metadata: labels: rhdh.redhat.com/app: # placeholder for 'backstage-<cr-name>' # --- truncated --- annotations: prometheus.io/scrape: 'true' prometheus.io/path: '/metrics' prometheus.io/port: '9464' prometheus.io/scheme: 'http' # --- truncated ---
Copy to Clipboard Copied! - Save your changes.
Verification
To verify if the scraping works:
Use
kubectl
to port-forward the Prometheus console to your local machine as follows:kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090
$ kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090
Copy to Clipboard Copied! -
Open your web browser and navigate to
http://localhost:9090
to access the Prometheus console. -
Monitor relevant metrics, such as
process_cpu_user_seconds_total
.
2.1.3. Configuring annotations for monitoring with Amazon Prometheus by using the Red Hat Developer Hub Helm chart
To enable logging to Amazon Prometheus, you can configure the required pod annotations by using the Red Hat Developer Hub Helm chart.
Procedure
To annotate the backstage pod for monitoring, update your
values.yaml
file as follows:upstream: backstage: # --- TRUNCATED --- podAnnotations: prometheus.io/scrape: 'true' prometheus.io/path: '/metrics' prometheus.io/port: '9464' prometheus.io/scheme: 'http'
upstream: backstage: # --- TRUNCATED --- podAnnotations: prometheus.io/scrape: 'true' prometheus.io/path: '/metrics' prometheus.io/port: '9464' prometheus.io/scheme: 'http'
Copy to Clipboard Copied!
Verification
To verify if the scraping works:
Use
kubectl
to port-forward the Prometheus console to your local machine as follows:kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090
kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090
Copy to Clipboard Copied! -
Open your web browser and navigate to
http://localhost:9090
to access the Prometheus console. -
Monitor relevant metrics, such as
process_cpu_user_seconds_total
.
2.2. Logging with Amazon CloudWatch
Logging within the Red Hat Developer Hub relies on the Winston library. The default logging level is info
. To have more detailed logs, set the LOG_LEVEL
environment variable to debug
in your Red Hat Developer Hub instance.
2.2.1. Configuring the application log level by using the Red Hat Developer Hub Operator
You can configure the application log level by using the Red Hat Developer Hub Operator.
Procedure
Modify the logging level by including the environment variable
LOG_LEVEL
in your custom resource as follows:spec: # Other fields omitted application: extraEnvs: envs: - name: LOG_LEVEL value: debug
spec: # Other fields omitted application: extraEnvs: envs: - name: LOG_LEVEL value: debug
Copy to Clipboard Copied!
2.2.2. Configuring the application log level by using the Red Hat Developer Hub Helm chart
You can configure the application log level by using the Red Hat Developer Hub Helm chart.
Procedure
Modify the logging level by adding the environment variable
LOG_LEVEL
to your Helm chartvalues.yaml
file:upstream: backstage: # --- Truncated --- extraEnvVars: - name: LOG_LEVEL value: debug
upstream: backstage: # --- Truncated --- extraEnvVars: - name: LOG_LEVEL value: debug
Copy to Clipboard Copied!
2.2.3. Retrieving logs from Amazon CloudWatch
Prerequisites
- CloudWatch Container Insights is used to capture logs and metrics for Amazon Elastic Kubernetes Service. For more information, see Logging for Amazon Elastic Kubernetes Service documentation.
- To capture the logs and metrics, install the Amazon CloudWatch Observability EKS add-on in your cluster. Following the setup of Container Insights, you can access container logs using Logs Insights or Live Tail views.
CloudWatch names the log group where all container logs are consolidated in the following manner:
/aws/containerinsights/<cluster_name>/application
/aws/containerinsights/<cluster_name>/application
Copy to Clipboard Copied!
Procedure
To retrieve logs from the Developer Hub instance, run a query such as:
fields @timestamp, @message, kubernetes.container_name | filter kubernetes.container_name in ["install-dynamic-plugins", "backstage-backend"]
fields @timestamp, @message, kubernetes.container_name | filter kubernetes.container_name in ["install-dynamic-plugins", "backstage-backend"]
Copy to Clipboard Copied!