Chapter 7. Managing alerts on the Ceph dashboard

Procedure

1. Log into the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Set the APIs for the monitoring stack:

   1. Specify the host and port of the Alertmanager server:

      Syntax

      ceph dashboard set-alertmanager-api-host ALERTMANAGER_API_HOST:PORT

      Example

      [ceph: root@host01 /]# ceph dashboard set-alertmanager-api-host http://10.0.0.101:9093
      Option ALERTMANAGER_API_HOST updated

   2. To see the configured alerts, configure the URL to the Prometheus API. Using this API, the Ceph Dashboard UI verifies that a new silence matches a corresponding alert.

      Syntax

      ceph dashboard set-prometheus-api-host PROMETHEUS_API_HOST:PORT

      Example

      [ceph: root@host01 /]# ceph dashboard set-prometheus-api-host http://10.0.0.101:9095
      Option PROMETHEUS_API_HOST updated

      After setting up the hosts, refresh your browser’s dashboard window.

      • Specify the host and port of the Grafana server:

        Syntax

        ceph dashboard set-grafana-api-url GRAFANA_API_URL:PORT

        Example

        [ceph: root@host01 /]# ceph dashboard set-grafana-api-url https://10.0.0.101:3000
        Option GRAFANA_API_URL updated

3. Get the Prometheus, Alertmanager, and Grafana API host details:

   Example

   [ceph: root@host01 /]# ceph dashboard get-alertmanager-api-host
   http://10.0.0.101:9093
   [ceph: root@host01 /]# ceph dashboard get-prometheus-api-host
   http://10.0.0.101:9095
   [ceph: root@host01 /]# ceph dashboard get-grafana-api-url
   http://10.0.0.101:3000

4. Optional: If you are using a self-signed certificate in your Prometheus, Alertmanager, or Grafana setup, disable the certificate verification in the dashboard This avoids refused connections caused by certificates signed by an unknown Certificate Authority (CA) or that do not match the hostname.

   • For Prometheus:

     Example

     [ceph: root@host01 /]# ceph dashboard set-prometheus-api-ssl-verify False

   • For Alertmanager:

     Example

     [ceph: root@host01 /]# ceph dashboard set-alertmanager-api-ssl-verify False

   • For Grafana:

     Example

     [ceph: root@host01 /]# ceph dashboard set-grafana-api-ssl-verify False

5. Get the details of the self-signed certificate verification setting for Prometheus, Alertmanager, and Grafana:

   Example

   [ceph: root@host01 /]# ceph dashboard get-prometheus-api-ssl-verify
   [ceph: root@host01 /]# ceph dashboard get-alertmanager-api-ssl-verify
   [ceph: root@host01 /]# ceph dashboard get-grafana-api-ssl-verify

6. Optional: If the dashboard does not reflect the changes, you have to disable and then enable the dashboard:

   Example

   [ceph: root@host01 /]# ceph mgr module disable dashboard
   [ceph: root@host01 /]# ceph mgr module enable dashboard

Procedure

1. Log into the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Configure the custom certificate for Grafana:

   Example

   [ceph: root@host01 /]# ceph config-key set mgr/cephadm/grafana_key -i $PWD/key.pem
   [ceph: root@host01 /]# ceph config-key set mgr/cephadm/grafana_crt -i $PWD/certificate.pem

3. If Grafana is already deployed, then run reconfig to update the configuration:

   Example

   [ceph: root@host01 /]# ceph orch reconfig grafana

4. Every time a new certificate is added, follow the below steps:

   1. Make a new directory

      Example

      [root@host01 ~]# mkdir /root/internalca
      [root@host01 ~]# cd /root/internalca

   2. Generate the key:

      Example

      [root@host01 internalca]# openssl ecparam -genkey -name secp384r1 -out $(date +%F).key

   3. View the key:

      Example

      [root@host01 internalca]# openssl ec -text -in $(date +%F).key | less

   4. Make a request:

      Example

      [root@host01 internalca]# umask 077; openssl req -config openssl-san.cnf -new -sha256 -key $(date +%F).key -out $(date +%F).csr

   5. Review the request prior to sending it for signature:

      Example

      [root@host01 internalca]# openssl req -text -in $(date +%F).csr | less

   6. As the CA sign:

      Example

      [root@host01 internalca]# openssl ca -extensions v3_req -in $(date +%F).csr -out $(date +%F).crt -extfile openssl-san.cnf

   7. Check the signed certificate:

      Example

      [root@host01 internalca]# openssl x509 -text -in $(date +%F).crt -noout | less

Procedure

1. Log into the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Configure the Alertmanager to use the webhook for notification:

   Syntax

   service_type: alertmanager
   spec:
     user_data:
       default_webhook_urls:
       - "_URLS_"

   The default_webhook_urls is a list of additional URLs that are added to the default receivers' webhook_configs configuration.

   Example

   service_type: alertmanager
   spec:
     user_data:
       webhook_configs:
       - url: 'http:127.0.0.10:8080'

3. Update Alertmanager configuration:

   Example

   [ceph: root@host01 /]#  ceph orch reconfig alertmanager

Procedure

1. From the dashboard navigation, go to Observability→Alerts.
2. View active Prometheus alerts from the Active Alerts tab.

3. View all alerts from the Alerts tab.

   To view alert details, expand the alert row.

4. To view the source of an alert, click on its row, and then click Source.

   

Procedure

1. From the dashboard navigation, go to Observability→Alerts.
2. On the Silences tab, click Create.

3. In the Create Silence form, fill in the required fields.

   1. Use the Add matcher to add silence requirements.

      Figure 7.2. Creating a silence

      

4. Click Create Silence.

   A notification displays that the silence was created successfully and the Alerts Silenced updates in the Silences table.

Procedure

1. From the dashboard navigation, go to Observability→Alerts.
2. On the Silences tab, select the row with the alert that you want to recreate, and click Recreate from the action drop-down.

3. Edit any needed details, and click Recreate Silence button.

   A notification displays indicating that the silence was edited successfully and the status of the silence is now active.

Procedure

1. Log in to the Dashboard.
2. On the navigation menu, click Cluster.
3. Select Monitoring from the drop-down menu.
4. Click the Silences tab.
5. To edit the silence, click it’s row.
6. In the Edit drop-down menu, select Edit.

7. In the Edit Silence window, update the details and click Edit Silence.

   Figure 7.3. Edit silence

   
8. You get a notification that the silence was updated successfully.

Procedure

1. From the dashboard navigation, go to Observability→Alerts.
2. On the Silences tab, select the row with the alert that you want to expire, and click Expire from the action drop-down.

3. In the Expire Silence notification, select Yes, I am sure and click Expire Silence.

   A notification displays indicating that the silence was expired successfully and the Status of the alert is expired, in the Silences table.