Chapter 7. Managing the Cluster

Procedure

1. Log in to the Dashboard.
2. On the navigation bar, click Cluster.

3. Click CRUSH map.

   

   In the above example, you can see the default CRUSH map, three nodes, and OSDs running on two of the three nodes.

4. Click on the CRUSH map name, nodes, or OSDs, to view details about each object.

   

   In the above example, you can see the values of variables related to an OSD running on the jb-rhel-osd3 node. In particular, note the id is 2.

Procedure

1. Log in to the Dashboard.
2. On the navigation bar, click Cluster.

3. Click Manager modules:

   

   The above screenshot shows the first of three pages of manager modules.

4. Click on a row for a module you want to configure:

   

   Not all modules have configurable parameters. If a module is not configurable, the Edit button is disabled.

5. Towards the upper left of the page, click the Edit button to load the page with the configurable parameters.

   

   The above screenshot shows parameters that you can change for the balancer module. To display a description of a parameter, click the question mark button.

6. To change a parameter, modify the state of the parameter and click the Update button at the bottom of the page:

   

   A notification confirming the change appears in the upper-right corner of the page:

   

Procedure

1. Log in to the Dashboard.
2. Click the Cluster drop-down menu in the top navigation bar.
3. Click Logs in the drop-down menu.

4. View the last thirty unfiltered log entries.

   

   1. To filter by priority, click the Priority drop-down menu and select either Info, Warning, or Error. The example below only shows log entries with the priority of Error.

      

   2. To filter by keyword, enter text into the Keyword form. The example below only shows log entries that include the text osd.2.

      

   3. To filter by date, click the Date form and either use the date picker to select a date from the menu, or enter a date in the form of YYYY-MM-DD. The example below only shows log entries with the date of 2019-10-15.

      

   4. To filter by time, enter a range in the Time range fields using the HH:MM - HH:MM format. Hours must be entered using numbers 0 to 23. The example below only shows log entries from 12:14 to 12:23.

      

   5. To combine filters, set two or more filters. The example below only shows entries that have both a Priority of Warning and the keyword of osd.

      

Procedure

1. Log in to the dashboard.
2. Click the Cluster drop-down menu in the top navigation bar.
3. Click OSDs in the drop-down menu.

4. Click the Cluster-Wide Flags drop-down menu.

   

5. Select Cluster-wide Recovery Priority in the drop-down.

   

6. Optional: Select Priority in the drop-down menu , and then click the Submit button.

   Note

   There are 3 predefined options: Low, Default, High

   

7. Optional: Click Customize priority values, make the required changes, and then click the Submit button.

   

8. A notification towards the top right corner of the page pops up indicating the flags were updated successfully.

   

Procedure

1. Log in to the Dashboard.
2. On the navigation bar, click Cluster and then click Configuration.

3. To view the details of the configuration, click its row:

   

   • You can search for the configuration using the Search box:

     

   • You can filter for the configuration using Level, Service or Source drop-down:

     

4. To edit a configuration, click its row and click the Edit button:

   

   1. In the dialog window, edit the required parameters and Click the Save button:

      

      A notification confirming the change appears in the upper-right corner of the page.

      

Procedure

1. Open and edit the /etc/prometheus/prometheus.yml file.

   1. Under the global section, set the scrape_interval and evaluation_interval options to 15 seconds.

      Example

      global:
        scrape_interval:     15s
        evaluation_interval: 15s

   2. Under the scrape_configs section, add the honor_labels: true option, and edit the targets, and instance options for each of the ceph-mgr nodes.

      Example

      scrape_configs:
        - job_name: 'node'
          honor_labels: true
          static_configs:
          - targets: [ 'node1.example.com:9100' ]
            labels:
              instance: "node1.example.com"
          - targets: ['node2.example.com:9100']
            labels:
              instance: "node2.example.com"

      Note

      Using the honor_labels option enables Ceph to output properly-labelled data relating to any node in the Ceph storage cluster. This allows Ceph to export the proper instance label without Prometheus overwriting it.

   3. To add a new node, simply add the targets, and instance options in the following format:

      Example

      - targets: [ 'new-node.example.com:9100' ]
        labels:
          instance: "new-node"

      Note

      The instance label has to match what appears in Ceph’s OSD metadata instance field, which is the short host name of the node. This helps to correlate Ceph stats with the node’s stats.

2. Add Ceph targets to the /etc/prometheus/ceph_targets.yml file in the following format.

   Example

   [
       {
           "targets": [ "cephnode1.example.com:9283" ],
           "labels": {}
       }
   ]

3. Enable the Prometheus module:

   [root@mon ~]# ceph mgr module enable prometheus

Procedure

1. Take the back-up of the Grafana database:

   1. On the grafana-server node, stop the Grafana service:

      Example

      [root@node04 ~]# systemctl stop grafana-server.service
      [root@node04 ~]# systemctl status grafana-server.service

   2. Take the back-up of the Grafana database:

      Example

      [root@node04 ~]# cp /var/lib/grafana/grafana.db /var/lib/grafana/grafana_backup.db

   3. On the grafana-server node, restart the Grafana service:

      Example

      [root@node04 ~]# systemctl restart grafana-server.service

2. Restore the grafana-server:

   1. On the grafana-server node, if the Grafana service is running, stop the service:

      Example

      [root@node04 ~]# systemctl stop grafana-server.service
      [root@node04 ~]# systemctl status grafana-server.service

   2. Move the backed-up grafana.db file to /var/lib/grafana/ directory:

      Example

      [root@node04 ~]# mv /var/lib/grafana/grafana_backup.db /var/lib/grafana/

   3. On the grafana-server node, restart the Grafana service:

      Example

      [root@node04 ~]# systemctl restart grafana-server.service

3. For the Prometheus alerts, you have to take external back-up of prometheus_data_dir directory, a Ceph-Ansible setting which by default is var/lib/prometheus directory and restore the service using the backed-up directory.

   1. On the grafana-server node, stop the Prometheus service:

      Example

      [root@node04 ~]# systemctl stop prometheus.service
      [root@node04 ~]# systemctl status prometheus.service

   2. Take the back-up of the default Prometheus directory:

      Example

      [root@node04 ~]# cp /var/lib/prometheus/ /var/lib/prometheus_backup/

   3. Replace the prometheus_data_dir directory with the backed-up directory:

      Example

      [root@node04 ~]# mv /var/lib/prometheus_backup/ /var/lib/prometheus_data_dir

   4. On the grafana-server node, restart the prometheus service:

      Example

      [root@node04 ~]# systemctl restart prometheus.service
      [root@node04 ~]# systemctl status prometheus.service

      Note

      If you have made changes to the Prometheus parameters in group_vars/all.yml file, then you have to rerun the playbook.

4. Optional: If the changes do not reflect on the Red Hat Ceph Storage Dashboard, then you have to disable and then enable the dashboard:

   Example

   [root@node04 ~]# ceph mgr module disable dashboard
   [root@node04 ~]# ceph mgr module enable dashboard